A Generic IOCP Framework
from:http://www.codeguru.com/cpp/i-n/internet/network/article.php/c7239/
Robert Simpson
October 18, 2004

Environment: VC++ 7.0, WinNT

点击浏览该文件 点击浏览该文件

Introduction
For building high-performance I/O-intensive applications, nothing beats I/O completion ports. I've read many articles and code samples for I/O completion ports, but it seemed like they were either too specific, or way too heavy for general purpose use. I'm a big fan of the KISS method (Keep It Simple Stupid—in case you've been living in isolation too long), so I decided to develop a few lightweight classes of my own.

Design Goals

The design goals were simple. First, I wanted to do this whole thing using a minimal number of classes. One thing I always hate about using someone else's code is that invariably they have a lot of their own classes for every little thing even remotely related to the core technology they're trying to demonstrate. A threadpool class, an event source base class, and a callback class will do nicely for us here. Any class wanting to use the threadpool and receive events on it would inherit the base event source class, and rather than using the OVERLAPPED struct, you'd use the callback class. The threadpool would be completely transparent, but also completely reusable for any kind of IOCP operation imaginable.

Additionally, I needed a few more capabilities. I needed to be able to cleanly dispose of any of these objects during a callback. I needed to be able to re-use a callback class from within a callback. Most importantly, this whole thing won't work at all if we have to serialize access to things using mutexes or critical sections during every single IOCP callback operation. The last thing we want is our worker threads potentially blocking on each other during every completed I/O event.

The Players
CAsyncEventSource A base class to be inherited by all classes that want to perform any kind of asynchronous I/O on the threadpool. Its job is to keep track of how many pending I/O operations are queued up for an instance of this class, and provide a way to wait for all pending operations to complete (for a clean disposal) for this instance. It also has several static utility members to associate a handle with the threadpool, check whether the current thread is part of the pool, and so forth.
CAsyncCallback A wrapper around the OVERLAPPED struct for starters. This class provides an entry point for the thread pool to execute on, which in turn allows the class to callback into your class. It also helps manage the pending I/O count in CAsyncEventSource, and provides some functions to post events to the threadpool.
CThreadPool2<> A template class, shamelessly stolen and modified from ATL7's CThreadPool template class. This one fixes bugs in ATL's implementation and provides a little more robust functionality. (More on that below.)

The Issues
The first issue that came up was ATL7's CThreadPool<> template class. The ThreadProc() function has a while() loop on GetQueuedCompletionStatus()'s return value. OOPS! If any queued I/O event fails, or the function call itself failed, the thread will exit! When the time came to destruct the class, it would wait indefinitely on a thread that was no longer running. Next, I needed a new method to query whether or not the current active thread was one that belonged to the thread pool. So, I made these changes to the class and decided to extract it out of the headers and call it CThreadPool2. Although we include atlbase.h, you don't have to be writing an ATL application to use this class. CThreadPool2 is the only class of the three that even remotely resembles ATL or depends on anything in atlbase.h.

When designing the CAsync classes, I had several issues to overcome. First, because CAsyncCallback replaces the OVERLAPPED struct, I now had the opportunity to create a robust callback mechanism. I needed to make sure that it was thread-safe, that a given instance of the class could be re-used from within a callback, and that it could be safely destroyed from within a callback. I also needed to make sure that it properly incremented/decremented the pending I/O operations for the CAsyncEventSource it was tied to, even if it was re-used or destroyed during a callback operation.

Next, the CAsyncEventSource has a WaitForPending() function to allow a parent class to wait for all overlapped operations to complete before destroying itself. However, this wait operation needed to be able to determine whether it was being called from within a callback. If it was being called from within a callback, the count of pending I/O operations would never reach zero (the operation won't complete until this function exits, which never happens), and would cause an infinite wait. The answer, of course, is to detect whether the active thread is a thread belonging to the thread pool. If it is, we have to artificially decrease our refcount to accommodate the fact that the current overlapped operation will not be complete until this function returns. That, unfortunately, presents us with another problem. If this function returns and the parent class is destroyed immediately afterwards (during the callback), we could be in trouble. You see, the CAsyncCallback, once it finishes its callback function, has to decrement the pending I/O refcounts. If the CAsyncEventSource class is destroyed during the function call, CAsyncCallback's pointer to it becomes invalid.

TLS to the Rescue
So, we needed a way for CAsyncCallback to meet our destruction requirements (legal to destroy during a callback, legal to re-use during a callback, and legal for the CAsyncEventSource base and parent to be destroyed during a callback). The solution was simple. Thread local storage. CAsyncCallback will store a value in our TLS index before making the callback. If that value remains unchanged after the function returns, CAsyncCallb[FS:PAGE]ack will assume the base CAsyncEventSource it is tied to still exists and will decrement the pending I/O refcount. CAsyncEventSource's destructor will (if called from a thread in the threadpool) reset this TLS index's value, thereby causing CAsyncCallback NOT to decrement the refcount on return.