Comparative measurements and analysis of I/O event notification mechanisms

Abstract

This paper describes a comparative measurements and analysis of the most popular I/O event notification mechanisms available on UNIX-like operating systems: select, poll, kqueue, /dev/poll, epoll. This article highlights the importance, functionality, performances, portability and the scalability of these notification mechanisms. This paper ends by presenting a portable framework based on these event notification mechanisms, which allows an application to be notified when an event occurs on a file descriptor, in a scalable and efficient manner. This framework can be extended to cover future notification mechanisms without changing the application interface.

Introduction and Motivation

The aim of this paper is to provide a complex analysis of event notification mechanisms available in UNIX-like operating systems. The article is addressed to all those who are interested in developing or analysis applications that handle a large number of simultaneous client connections using "event-driven" mechanisms.

The problem statement is that in our days, many client/server applications need to read input from or write output to multiple file descriptors at a time. For example, modern Web browsers open many simultaneous network connections to reduce the loading time for a Web page. This allows them to download the multiple images that appear on most Web pages more quickly than consecutive connections would allow.

The easiest way to handle all these files is to read from each file in turn and process whatever data that file delivers (a read() system call on a network connection returns whatever data is currently available and blocks only if no bytes are ready). This approach works fine, as long as all the connections are delivering data fairly regularly.

If one of the network connections gets behind, problems start. When the browser next reads from that file, the browser stops running while the read() blocks, waiting for data to arrive. Needless to say, this is not the behavior the browser's user would prefer. [17]

The motivation to study this approach is that many servers are designed using the thread-pooling model because of its simplicity. Using thread pooling, one file descriptor is added to a worker thread for the lifetime of the connection. Using a single connection per thread lets the data buffer to be local on the thread stack, simplifying buffer and state management. This reduces the development time needed to bring servers to market. Unfortunately, there are three drawbacks to thread pools:

• The number of threads an operating system can create per process.
• The time it takes to switch between threads (context switching).
• Worker threads are I/O bound. 

These three drawbacks, which are limit the number of concurrent connections a socket server can handle using the thread-pooling model are detailed by Ian Barile in [5].

A solution to all these problems is the I/O multiplexing. Multiplexed I/O allows an application to concurrently block on multiple file descriptors, and receive notification when any one of them becomes ready to read or write without blocking.

I/O multiplexing is typically used in networking applications in the following scenarios:

• When a client is handling multiple descriptors (interactive input and a network socket)
• When a client handle multiple sockets at the same time (for example a Web client)
• When a TCP server handles both a listening socket and its connected sockets
• When a server handles both TCP and UDP protocols
• When a server handles multiple services and multiple protocols

When they first appear in the BSD systems, the I/O multiplexing calls like select and poll does not scale well with an increasing number of descriptors. In addition, these interfaces were also limited in the respect that they were unable to handle other potentially interesting activities that an application might be interested in, these might include signals, file system changes, and AIO completions. Today, after many researches, all-major OS includes facilities that solve all these problems, for example, epoll on Linux, kqueue on BSD clones and /dev/poll on Solaris based systems.

I/O models comparative presentation

Under UNIX based systems there are five I/O models as follow:

• blocking I/O
• nonblocking I/O
• I/O multiplexing (select, poll, epoll, /dev/poll, kqueue)
• signal driven I/O (SIGIO)
• asynchronous I/O (the Posix.1 aio_ functions)

The blocking I/O model, block the progress of a program while the communication is in progress, leaving system resources idle. When a program performs many I/O operations, the processor can spend almost all of its time idle waiting for I/O operations to complete.

When we use a socket in nonblocking I/O model, the kernel will not put the process to sleep when an I/O operation cannot be completed, will return an error instead. When an application sits in a loop calling recvfrom() or read() system call on a nonblocking descriptor, it is called polling. The application is continually polling the kernel to see if some operation is ready and this is often a waste of CPU time.

With I/O multiplexing, we call select, poll (or any other event notification mechanism) and block in one of these system calls, instead of blocking in the actual I/O system call. We block in a call to our event notification mechanism, waiting for the datagram socket to be readable. When our call returns means that the socket is readable, and we can read without blocking. At first sight, there is a slight disadvantage because using these event notification mechanisms requires two system calls instead of one. However, the advantage in using these is that we can wait for more than one descriptor to be ready.

Signal driven I/O (SIGIO), telling the kernel to notify a process with the SIGIO signal when the descriptor is ready. In order to use this method we enable the socket for signal-driven I/O and install a signal handler using the sigaction system call. The return from this system call is immediate and our process continues without blocking. Regardless of how we handle the signal, the advantage to this model is that we are not blocked while waiting for the datagram to arrive. The main loop can continue executing and just wait to be notified by the signal handler that either the data is ready to process or the datagram is ready to be read.

Asynchronous I/O functions work by telling the kernel to start the operation and to notify us when the entire operation (including the copy of the data from the kernel to our buffer) is complete. The main difference between this model and the signal-driven I/O model in the previous section is that with signal-driven I/O, the kernel tells us when an I/O operation can be initiated, but with asynchronous I/O, the kernel tells us when an I/O operation is complete.

Richard Stevens detail these five models with their advantages and disadvantages in [1].

Synchronous and Asynchronous I/O

POSIX defines these two terms as follows:

• A synchronous I/O operation causes the requesting process to be blocked until that I/O operation completes. In synchronous file I/O, a thread starts an I/O operation and immediately enters a wait state until the I/O request has completed.[11]
• An asynchronous I/O operation does not cause the requesting process to be blocked. A thread performing asynchronous file I/O sends an I/O request to the kernel by calling an appropriate function. If the request is accepted by the kernel, the calling thread continues processing another job until the kernel signals to the thread that the I/O operation is complete. It then interrupts its current job and processes the data from the I/O operation as necessary.[11]

Using these definitions, the first four I/O models presented (blocking, nonblocking, I/O multiplexing, and signal-driven I/O) are synchronous because the actual I/O operation blocks the process. Only the asynchronous I/O model matches the asynchronous I/O definition.

In situations where an I/O request is expected to take a large amount of time, such as a refresh or backup of a large database or a slow communications link, asynchronous I/O is generally a good way to optimize processing efficiency. However, for relatively fast I/O operations, the overhead of processing kernel I/O requests and kernel signals may make asynchronous I/O less beneficial, particularly if many fast I/O operations need to be made. In this case, synchronous I/O would be better.

A detailed presentation and comparison of these two I/O synchronization models is described in [11]. These two synchronization types are illustrated in the (Figure 1).

Comparison of the I/O Models

In (Figure 2) is a comparison of the five different I/O models. It shows that the main difference between the first four models is the first phase, as the second phase in the first four models is the same: the process is blocked in a call to recvfrom or read while the data is copied from the kernel to the caller's buffer. Asynchronous I/O, however, handles both phases and is different from the first four.

I/O event notification mechanisms presentation

Multiplexing using select system call

A call to select() will block until the given file descriptors are ready to perform I/O, or until an optionally specified timeout has elapsed. The watched file descriptors are broken into three sets, each waiting for a different event (read, write, exception).

In order to use the select() system call the following headers are mandatory:

#include <sys/time.h>
#include <sys/types.h>
#include <unistd.h>

The necessary functions and macros are defined as follows:

int select (int n, fd_set *readfds, fd_set *writefds, fd_set *exceptfds, struct timeval *timeout);

FD_ZERO(fd_set *set);
FD_SET(int fd, fd_set *set);
FD_CLR(int fd, fd_set *set);
FD_ISSET(int fd, fd_set *set);

File descriptors listed in the readfds set are watched to see if a read operation will complete without blocking. File descriptors listed in the writefds set are watched to see if a write operation will complete without blocking. Finally, file descriptors in the exceptfds set are watched to see if an exception has occurred, or if out-of-band data is available (only for sockets). A given set may be NULL, in which case select() does not watch for that event. [3]

On successful return, each set is modified and will contains only the file descriptors that are ready for I/O. (Figure 3) present a summary of conditions that cause a socket to be ready for select.

The first parameter, n, is equal to the value of the highest-valued file descriptor in any set, plus one. Consequently, a caller to select() is responsible for checking which given file descriptor is the highest-valued, and passing in that value plus one for the first parameter.

The timeout parameter is a pointer to a timeval structure, if this parameter is not NULL, the call to select() will return after a time specified in this structure, even if no file descriptors are ready for I/O. In case the timeval is set to zero, the call will return immediately, reporting any events that were pending at the time of the call, but not waiting for any subsequent events.

The sets of file descriptors are not manipulated directly, they are managed through the 4 helper macros presented above:

• FD_ZERO - removes all file descriptors from the specified set. It should be called before every invocation of select( )
• FD_SET - adds a file descriptor to a given set
• FD_CLR - removes a file descriptor from a given set
• FD_ISSET - tests whether a file descriptor is part of a given set. It returns a nonzero integer if the file descriptor is in the set, and 0 if it is not.

The most important limitation of select system call refers to the number of descriptors that can be watched. This number is limited to FD_SETSIZE (some operating systems define FD_SETSIZE as 32 other as 1024). The only way to increase the size of the descriptor sets is to increase the value of FD_SETSIZE (from <sys/types. h>) and then recompile the kernel. Changing the value without recompiling the kernel is inadequate.[1]

Multiplexing using poll system call

The poll() function derived from SVR3 (one of the first commercial versions of the UNIX operating system) and was originally limited to streams devices. SVR4 removed this limitation, allowing poll() to work with any descriptor. poll() provides functionality that solves several deficiencies of select system call.
In order to use the poll() system call the following header is mandatory:

#include <sys/poll.h>

The poll() system call is defined as follows:

int poll (struct pollfd *fds, unsigned int nfds, int timeout);

Unlike select(), with its inefficient three bitmask-based sets of file descriptors, poll() involves a single array of nfds pollfd structures, pointed to by fds. The structure is defined as follows:

struct pollfd
{
    int fd; 	     /* file descriptor */
    short events;    /* requested events to watch */
    short revents;   /* returned events witnessed */
};

Each pollfd structure specifies a single file descriptor to watch. Multiple structures may be passed, instructing poll() to watch multiple file descriptors. The events field of each structure is a bitmask of events to watch for on that file descriptor. The user sets this field. The revents field is a bitmask of events that were witnessed on the file descriptor. The kernel sets this field on return. All of the events requested in the events field may be returned in the revents field. With poll(), unlike select(), you need not explicitly ask for reporting of exceptions.

All available events and revents are described in (Figure 4).

The timeout parameter specifies the length of time to wait, in milliseconds, before returning regardless of any ready I/O. A negative value denotes an infinite timeout. A value of 0 instructs the call to return immediately, listing any file descriptors with pending ready I/O, but not waiting for any further events. In this manner, poll() is true to its name, polling once, and immediately returning.[4]

On success, poll() returns the number of file descriptors whose structures have nonzero revents fields. It returns 0 if the timeout occurred before any events occurred. On failure, -1 is returned, and errno is set to a specific error code. [2]

The number of file descriptors that can be watched is limited to maximum file descriptor number that can be opened by a process. This number can be changed using the setrlimit UNIX command.

Despite the fact they perform the same job, the poll() system call is superior to select() for a handful of reasons:

• poll() does not require that the user calculate and pass in as a parameter the value of the highest-numbered file descriptor plus one.
• poll() is more efficient for large-valued file descriptors. Imagine watching a single file descriptor with the value 700 via select() - the kernel would have to check each bit of each passed-in set, up to the 700th bit.
• select()'s file descriptor sets are statically sized, introducing a tradeoff: they are small, limiting the maximum file descriptor that select() can watch, or they are inefficient.
• with select(), the file descriptor sets are reconstructed on return, so each subsequent call must reinitialize them. The poll() system call separates the input (events field) from the output (revents field), allowing the array to be reused without change.
• The timeout parameter to select() is undefined on return. Portable code needs to reinitialize it.

Multiplexing using epoll interface

Recognizing the limitations of both poll()select() and select(), the 2.6 Linux kernel introduced the event poll (epoll) facility. This new event notification mechanism solves the fundamental performance problem shared by both of them, and adds several new features.

Unlike these earlier system calls, which are O(n), epoll is an O(1) algorithm. This means that it scales well as the number of watched file descriptors increase. select() uses a linear search through the list of watched file descriptors, which causes its O(n) behavior.[9]

Epoll fix this problem by decoupling the monitor registration from the actual monitoring. One system call initializes an epoll context, another adds monitored file descriptors to or removes them from the context, and a third performs the actual event wait.

Another fundamental difference of epoll is that it can be used in an edge-triggered, as opposed to level-triggered, fashion. This means that you receive "hints" when the kernel believes the file descriptor has become ready for I/O. This has a couple of minor advantages: kernel space does not need to keep track of the state of the file descriptor, although it might just push that problem into user space, and user space programs can be more flexible. More details about this feature are described in [15].

In order to use the epoll system call the following header file is mandatory:

#include <sys/epoll.h>

To use epoll method an application should create an epoll context using epoll_create().

int epoll_create (int size)

A successful call to epoll_create()instantiates a new epoll instance, and returns a file descriptor associated with the instance. This file descriptor it is just a handle to be used with subsequent calls using the epoll facility (has no relationship to a real file).

The size parameter is a hint to the kernel about the number of file descriptors that are going to be monitored(it is not the maximum number). Passing in a good approximation will result in better performance, but the exact number is not required. On error, the call returns -1, and sets errno to the occurred error.

The file descriptor returned from epoll_create() should be destroyed via a call to close() after polling is finished.

After creating an epoll context an application is able to add or remove file descriptors using epoll_ctl() which is defined as follow:

int epoll_ctl (int epfd, int op, int fd, struct epoll_event *event);

A successful call to epoll_ctl() controls the epoll instance associated with the file descriptor epfd. The parameter op specifies the operation to be taken against the file associated with fd. The event parameter further describes the behavior of the operation.
The valid values for the second parameter, op, is one of the following:

• EPOLL_CTL_ADD - The file descriptor fd is added to the file descriptor set with the event set events. If the file descriptor is already present, it returns EEXIST.
• EPOLL_CTL_DEL - The file descriptor fd is removed from the set of file descriptors that is being monitored. The events parameter must point to a struct epoll_event, but the contents of that structure is ignored.
• EPOLL_CTL_MOD - The struct epoll_event for fd is updated from the information pointed to by events. This allows the set of events being monitored and the data element associated with the file descriptor to be updated without introducing any race conditions.

The valid values for event parameter are described in (Figure 5) .

The system call epoll_wait() waits for events on the file descriptors associated with the given epoll instance:

int epoll_wait (int epfd, struct epoll_event *events, int maxevents, int timeout);

A call to epoll_wait() waits up to timeout milliseconds for events on the files associated with the epoll instance epfd. Upon success, events points to memory containing epoll_event structures describing each event, up to a maximum of maxevents events. The return value is the number of events, or -1 on error, in which case errno is set to the occurred error. If timeout is 0, the call returns immediately, even if no events are available, in which case the call will return 0. If the timeout is -1, the call will not return until an event is available.

When the call returns, the events field of the epoll_event structure describes the events that occurred. The returned events are presented in (Figure 5).

Multiplexing using /dev/poll interface

Solaris provides a special file called /dev/poll, which provides a more scalable way to poll large numbers of file descriptors. The /dev/poll device maintains state between calls so that a program can set up the list of descriptors to poll and then loop, waiting for events, without setting up the list again each time around the loop.

After opening /dev/poll, the polling program must initialize an array of pollfd structures (the same structure used by the poll function, but the revents field is unused in this case). The array is then passed to the kernel by calling write to write the structured directly to the /dev/poll device. The program then uses an ioctl call, DO_POLL, to block, waiting for events. The following structure is passed into the ioctl call:

struct dvpoll 
{
   struct pollfd* dp_fds;
   int dp_nfds;
   int dp_timeout;
}

The field dp_fds points to a buffer that is used to hold an array of pollfd structures returned from the ioctl call. The field dp_nfds field specifies the size of the buffer. The ioctl call blocks until there are interesting events on any of the polled file descriptors, or until dp_timeout milliseconds have passed. Using a value of zero for dp_timeout will cause the ioctl to return immediately, which provides a nonblocking way to use this interface. Passing in -1 for the timeout indicates that no timeout is desired [2 ][14].

Various implementations of /dev/poll were tried on Linux, but none of them performs as well as epoll, and were never really completed for that reason /dev/poll use on Linux is not recommended and was not adopted in the Linux kernel.

In [8] Niels Provos and Chuck Lever describe a version of thttpd web server, modified to support /dev/poll. Performance is compared with phhttpd using POSIX RT signal.

Multiplexing using kqueue interface

FreeBSD introduced the kqueue interface in FreeBSD version 4.1. In "Kqueue: A generic and scalable event notification facility" [7], Jonathan Lemon describes the design of this new event notification mechanism.

The kqueue event mechanism addressed many of the deficiencies of select and poll for FreeBSD systems. In addition to separating the declaration of interest from retrieval, kqueue allows an application to retrieve events from a variety of sources including file/socket descriptors, signals, AIO completions, file system changes, and changes in process state.[12]

In order to use the kqueue interface, the following headers are mandatory:

#include <sys/types.h> 
#include <sys/event.h> 
#include <sys/time.h>

The kqueue interface includes the following two functions and macro:

int kqueue(void); 

int kevent(int kq, const struct kevent *changelist, 
           int nchanges, struct kevent *eventlist, 
           int nevents, const struct timespec *timeout) ; 

void EV_SET(struct kevent *kev, uintptr_t ident, short filter, 
            u_short flags, u_int fflags, intptr_t data, void *udata);

The kqueue function returns a new kqueue descriptor, which can be used with future calls to kevent.

The kevent function is used to both register events of interest and determine if any events have occurred. The changelist and nchanges parameters describe the changes to be made to the events of interest, or are NULL and 0, respectively, if no changes are to be made. If nchanges is nonzero, each event filter change requested in the changelist array is performed.

Any filters whose conditions have triggered, including those that may have just been added in the changelist, are returned through the eventlist parameter, which points to an array of nevents struct kevents. The kevent function returns the number of events that are returned, or zero if a timeout has occurred.

The timeout argument holds the timeout, which is handled just like select:

• NULL to block
• a nonzero timespec to specify an explicit timeout
• a zero timespec to perform a nonblocking check for events.

Note that the timeout parameter is a struct timespec, which is different from select's struct timeval in that it has nanosecond instead of microsecond resolution. [16]

Portability of I/O event notification mechanisms

The portability of these event notification mechanisms is presented in (Figure 6). As we can observe select and poll are portable. Epoll , /dev/poll and kqueue are platform based mechanisms which came to solve the issues of select and poll.

Performance measurements and analysis

To measure the performances we have used the "Litevent framework" [6] (described in detail in the following chapter). This framework abstracts these mechanisms into a common interface, and provides both: high performance and portability.

The benchmark program is very simple: first a number of socket pairs are created, then event watchers for those pairs are installed and then a (smaller) number of "active clients" send and receive data on a subset of those sockets.

Each run of the benchmark program consists of two iterations outputting the total time per iteration as well as the time used in handling the requests only. Each run is composed of 10 individual runs of run_once() function listed hereinafter, and the result used is the minimum time from these runs.

static int count, writes, fired; static int *pipes; static int num_pipes, num_active, num_writes; CFdEventMonitor *evMonitor = NULL; void read_cb(int fd, void *ctx, int revents) //callback function { int idx = (int) ctx, widx = idx + 1; u_char ch; count += read(fd, &ch, sizeof (ch)); if (writes) { if (widx >= num_pipes) widx -= num_pipes; write(pipes[2 * widx + 1], "e", 1); writes--; fired++; } } long run_once() { static struct timeval ta, ts, te; gettimeofday(&ta, NULL); int *cp, i, space; for (cp = pipes, i = 0; i < num_pipes; i++, cp += 2) { evMonitor->RemoveEventWatcher(cp[0]); evMonitor->SetEventWatcher(cp[0], read_cb, (void*) i, FDEVENT_IN); } evMonitor->Loop(EVENT_LOOP_ONCE | EVENT_LOOP_NONBLOCK); fired = 0; space = num_pipes / num_active; space = space * 2; for (i = 0; i < num_active; i++, fired++) write(pipes[i * space + 1], "e", 1); count = 0; writes = num_writes; int xcount = 0; gettimeofday(&ts, NULL); do { evMonitor->Loop(EVENT_LOOP_ONCE | EVENT_LOOP_NONBLOCK); xcount++; } while (count != fired); gettimeofday(&te, NULL); long elapsed_all = timerElapsedMicroseconds(te, ta); long elapsed_poll = timerElapsedMicroseconds(te, ts); fprintf(stdout, "%8ld %8ld\n", elapsed_all , elapsed_poll); return elapsed_all; }

To assure the quality of measurements the following six tests were run on the same machine, a Core2 Duo E8400 at 3.0 GHz. All applications installed on the specific platform were stopped to avoid as possible the alteration of measurements.

The tests were performing for one active connection on different number of file descriptors (1, 50, 100, 200, 300, 400 and 500). (Table 1),( Table 3) and (Table 5) measure the total time spent (in microseconds) to perform a complete iteration. (Table 2),( Table 4) and (Table 6) measure the time spent in event processing (microseconds) .

As we can observe from the above tests select and poll performance in event processing becomes poor (on all analyzed operating systems) if the number of file descriptors increase. This think is not true for epoll, /dev/poll or kqueue which are O(1) algorithms. These scales well as the number of watched file descriptors increase.

Analyzing the aggregate indicator (the medium value of measurements) results the following things:

1. Poll performs better than select on all analyzed platforms
2. Kqueue on FreeBSD performs better than epoll on Linux 2.6 or /dev/poll on OpenSolaris 2009.06
3. The implementation of poll on OpenSolaris 2009.06 performs better than the implementation of the same interface on the other analyzed OS.
4. The implementation of select on OpenSolaris 2009.06 performs better than the implementation of the same interface on the other analyzed OS.

Another performance comparison using these event notification mechanisms (except /dev/poll) was performed by Nick Mathewson and Niels Provos [18] using the Libevent framework. They declare interest to measure the performance using a large number of connections of which most are cold and only a few are active.

The (Figure 7) measures how long it takes to serve one active connection and exposes capability issues of traditional interfaces like select or poll.

Another benchmark using Libevent presented in (Figure 8) measures how long it takes to serve one hundred active connections that chain writes to new connections until thousand writes and reads have happened.

The above two graphics use a logarithmic x-axis to sensibly display the large range of file descriptor number.

Related Work

This section presents "Litevent" [6], a portable framework based on these event notification mechanisms, analyzed in this paper, which allows an application to be notified when an event occurs on a file descriptor, in a scalable and efficient manner.

Litevent is meant to replace the event loop found in event driven network servers. An application just needs to call Loop method from the CFdEventMonitor class and then add or remove events dynamically without having to change the event loop. (Figure 9) presents the framework diagram.

The internal event mechanism is completely independent of the exposed event API, and a simple update of "Litevent" can provide new functionality without having to redesign the applications.

In order to use "Litevent" you should create an instance of CFdEventMonitor as follow:

CFdEventMonitor *evMonitor = new CFdEventMonitor(EnumEventHandlerTypeSelect);

The CFdEventMonitor constructor receive a parameter which indicates what event notification mechanism do you want to use. In case this parameter is equal with EnumEventHandlerTypeUnset the framework will use the best one available in your operating system.

To add an event, an application should call one of the SetEventWatcher form. This method is overload to allow applications to register as callback, class members or independent functions. For example to register a function as a callback, you should call this method as follow:

evMonitor->SetEventWatcher(cp[0], read_cb, (void*) i, FDEVENT_IN);

To remove an event an application should call RemoveEventWatcher method as follow:

evMonitor->RemoveEventWatcher(cp[0]); where the specified parameter is the file descriptor that is removed.

To check if any event occurs on the registered file descriptors the application should call the Loop function as follow:

evMonitor->Loop(EVENT_LOOP_ONCE | EVENT_LOOP_NONBLOCK);

The specified argument means that the application will loop once, and will return immediatly, even if no events are available. In case an event has occurred, the callback function registered on that file descriptor is automatically called.

As a result, Litevent allows for portable application development and provides the most scalable event notification mechanism available on an operating system. Litevent should compile on Linux, BSD, Mac OS X and Solaris. If it is compiled using HAVE_MULTITHREAD_LITEVENT argument it can be used in multithreaded applications.

Conclusion

Event notification mechanisms have a long history in operating systems research and development, and have been a central issue in many performance studies [7][8][12][13] because applications handling a large number of events are dependent on the efficiency of event notification and delivery. Much of the aforementioned work is tracked and discussed on the web site, "The C10K Problem" [10].

This paper has presented five of the most important event notification mechanisms available on the UNIX based operating systems.

As we had observed from our measurements the select() and poll() system calls performance becomes poor if the number of file descriptors increase. This think is not true for epoll, /dev/poll or kqueue which are O(1) algorithms - this means that it scales well as the number of watched file descriptors increase.

References

[1] Richard W. Stevens, UNIX Network Programming, Volume 1, Second Edition: Networking APIs: Sockets and XTI, Prentice Hall, New Jersey, pp.143-175, 1998

[2] Richard W. Stevens, Stephen A. Rago, Advanced Programming in the UNIX Environment: Second Edition , Addison-Wesley Professional, New Jersey, pp.394-402, 2005

[3] Robert Love, Linux System Programming, O'Reilly Media, California, pp.47-57, 2007

[4] Brian "Beej Jorgensen" Hall, Beej's Guide to Network Programming Using Internet Sockets, 2009, Accessed at: 11/10/2010

[5] Ian Barile, I/O Multiplexing & Scalable Socket Servers, 2004, Accessed at: 10/11/2010

[6] Silviu Caragea, litevent, 2010, Accessed at: 11/14/2010

[7] Jonathan Lemon, Kqueue: A generic and scalable event notification facility, 2001 USENIX Annual Technical Conference, June 25-30, 2001, Boston, Massachusetts, USA, USENIX Association, Proceedings of the FREENIX Track: 2001 USENIX Annual Technical Conference, USENIX Association, Califonia, pp.141-153, 2001

[8] Niels Provos, Chuck Lever, Scalable Network I/O in Linux, 2000 USENIX Annual Technical Conference, June 18-23, 2000, San Diego, California, USA, USENIX Association, Proceedings of the FREENIX Track: 2000 USENIX Annual Technical Conference, USENIX Association, Califonia, pp.109-120, 2000

[9] Alexey Kovyrin, Using epoll() For Asynchronous Network Programming, 2006, Accessed at:11/17/2010

[10] Dan Kegel, The C10K problem, 2006, Accessed at:11/17/2010

[11] Microsoft Developer Network (MSDN), Synchronous and Asynchronous I/O, 2010, Accessed at:11/17/2010

[12] Louay Gammo, Tim Brecht, Amol Shukla, David Pariag, Comparing and Evaluating epoll, select, and poll Event Mechanisms, Linux Symposium 2004, July 21st-24th, 2004, Ottawa, Ontario Canada, Linux Symposium, Proceedings of the Linux Symposium Volume One, Linux Symposium, Ottawa, Ontario Canada, pp.215-226, 2004

[13] Michal Ostrowski, A Mechanism for Scalable Event Notification and Delivery in Linux, 2000, Accessed at:11/17/2010

[14] Bruce Chapman, Polling Made Efficient in the Solaris 7 OS, 2002, Accessed at:11/17/2010

[15] Linux Kernel Organization, Linux Programmer's Manual - epoll(7), 2010, Accessed at:11/17/2010

[16] Jonathan Lemon, FreeBSD System Calls Manual - KQUEUE(2), 2009, Accessed at:11/16/2010

[17] Michael K. Johnson, Erik W. Troan, Linux application development, Addison-Wesley Professional, New Jersey, pp.264-269, 2004

[18] Nick Mathewson, Niels Provos, Libevent, 2004, Accessed at: 22/1/2011

License

This article, along with any associated source code and files, is licensed under The Intelliproject Open License (IPOL)

About the author