Bazsi's blog

Although I was not posting on this blog, I was working on syslog-ng multi-thread support in the last couple of weeks. Most of the preparation was done during the Netfilter Workshop (I know it wasn’t netfilter related and I’ve since used up any possible occassions to work on the code instead of writing about it.

I’ve decided that instead of using a per-connection thread model, I’d like to use something that’d keep the number of threads close to the number of CPU cores to avoid bad cache effects and context-switch overhead. Since syslog-ng may serve thousands of clients, the per-connection thread model would have meant that we’d have thousands of threads, so I gave up on that thought.

In my current architecture a single thread would be watching the file descriptors for events and a set of worker threads would perform the work. Since most of the time, most of the fds are idle, this would definitely use a lower number of threads and if I’m smart enough the same thread would be dispatching for the same client, which means that our cache would be hot by the time the 2nd round of events are coming in.

Also, since the model of GLib’s main loop is inherently slow, I’ve decided to switch away from it. GLib uses a linked list of GSource objects, which are iterated _twice_ every poll iteration, once for the prepare() phase, and once for check(). In case we’re polling 1000 fds, that’s a loop over 2000 items, which, if done thousands of times a second, poses a serious overhead.

Getting away of GLib is not easy, since a lot of logic is implemented in those prepare/check callbacks, but anyway it was an aim worth pursuing.

Also, as an added bonus I wanted to use faster kernel interfaces instead of poll. Linux has epoll, FreeBSD has a kqueue based implementation, Solaris has /dev/poll, these all advertise themselves as much more performant than the traditional interfaces.

I was both looking at alternative main loop libraries and also thinking about rolling my own. Here are the ones I’ve considered:

• I’ve immediately closed out C++ libraries.
• libevent: probably the most widespread availability, but I didn’t like the API too much.
• libev: has a libevent compatible API and a lower level one. This API was better, but reading through the CVS history, I didn’t like the stance of the primary author to things like AIX support and the like.
• ivykis: has the best API, quite Linux kernel-like, lightweight enough for me to confidently navigate in the code or to modify it. Has nice thread integration, with worker thread polls and cross-thread calls. Not really available anywhere, so syslog-ng will probably have to carry a copy.

Right now, ivykis is the winner, but I don’t know about its real portability (uses __thread keywords for example, which may not be available everywhere). So changing this still has a chance.

I’ve quickly gave up the idea of rolling my own similar implementation, doing this for all the various kernel interfaces correctly would be too much work.

You can find the latest version of this work on git.balabit.hu. git://git.balabit.hu/bazsi/ivykis.git is my modified version of ivykis, and the wip/epoll branch of syslog-ng-3.3.git repository.

Even with a single client and one destination without using threads, I could measure about 10% (55k msg/sec -> 60k msg/sec) performance increase on my development laptop. This is certainly promising.