First incarnation of LogStore

I've disappeared from this blog in the recent month but I've not been idle: I've implemented initial support for LogStore in the Premium Edition of syslog-ng.

LogStore is a binary log file format, in semantics very similar to a plain log file. But the format allows much more:

• on-line compression via gzip,
  
• encryption via AES and X.509 certificates,
  
• integrity protection via hmac-sha1.
  

And furthermore: it is indexed based on time, and it is quite efficient to look for a specific time range in GBs of log data. I'm quite satisfied, although there are some more work left to be done, for instance the query interface for the time based indexing is not completed.

In use it is quite simple: replace the "file" destination with "logstore" and you are done. More or less the same amount of options are supported: macro based file names, template based formatting, etc.

I'm still pondering with the idea of storing the complete internal representation of the logrecord in serialized form, so it'd be possible to perform template() based formatting in off-line mode.

This code will be released as an experimental part of syslog-ng PE 2.1 and will be finalized in syslog-ng PE 2.2.

libdbi patches online

I've published our set of dbi and dbi-drivers patches in a git repository to push changes upstream. The patches were updated against the latest libdbi versions.

You can find these repositories at the BalaBit's git server, more precisely:

• git://git.balabit.hu/bazsi/libdbi.git
• git://git.balabit.hu/bazsi/libdbi-drivers.git

The "master" branch contains the direct import of the libdbi CVS tree, our fixes are in the 'upstream-fixes' branch. This setup will make it easier for me to publish patches and regularly rebase the not-yet-merged set against the latest upstream.

Among other small things, you can find a quite important patch against the Oracle driver. Without this patch Oracle 10.2 (the server!) segfaults and dumps core. So beware.

syslog-ng feature sheet

We were asked to publish some more detailed "syslog-ng feature sheet". Albeit it might go into syslog-ng specific details we tried to be as generic as possible. And certainly everyone doing such feature sheets is biased, just as we were :)

It is available at http://www.balabit.com/network-security/syslog-ng/features/detailed/.

syslog-ng OSE 2.1 released

I have just uploaded the first release in the syslog-ng Open Source Edition 2.1 branch to our website. It is currently only available in source format at this location:

http://www.balabit.com/downloads/files/syslog-ng/sources/2.1/src

This release synchronizes the core of syslog-ng to the latest PE version and adds the SQL destination driver.

This is an alpha release and thus might be rough around the edges, but it basically only contains code already tested in the context of the Premium Edition. The SQL functionality requires a patched libdbi package, which is available at the same link. We're going to work on integrating all our libdbi related patches to the upstream package.

If you want to know how the SQL logging works, please see the Administrator's Guide or our latest white paper Collecting syslog messages into an SQL database with syslog-ng. The latter describes the Premium Edition, but it applies to the Open Source one equally well.