syslog-ng rewrite use case: dpkg logs

One of my collegues (Péter Höltzl, he does all our trainings) has created a nice detailed example on how to use the parser/rewrite framework to pull in yet another application into syslog: dpkg, the Debian package manager.

If you are interested in what rewrite/parser can do for you, but didn't have the time to find out, the blog post is worth a read.

syslog-ng pipelines

The other day someone wanted a special syslog-ng macro that would expand into digit changing every 5 seconds (e.g. R_UNIXTIME % 5) and although I couldn't give an exact solution to his problem, I've came up with this configuration snippet:

rewrite p_date_to_values {
set("$R_DATE", value("rdate"));
};

filter f_get_second_chunk {
match('^... .. [0-9]+:[0-9]+:(?<rdate.second_tens>[0-9])[0-9]$'
type(pcre) value('rdate'));
};

The way it works is as follows:

• the rewrite statement sets the name-value pair named "rdate" to $R_DATE (the macro)
• the filter statement uses Perl Compatible Regular Expressions to parse the value of the "rdate" value and uses a named subpattern on the tens of seconds position to store that character in a value named "rdate.second_tens"
• Later on in the configuration you can use "rdate.second_tens" just like any other macro/value.

This proves that the current rewrite/parser/filter subsystems are really powerful, however even though this proved to be possible, there are some lessons learned from this example:

• the macro and name-value space should really converge to each, this would mean that the match() filter could directly match against the macro value $R_DATE without the need for the separate rewrite statement
• when you are after a given goal, you don't really want to differentiate rewrite/parser/filter rules at all. The current syntax of using separate blocks for separate type of log processing elements is a pain.
  

So I'm thinking about inventing yet another block, which simply wouldn't care what kind of processing element is added to it, something along the lines:

pipeline rdateseconds {
set("$R_DATE", value("rdate"));
match('^... .. [0-9]+:[0-9]+:(?[0-9])[0-9]$'
type(pcre) value('rdate'));

};

And then:

log {
source(src);
pipeline(rdateseconds);
destination(dst);
};

Maybe I should even allow the creation of rewrite/parser/filter elements right there in the log statement:

log {
source(src);
filter(facility(mail));
destination(dst);
};

What do you think?

Nordic Meet on Nagios 2009

I'm sitting at NMN 2009 right now, and although the event title says it is a Nagios meet, I'm going to give a presentation on syslog-ng and the new features that 3.0 brings and an example on how to integrate syslog-ng and Nagios.

If you are here and have a question just feel free to find me in the "BalaBit" T-Shirt. :) There's also live streaming on the conference website, so you can catch me at 15:50 Central European Time.

syslog-ng 4.0 roadmap plus release policy changes

I've updated the syslog-ng OSE roadmap on the syslog-ng webpage to include information about the upcoming syslog-ng version:

http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/roadmap/

Also, I'd like to bring the changed release/support policy to your attention, that you can read at the same location above. I'd like to introduce stable track and feature track releases, the first being supported for a long time, whereas feature track releases are only supported until the next feature/stable release is published. When a sufficient number of features were published via feature track releases, the last one becomes stable and the cycle continues. Note that feature releases are NOT development snapshots, they are releases just like the major versions previously, the only difference is that instead of a large feature list like with syslog-ng 3.0, only a smaller set of changes are included.

This makes it possible to publish features more often, always concentrating on a few of them at a time, instead of doing development for a long time and come out with a feature packed release. I hope to increase the pace of syslog-ng development with this change and also to cause less problem for users who prefer stability over features. Please read the details on the roadmap page.

I've also opened the syslog-ng 3.1 repository and pushed it to our git server. Right now there are no differences (except for the version number) between 3.0 and 3.1, I'm planning to integrate Marton's message tagging and patterndb changes as soon as possible (his git tree is here). Hopefully the 3.1 cycle will be quite short as most of the things on the roadmap are already implemented, although scattered around in various public and private trees.

With the opening of the 3.1 branch, I'm also obsoleting 2.0 (in the new support model two stable track versions are supported at any given time and we have 2.0, 2.1 and 3.0 right now), but that'll go in a separate post/announcement.

syslog-ng OSE 3.0.2 released

After a long time and a lot of accumulated bugfixes, I've pressed the "release" button and syslog-ng OSE 3.0.2 was published on our website. The first official version to feature binary packages for Linux and BSD platforms. Since there was a long time between 3.0.1 and 3.0.2 the changelog is quite large, however most of it are bugfixes, only some minor enhancements here and there.

Hopefully I didn't miss any important bugs and problems. It must be much better stability/functionality wise than 3.0.1 was.

The diffstat since 3.0.1:
150 files changed, 4332 insertions(+), 3000 deletions(-)

You can also check the patches in our git repository.

If you are using the 3.0 branch you are really recommended to check out this release. If you are using anything earlier than 3.0 you are also recommended upgrade, syslog-ng 3.0 is revolutionary to previous versions in many ways, especially if you want to do more to your logs than merely store them in a plain text file.

OSDC 2009 slides

I've uploaded my OSDC 2009 presentation slides to
http://people.balabit.hu/bazsi/slides/osdc-2009-syslog-ng-3.0.odp Which has an example for processing iptables logs with db-parser() and putting the results in a customized SQL table.

Nordic Nagios Meet 2009

I'm going to give a talk on syslog-ng on the upcoming Nordic Nagios Meet 2009. I expect the event to be great fun, just like last year. If you are in the Nordic region and use Nagios, rrdtools or syslog-ng, I recommend to pay a visit as you can meet the primary authors and some active contributors to these projects.

If you are there and have anything to ask/talk about syslog-ng, just feel free to approach me, I'm probably going to wear a badge, so you can recognize me :)

OSDC 2009 and syslog-ng automatic testing

I've spent the last week in the nice city of Nuremberg where Open Source Data Center Conference took place, organized by Netways AG. I really liked the talks about Puppet, DRBD and the description of the booking.com infrastructure which runs MySQL.

Although I really enjoyed the conference I also had some free time to improve the automatic test program for syslog-ng, which now also covers TLS encrypted source and SQL destinations. I've also implemented a small script to collect coverage data of the testcases, thus right now I know that about 63% of syslog-ng is covered by automatic tests. (initially it was 55% but there were some low hanging fruits). I expect to raise this number easily to around 80%, then it'll probably become much more difficult to increase it further as the rest is error processing paths, and unless I come up with something to inject errors from the testcases those are difficult to test.

Of course having a test suite is not a replacement for real-life, field testing, but nevertheless it makes it much easier to do releases as it ensures that no important functionality is broken completely.

Based on this test infrastructure I'm going to release 3.0.2, after which I'll probably change the way I manage releases for syslog-ng, but I'll talk about that in a forthcoming post.

My son is 7 weeks old

From Dani-aprilis-25

The reason I was absent from this blog in the couple of last weeks is my now 7 weeks old son, Dániel. You can find a picture of him right here in the post, but some additional ones in my Picasa albums.

Features that fell off the radar

I was long pondering with the problem that it is quite tricky to enter regexps into syslog-ng configuration file, since if you enclose the string in double quotes (e.g. in ""), the backslash character needs to be escaped.

Since backslash is used in regexps quite often, it can become cumbersome to enter regexps like:

match("[a-z\\-]+");

Note that the backslash is doubled because otherwise the syslog-ng string parser would pass the sequence to the regexps compiler as: "[a-z-]+" which is certainly different in meaning what the above expression says.

I always remembered that syslog-ng also supports single quotes (aka apostrophes), but I remembered they behaved just as if you used normal quotation marks. Therefore I was thinking about a 3rd string format, one that would not require escaping.

However I was reading the related code the other day, and found that apostrophes work exactly the way I planned this 3rd string syntax to behave: not to get in the way when entering regexps. In fact it behaves just like apostrophes in the UNIX shells. It does not care about escaping, it only cares about the terminating apostrophe.

I was dealing with regexp related questions on the mailing list a lot, and the root cause of the problems was most times this escaping stuff, and I never knew the proper answer and behaviour is already in syslog-ng, I've just forgotten about it completely.

And now as I check the documentation for syslog-ng, it does not mention this syntax either, even though it had been present even in the 1.6.x times.

So if you had trouble writing lots of regexps in syslog-ng configuration, and I told you to properly escape your regexps, please forgive me. syslog-ng is better than I've thought :)