syslog-ng OSE 3.1beta2 release

I've mentioned shortly in my previous post, but here's a more official announcement: I've released syslog-ng OSE 3.1beta2, containing some important bugfixes.

The list of changes: http://www.balabit.com/downloads/syslog-ng/open-source-edition/3.1beta2/changelog-en.txt

Thanks to Martin Holste for the feedback he provided, hopefully we can forget about the "beta" part soon.

Patterndb release for syslog-ng 3.1

You may probably know that starting with syslog-ng 3.0, we started poking into the message payload by being able to extract information from the log messages and use that information in structured form for message routing, filtering and storing them as separate fields in a database table.

You may have read about patterndb on this blog or on Marci's blog and we have also given talks about it on different conferences: NNM 2009 and LSM/RMLL 2009.

The reason I'm raising the topic here again is that we have now released about 8000 patterns covering about 200 applications for patterndb and are now in the process of creating a community site to maintain this database.

You can download the database from www.balabit.com.

Also an important thing to know that syslog-ng OSE 3.1 features enhanced performance with regard to handling information extracted from the message payload and it also has support for the latest patterndb database format. So if you want to try the new database, fetch a copy of the latest 3.1beta2 release.

syslog-ng OSE 3.1beta1 released

I'm proud to announce that syslog-ng OSE 3.1 has been released and uploaded to our webserver. This version is new in two ways:

1) of course it has new features, see below for the most interesting bits

2) it is a "feature release", which means that once syslog-ng 3.2 or syslog-ng 4.0 is released, the support for this release will be ceased. See our new version policy at this link:

https://www.balabit.com/network-security/syslog-ng/opensource-logging-system/roadmap.bbx

Since the documentation is not yet up to date with this beta release, I'll try to include the most crucial information about the new features right here in this announcement.

For those who hurry, here's a link for the source code:

https://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.1beta1/source/syslog-ng_3.1beta1.tar.gz

And here are the binaries for Linux/FreeBSD systems:

https://www.balabit.com/network-security/syslog-ng/opensource-logging-system/

Select the Downloads tab, and in the Version selector select 3.1beta1.

Please try this beta version. Any feedback, positive or negative is appreciated, if you have comments, please post them on the mailing list at: syslog-ng@lists.balabit.hu

And now the new features in this release:

Support for patterndb v3

syslog-ng 3.1 now supports the patterndb v3 format, along with a bunch of new parsers: ANYSTRING, IPv6, IPvANY and FLOAT.Patterndb (more exactly the db-parser()) is a high performance message classifier and information extraction tool, that makes it easy to get away from the unstructured nature of syslog.

Patterndb has evolved since it was first introduced in syslog-ng 3.0. It is at the 3rd iteration, hopefully slowly reaching its final form. syslog-ng OSE 3.0 supported v1, our SSB product supports v2 and now syslog-ng OSE is the first version supporting v3.

Patterndb in general and the v1 format database is described in the syslog-ng manual at:

http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch02s12.html

The XML schemas that describe the different patterndb versions are available in the syslog-ng source tree:

http://git.balabit.hu/?p=bazsi/syslog-ng-3.1.git;a=tree;f=doc/xsd;hb=HEAD

The changes in the patterndb format as they evolved were described in Marton Illes's blog at

http://marci.blogs.balabit.com/2009/06/new-db-parser-format-and-other.html

But see the other related posts as well.

Old patterndb databases can be converted to the new format by putting them in the /opt/syslog-ng/etc/patterns.d directory and using the pdbtool utility using the command:

$ pdbtool merge -p /opt/syslog-ng/var/patterndb.xml \
-D /opt/syslog-ng/etc/patterns.d

Assuming the installation prefix of syslog-ng is /opt/syslog-ng. The above filenames are also the recommended and default names for patterndb related files.

Some v2 format patterns are distributed by BalaBit itself for its SSB product, download location:

https://www.balabit.com/downloads/files/patterndb/1.0-20081117/patterndb/

You can convert these db files using pdbtool as described above.

Work is ongoing to publish a more comprehensive patterndb, but more on that in a separate post.

pdbtool

Added a new "pdbtool" utility to manage patterndb files: convert them from v1 or v2 format, merge mulitple patterndb files into one and look up matching patterns given a specific message.

See the manpage (by adding /opt/syslog-ng/share/man in the MANPATH) and Marci's post:

http://marci.blogs.balabit.com/2009/08/db-parser-new-utility-pdbtool.html

Message tags

Support for message tags was added: tags can be assigned to log messages as they enter syslog-ng: either by the source driver or via patterndb. Later it these tags can be used for efficient filtering.

http://marci.blogs.balabit.com/2009/05/tag-support-in-syslog-ng.html

Rewrite structured data

Earlier structured data fields in the new RFC5424 style syslog protocol were only read-only values that could be referenced in a template, but they couldn't be changed, and neither was it possible to add new fields in an already existing syslog message.

Now all these became possible by using the same syntax that didn't work earlier, e.g.

rewrite r_sd { set("55555" value(".SDATA.meta.sequenceId")); };

Macro and name-value integration

Macros and name-value pairs got a little tighter integration. syslog-ng 3.0 was limited in the use of macros in the value() option of the match() filter: it could only use name-value pairs, although intiutively it should have supported macros as well. This was changed, starting with 3.1 it is now possible to use macros as well.

The following now works:

match("regexp" value("R_DATE"));

syslog-ng is now warning you in case you are using '
Name-value pair performance improvements

With the advent of patterndb and the spreading use of name-value pairs in syslog-ng, a strong limitation was the performance penalty of using dynamically created name-value pairs. This was now solved, 3.1 features a new data structure to store message payload and name-value pairs in, which results in a 3 times better performance when looking up a name-value pair.

Patterndb parser enhancements

Some parsers got additional features: NUMBER is now able to parse hexadecimal numbers, ESTRING is now able to search for a sequence of characters as the end of the string. These changes make it easier to describe log messages in patterns.

Information about non-portable facilities

Added non-standard and non-portable facility codes (range 10-15), and decoupled syslog-ng facility name database from the system used to compile syslog-ng on.

Until this time the facility codes as understood by syslog-ng were dependant on the platform syslog-ng was compiled on. This is not true anymore, syslog-ng comes with its own "facility" code assignments, based on the RFC, and adding some non-standard values found on various UNIX systems. prefix in the value syntax, because you can't use the full template syntax when you specify a value to match against.

Name-value pair performance improvements

With the advent of patterndb and the spreading use of name-value pairs in syslog-ng, a strong limitation was the performance penalty of using dynamically created name-value pairs. This was now solved, 3.1 features a new data structure to store message payload and name-value pairs in, which results in a 3 times better performance when looking up a name-value pair.

Patterndb parser enhancements

Some parsers got additional features: NUMBER is now able to parse hexadecimal numbers, ESTRING is now able to search for a sequence of characters as the end of the string. These changes make it easier to describe log messages in patterns.

Information about non-portable facilities

Added non-standard and non-portable facility codes (range 10-15), and decoupled syslog-ng facility name database from the system used to compile syslog-ng on.

Until this time the facility codes as understood by syslog-ng were dependant on the platform syslog-ng was compiled on. This is not true anymore, syslog-ng comes with its own "facility" code assignments, based on the RFC, and adding some non-standard values found on various UNIX systems.

The neatest syslog-ng hack ever

One of my collegues probably felt like crazy and implemented a twitter() destination driver for syslog-ng. Although the value is dubious, I think it is the neatest contribution to syslog-ng so far. :)

syslog-ng 3.1 status

Like I announced in one of my previous posts, towards the syslog-ng OSE 4.0 release I'm going to make smaller, short-term supported releases. The first of these, called syslog-ng 3.1 is nearing completion, and thus a status report is due.

Here's the original plan (quoting the roadmap page here):

• support tags for syslog messages: each message can be marked with one or more tags, then apply filtering based on tags
• patterndb: add tag support
• patterndb: v2 database format support
• patterndb: add parsers for IPv6 addresses and hex numbers
• converge macros in templates and name-value pairs even more (right now it is not possible to use any macro in match())

I've just pushed out another set of updates to our git repository, which:

• adds tag support: a new tags() filter and a tags() option for all sources and a builtin logic to assign the syslog-ng source name as a tag (in the format: .source.)
  
• adds support for patterndb v2 and a newly introduced but compatible v3 format
• adds "pdbtool" a new utility for managing patterndb files (not yet complete)
• a couple of new parsers (IPv6, ANYSTRING, FLOAT)

The last item in the roadmap is not yet addressed, in fact I haven't even started it yet. I'm thinking about leaving that out altogether in order to have 3.1 released as soon as possible. If you have an opinion about that please don't hesitate to post it here on the mailing list.

If you are experimenting with patterndb you are advised to use the 3.1 branch as development happens here. Of course if we find something that affects our current stable 3.0, I'm backporting the fix, but since 3.0 is stable, I'm only backporting bugfixes and not new functionality.

If you are interested you can get the sources via git, or if you prefer a tarball, just drop me an email.

Developer tools

BalaBit has grown quite a lot in the last 9 years since it was founded, these days there are about 60 employees and more than 50% of that is working in the development field (give or take a couple, I've lost count some time ago). As we currently work on 4 products, support 5-6 CPU architectures and a host of different Operating Systems, automation in development is a must.

We try to automate everything and that means a lot. Some examples:

• preparing the development workstation for development/testing work in 15 minutes for any of our products
• building source code for tens of CPU/OS combinations by issuing a single command
• creating bundles of intermediate components when generating setup packages
• doing releases
• test automation
• and a host of other things

Some of these solutions are completely our own development, others are derived from public projects, and as BalaBit tries hard to be a good friend of Free, Libre and Open Source Software (FLOSS) we try to contribute back to projects that we use.

A couple of weeks ago, I published our modified version of dogtail, a test automation framework for AT-SPI based applications. We maintain our own dogtail in-house and since our patches were not accepted, we published our changes in a public git repository.

Earlier, one of our developers contributed to WAF to support building with Microsoft Visual C++, we've been using his work in two of our internal projects.

And this time, we published cccl a wrapper for MSVC to make it compatible with the gcc command line, in order to compile autoconf based projects under MSVC.

As you could guess, BalaBit is primarily a UNIX/Linux shop, but we need to support products aimed at Microsoft Windows, however with some heavylifting combining the best of both worlds is possible. And we've never been afraid of challenges. :)

Hopefully you can use some of these results, maybe even contribute back.

patterndb updates pushed in syslog-ng OSE 3.1

According to the plan of my recently published syslog-ng OSE roadmap, I've worked on integrating the various patterndb related fixes/enhancements in the syslog-ng OSE 3.1 tree.

This now means that OSE 3.1 is now capable of working with all the version2 style pattern databases that syslog-ng Store Box is using. Here is a link for the SSB patterns: http://www.balabit.com/downloads/files/patterndb/1.0-20081117/patterndb/

I still need to work on integrating the new tags framework and the integration between tags and patterndb. Once that is done, I only have one item left for the 3.1 feature release.

So with some luck, we can have a new shiny syslog-ng OSE release this summer.

Please note that this is not released code yet and is only available via git, however if there's demand, I'm willing to create an alpha release (with binaries) if you want to try it. Just drop me an email, or simply write a comment to this post, and I'm going to create one for you.

Stay tuned.

syslog-ng rewrite use case: dpkg logs

One of my collegues (Péter Höltzl, he does all our trainings) has created a nice detailed example on how to use the parser/rewrite framework to pull in yet another application into syslog: dpkg, the Debian package manager.

If you are interested in what rewrite/parser can do for you, but didn't have the time to find out, the blog post is worth a read.