Bazsi's blog

I’ve been playing with github the last couple of months for git hosting, and since I like what I see and users seem to have found this out on their own, I figured this should be official.

The git repository is being moved from their old git.balabit.hu location to:

syslog-ng 3.3
http://github.com/bazsi/syslog-ng-3.3.git

syslog-ng 3.4
http://github.com/bazsi/syslog-ng-3.4.git

Older releases (like 3.2) will remain on git.balabit.hu for now. Please update your git clones with the new URL.

The syslog-ng webpages will be updated shortly.

Update: The syntax of this feature has slightly changed due to discussions on the mailing list (e.g. change the log keyword to channel), and I’ve updated it to use the current syntax.

syslog-ng is often referred as a very flexible application when it comes to processing logs. Over the years however, I began to feel that some things are a bit more difficult to achieve in the configuration language than it should be. For instance it is sometimes too rigid when you need a combination of parsers (patterndb with db-parser) and rewrite rules to achieve the goal you wanted. Parsers and rewrite rules are distinct part of the configuration, it is not possible to combine them into a single functionality. Also, declaring objects first and then referencing them later, makes the configuration easy to read, however sometimes it is quite cumbersome, when you only need to invert the result of an already existing filter.

To solve this situation, I’ve set out to implement an idea I had on mind for some time now. It is quite difficult to describe the feature in clear and concise words, as it is a combination of various changes that together makes syslog-ng configuration more flexible and easier to use, without sacrificing readability. Curious? Please read on.

In-line objects

Perhaps the simplest of all features is that you can now define the contents of a given object right on the spot, without having to use a separate statement. For example, earlier you had to write:

log {
  source(s_local);
  filter(f_postfix);
  destination(d_postfix);
};

Sometimes, f_postfix filter is only used once and is trivial. This can now be written as:

log {
 source(s_local);
 filter { program("^postfix/"); };
 destination(d_postfix);
};

Furthermore both the source() and destination() options can be written in-line, you simply use braces instead of parentheses. The same functionality applies to everything: sources, destinations, filters, parsers and rewrite rules.

Junctions

A limited form of junctions has been supported since syslog-ng 3.0 in the form of “embedded log statements”, which has been generalized now. Within syslog-ng, when a message is received it is dispatched to a log processing path or pipeline, which carries out the task at hand. A junction is a point in the log processing path where the processing is performed on multiple independent branches, each doing its own specific thing with the message.

The limited functionality in 3.0 only allowed the processing tree to split (or fork) into independent branches, each of the branches was a “sink”, where processing also ended. Configuration example:

log {
  source(s_all); filter(f);
  log { filter(f1); destination(d1); };
  log { filter(f2); destination(d2); };
};

This sample forks the processing path into two branches starting with the “log” keyword within the top-level log statement. The first branch evaluates the filter f1 and the writes matching messages to the d1 destination, effectively sending all messages that match (f AND f1) to d1. Likewise, d2 receives all messages that match (f AND f2).

The limitation of the embedded log statement concept was simple: it could only be listed at the very end of a log statement, and the end-result of the branches couldn’t be processed further. Effectively the message at the end of each branch “fell off”. Junctions on the other hand makes it possible to do things to messages once the branches converge to the same point again. Repeating the sample above, it is now possible to write:

log {
  source(s_all); filter(f);
  junction {
    channel { filter(f1); destination(d1); };
    channel { filter(f2); destination(d2); };
  };
  destination(d_all);
};

The new thing is that you can now add processing after the branches finish their processing. A bit more useful example would be:

log {
  source(s_apache_files);
  source(s_syslog);
  junction {
    channel { filter(f_apache_files); rewrite(r_apache_remove_file_header); parser(p_apache); flags(final); };
    channel { filter(f_apache_syslog); parser(p_apache); flags(final); };
  };
  destination(d_files);
};

This example does an alternative processing of incoming logs based on where the message came from.

Everything is a log expression

This feature is probably the most complicated, however provides very nice properties and expressiveness to the configuration. From now on, not just the well known log statement allows the specification of log processing rules, but all the objects in the syslog-ng configuration file can use the same expressive power.

It is now possible to use embedded log statements, junctions and in-line object definitions within source, destination, filter, rewrite and parser definitions. Huh, you could ask: what does it bring to me as a benefit? Well, until now, objects of different types were separate entities, connected using log statements, with this change a source can also specify a rewrite rule and that combination used as a log source in a log statement.

For instance, a usual source definition looked like this:

source s_apache {
  file("/var/log/apache/error.log");
};

If you wanted to process this log file in a specific way, you needed to define the accompanying processing rules (parsers and rewrite expressions) and combine them in a log statement. But how about this:

source s_apache {
  channel {
    source { file("/var/log/apache/error.log"); };
    parser(p_apache_parser); };
  };
};

log { source(s_apache); ... };

Can you see? The s_apache source used a file source and the reference of a specific parser and all messages read from the apache error log file would be processed by that parser. The log statement is just as simple as if s_apache would be a “normal” source definition. This feature allows pairing the essential log preprocessing functionality very close to the source itself, making it very easy to write and read the log statements. As an added bonus, it becomes very easy to distribute application specific source & parser definitions as an SCL configuration snippet.

Where?

This stuff is available in the syslog-ng 3.4 git tree, on master. It passes the included regression test, so it is at least dogfoodable. The nice thing about the implementation is that it only slightly increased the code size, but brought a lot of new features. If you have trouble getting the code from git, let me know, I’m willing to create an alpha release, so that it becomes easier to play with it.

Feedback

I see a lot of potential in this functionality, however my examples may have not been the best ones. I would really appreciate any kind of feedback, please be sure to send those to the syslog-ng mailing list or post me as a private email.

CzP has found out that syslog-ng is used in the Amazon Kindle. Seems like our userbase is in the millions.

I got some interesting comments in a forum posting, outlining a perception how syslog-ng’s development is driven by BalaBit. The original post is here, but the interesting quote I’d like to react is this:

@all Some general points:
A main difference between rsyslog and syslog-ng is that syslog-ng is backed by a large commercial organisation and a commercial fork of syslog-ng. This is not necessarily bad. But it is a difference to how rsyslog is driven: I get some funding for rsyslog development from Adiscon, but Adiscon is much, much smaller. So rsyslog is more of a community project. I have to admit that we at Adiscon still hope to find some commercial funding, e.g. via support contracts or custom development. But we do have far less of the traditional commercial machinery behind that (maybe it would be better we had, at least for me ).

[...]

To sum up: both rsyslog and syslog-ng are quite good implementations, each with their own strength. I think the main difference is the process in which they are created (more commercial-focus vs. community/technology-focus).

This is an interesting perspective, although as you may imagine doesn’t match my own idea how things work. But probably many of my reasons are not visible from the outside, thus this post. I hope to shed some light on how syslog-ng development is being done, maybe that’ll help you understand the situation better.

BalaBit may be larger than Adiscon, but syslog-ng development has never been its primary focus. syslog-ng was represented differently in the company for a long time: it was my hobby. Later on, we’ve received so many development requests and support queries that we’ve decided to build a product that is not simply a game (and responsibility) of one single guy, and thus Premium Edition was born. The way we’ve carried that out could have been done better, but I think this was discussed well enough in earlier posts. (here, and here). I think we’ve addressed most of the concerns in this area with the change in licensing, dropping the requirement for copyright assignment and so on.

Today, there’s a development team inside BalaBit, which delivers Premium Edition to paying customers, and there’s me and a couple of other guys in the company, who are not members of that team, but still want to work on syslog-ng. Since we fulfill other roles, in work-time we are not working on syslog-ng, but in our free-time (and coffee breaks we can talk, extend and work on syslog-ng hand-in-hand with the community.

That’s how syslog-ng is driven.

This means that the Open Source and the Premium editions are on a different track, features/bugfixes may be present in one, and not-present in the other. This would be quite cumbersome to maintain, thus from time-to-time we synchronize them up. That’s when features go to the OSE version or vice-versa.

The point is that syslog-ng Open Source Edition is independent from the Premium Edition, different people are doing it, with a different motivation. The OSE version in our case is not a bait for people to buy the Premium Edition, you can be a happy OSE user and be completely satisfied, or you can be a PE user if you need the backing services. It’s up to you.

The motivation for customers to buy syslog-ng Premium Edition is the professional support, the predictable release schedule, the number of platforms it realiable runs (and gets tested) on, the installation packages and the fact that you can actually have legal assurances that a piece of your infrastructure works, and there’s someone to blame and turn to if it doesn’t. Also, the PE team does a thorough testing on functionality coming from the OSE branch, not because it’d be done badly on purpose, but people usually focus to different things when they implement a functionality in an open source setting.

I don’t see the big difference between syslog-ng and rsyslog in the way it is driven. In fact Rainer is in about the same position at Adiscon, as I’m at BalaBit.