Bazsi's blog

The 3.2 git tree received yet another feature I still wanted to shove into the 3.2 final: template functions. A template function is a transformation: it is able to modify the way macros or name-value pairs are expanded.

For example, let’s assume that you have a log file where you want to write CSV formatted data, which you could do with the following template:

template d_csv { file(“/var/log/data.csv” template(“$DATE,$HOST,$PROGRAM,$PID,$MSGn”)); };

This works fine until you discover that the CSV format can be broken if one of the expanded values contain a ‘,’ which is normally used to separate values. The CSV format solves this by quoting the values or by escaping the offending characters, e.g. you’d have to write something like this:

template d_csv { file(“/var/log/data.csv” template(‘”$DATE”,”$HOST”,”$PROGRAM”,”$PID”,”$MSG”n’)); };

This would solve the comma problem, but introduces another: the quote character can also be present in the macros expanded above, which if happens to be the case will also spoil the nice CSV format. Lucky us, the authors of syslog-ng had one little hack for this problem for a long time now: template-escape:

template d_csv { file(“/var/log/data.csv” template(‘”$DATE”,”$HOST”,”$PROGRAM”,”$PID”,”$MSG”n’) template-escape(yes)); };

This causes syslog-ng to automatically escape the apostrophe, the quote character and the backspace whenever it is encountered in the expanded string. But please note that it is only escaped when escaping and not in the literal portion of the string. However what happens if you don’t like the CSV format or you have different escaping rules? Or perhaps you only want to increase the size of your output file when the quotes are absolutely necessary? You could perhaps use rewrite rules, but that has some drawbacks: it increases the size of the log message and it is inconvinient to use. Until now, those were to only options. With a template-function however, the solution is easy:

template d_welf { file(“/var/log/data.welf” template(“time=$(escape-welf $DATE) host=$(escape-welf $HOST) program=$(escape-welf $PROGRAM) pid=$(escape-welf $PID) msg=$(escape-welf $MSG)n”)); };

Of course it is possible to add new template functions via the new plugin mechanism. Right now it is more of a framework and a syntax embedded in templates, and only the “echo” function is available, but I expect growth in this area.

In case you are interested this patch in the git implements template functions and this one adds plugin support on top of it. Adding a new function is simple, just look at the file named builtin-tmpl-func.c in the source tree.

As you probably know one goal for patterndb is to implement message classification.

E.g. in addition to extracting information from log messages, it also associates a “class”, later available in the “${.classifier.class}” value.

Right now, syslog-ng doesn’t really care what this string is. But the XML schema validating patterndb file lists the following four classes (taken from the logcheck project)

violation     – security violation
security      – other security events
system        – system information
unknown       – no rule matches

One one hand, the tagging functionality (e.g. the ability to also associate tags with each message) is superior to classes.

On the other hand, all tags are equivalent, thus if a message has 5 tags, then currently syslog-ng only provides functions to _filter_ based on tags, but not use it as a macro.

So for example it is possible to do:

file d_class_files { file(“/var/log/messages.${.classifier.class}.log”); };

But it is difficult to do with tags (except for using filters and different destinations), as there’s no such functionality. Another problem is that tags/classes are completely independent, in order to filter on the class of the message, one would have to use a match() filter like this:

filter f_class { match(“violation” value(“.classifier.class”)); };

My conclusion is that classes are better when used in templates, tags are better when filtering. The two should be merged somehow.

So I’m thinking on how to move forward. Here are the alternatives I’m considering:

1) the class of the message is always a tag in some generated format (e.g. if a message has class XXX, then a tag named “.class.XXX” would be automatically associated with the message.

This is somewhat cumbersome.

2) the class of the message is created as a tag as well, with the same name as the class.

e.g. we’d have a tag named “violation”, but that’d preclude the use of the “violation” name as a tag.

3) drop the class stuff and implement a macro trick that makes it possible to use tags in macro context

One way to do this:

file d_class_files { file(“/var/log/messages.$(expand-tag-name violation security system unknown).log”); };

The “expand-tag-name” macro function would try to look for the tags listed as parameters, and if the message matches it’d expand to the tagname.

This is not intuitive and if someone wants to use such an expansion in a lot of templates, it is also irritating and difficult to get right.

I’m leaning towards option number 2 above.

On an independent matter, the set of classes may need some thought. As I said the original list is borrowed from logcheck, but I think it probably needs to be expanded. Last time I got patterns for DNS queries, and although I could shove them into “system”, right now I feel that the point of classification is to categorize events by “importance”, in a similar spirit to syslog severity, but one that works even if the application developer uses a bogus severity when sending syslog messages.

So one email, two questions, feedback appreciated.
Thanks.