ABOUT
Archive for the ‘Open Source’ Category
syslog-ng 3.4beta1 released
Just before christmas I’ve released syslog-ng 3.4.0beta1, hopefully the last before the final release of syslog-ng 3.4. You can find the list of changes since 3.4.0alpha3 here.
Here’s the teaser for those wondering what 3.4 can do for them in addition to 3.3:
- Junctions, channels and the improved configuration format allows specifying log processing rules at even more flexibility.
- Full json support by the introduction of json-parser() and enhancements to $(format-json) template function.
- Support for the amqp() destination that implements support for the AMQP queueing protocol was added.
- MongoDB improvements to support replicasets, UNIX domain sockets. The performance was also improved by using insert operations instead of upserts.
- Added support for sending emails via the smtp() destination.
- Allow huge messages, instead of the old limits of 256k per message and 64k per value limits, the limits are 4GB for both.
- Added support for parsing the syslog message format after the initial reception. This can be used to “fix” messages before it actually reaches the syslog parser stuff.
- Native support for systemd.
- Demand loadable plugins to avoid having to explicitly write “@module” statements. This makes writing syslog-ng.conf files easier.
- A number of new template functions, like $(uuid), $(hash) and so on.
- A number of new macros $LOGHOST, $C_DATE and friends.
- A number of new parsers in db-parser, @PCRE@, @EMAIL@, @SET@, …
- Added rewrite operations to change message tags.
- Improved value-pairs expression support that allows specifying
More details can be found in the individual release announcements:
- 3.4.0alpha1: http://bazsi.blogs.balabit.com/2012/03/first-alpha-release-of-syslog-ng-3-4-published/
- 3.4.0alpha2: http://lists.balabit.hu/pipermail/syslog-ng/2012-May/018746.html
- 3.4.0alpha3: http://lists.balabit.hu/pipermail/syslog-ng-announce/2012-June/000144.html
As with all versions of syslog-ng, this wouldn’t have been possible without the help of the syslog-ng community. The role of the community is increasing with release-to-release, larger and larger features are contributed outside BalaBit and the number of contributors is increasing steadily too. I’d like to grab this opportunity to say thanks for everyone involved. Help is welcome and appreciated, be that code, documentation, a description of a use-case or simply just feedback. Thank you.
Stay tuned for the final version!
First alpha release of syslog-ng 3.4 published
I’ve just uploaded the first release in the upcoming 3.4.x series. This is an incremental step over 3.3.x, continuing to enhance syslog-ng with features that allows more in-depth processing of messages.
I consider the most important one the ability to freely combine different kind of processing elements (parser & rewrite rules and filters) with sources and/or destinations and handle the combination as a single object. This is listed “junctions & channels” below, but you can also read more details in this blog post.
Certainly, this release is not meant to be used in production, however it also helps if you try to run your production configuration, and report back on the results. The syslog-ng configuration parser was heavily modified in this release, as little as this can also help to improve syslog-ng. I hope that binaries for the experimental repositories of various distributions will show up shortly, until then you can always clone the git tree.
Here’s a excerpt of the NEWS entry that describes the changes compared to 3.3.x:
Features:
- Support for junctions & channels were added, which improve the flexibility of the syslog-ng configuration language. This allows combining sources with their closely tied processing functionality (like parser, rewrite and filter statements).Read this blog post for more information: http://bazsi.blogs.balabit.com/2012/01/syslog-ng-flexibility-improvements/
In the final form of the functionality the “log” keyword as described in the blog post above was replaced with “channel”.
- The functionality to query and manipulate sets of name-value pairs (often referenced as value-pairs and used in the mongodb() destination driver and the $(format-json) template function). got significantly improved. It is now possible to change the name of the keys when creating the output. See this commit for more information:https://github.com/bazsi/syslog-ng-3.4/commit/ddc7c2539bd66fa35e8df441e4baf58e87b6708d
- Plugins & modules are now demand-loaded automatically if the “autoload-compiled-modules” global variable is set to 1, which is the default. Any shared libraries found on the module search path is considered for loading if the configuration file contains a reference to a functionality it provides.To disable this functionality simply set the referenced variable to 0 with a “@define” statement and load modules explicitly via”@module” statements.
To list the available plugins & modules, use the –module-registry command line option for syslog-ng, which results in a detailed listing.
- Added a new parser named json-parser() to parse incoming JSON formatted messages. See this commit for more information:https://github.com/bazsi/syslog-ng-3.4/commit/e5569687bba2551c89a78faee55bcf8b4944066f
- Added a number of template functions:
$(length ARG) – length of a template expression
$(substr ARG START [LEN]) – substring of a string
$(strip ARG) – remove white space from the start and end
$(sanitize ARG1 ARG2) – join args to form a filename while removing special characters like ‘/’
$(+), $(-), $(*), $(/), $(%) – perform numeric operations
- Added support for replicasets to the mongodb() destination driver:https://github.com/bazsi/syslog-ng-3.4/commit/a980b9d268efa54879d3deeb4a53fa5a281629ba
- Added support for safe-mode() to the mongodb() destination driver that ensures inserts were properly executed at the cost of some performance penalties.https://github.com/bazsi/syslog-ng-3.4/commit/768f0c6ec8eba2ad51531f2331fb5635fe12c063
- Added support for SMTP destination to send out emails triggered by log messages.https://github.com/bazsi/syslog-ng-3.4/commit/404ceb959efe9715ce7437d7dcdc28ababbac590
- Added support for generating UUIDs via the $(uuid) template function:https://github.com/bazsi/syslog-ng-3.4/commit/a16798c653c057239236945034114d8abf320e44
- Added the @SET@ parser to db-parser().https://github.com/bazsi/syslog-ng-3.4/commit/ab08c84abbeda5a200b8150a6af7b02c64d84994
- Added support for dbd-option() in the sql() destination driver that makes it possible to supply driver specific options to the DBI driver.https://github.com/bazsi/syslog-ng-3.4/commit/9254fe9e6236746bf1f955f0cac1274634b1beda
- Reload of the configuration can now be triggered using “syslog-ng-ctl reload”.
- A new macro named $LOGHOST was added, which expands to the local hostname running syslog-ng.
- A set of time macros were added prefixed with “C_” that use the current time instead of the reception time (prefixed R_) and and the time that was included in the message (prefixed S_). This means that C_DATE expands to the current date, whereas R_DATE would expand to the date the current message was received at.https://github.com/bazsi/syslog-ng-3.4/commit/c2d17009e2ce14960acb519750fe2537b05e6f46
- Improved error reporting by including the configuration-file location of the object associated with the error. This makes it easier to diagnose errors even in the case of otherwise unnamed objects.
- This release also includes all fixes of the 3.3 branch, which are not listed here for brevity’s sake. The merged commit ID is: bf742b0, which is a couple of patches ahead of “3.3.4″.
Credits:
syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng.
Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute.
These people have helped in this release:
- Andreas Piesk
- Balazs Scheidler (BalaBit)
- Balint Kovacs (BalaBit)
- Evan Rempel (University of Victoria)
- Gergely Nagy (BalaBit)
- Heiko Gerstung
- Hendrik Völker (Verizon)
- Jakub Jankowski (superhost.pl)
- Martin Grauel (BalaBit)
- Matthias Runge (Fedora)
- Patrick Hemmer
- Russ Milne (Seccuris)
Project Lumberjack to improve Linux logging
In a lively discussion at the RedHat offices two weeks ago in Brno, a number of well respected individuals were discussing how logging in general, and Linux logging in particular could be improved. As you may have guessed I was invited because of syslog-ng, but representatives of other logging related projects were also in nice numbers: Steve Grubb (auditd), Lennart Poettering (systemd, journald), Rainer Gerhards (rsyslog), William Heinbockel (CEE, Mitre) and a number of nice people from the RedHat team.
We discussed a couple of pain points for logging, logging is usually an afterthought during development, computer based processing, correllation of application logs is nearly impossible. We roughly agreed that the key to improve the situation is to involve the community at large, initiate a momentum and try to get application developers on board and have them create structured logs. We also agreed that this will not happen overnight and we need to take a gradual approach.
To move into that direction, the benefits of good logging needs to be communicated and delivered to both application developers and their users.
We also talked about what kind of building blocks are needed to deliver a solution fast, and concluded that we basically have everything available, and even better they are open source. The key is to tie these components together, document best practices and perhaps provide better integration.
Thus project Lumberjack was born, hosted as a Fedora project at https://fedorahosted.org/lumberjack/.
The building blocks that need some care are:
- some applications already produce logs in structured format, those should be integrated (auditd for instance)
- we need to define a mechanism to submit structured logs to local logging services for further processing (ELAPI and some enhanced syslog)
- we need to make sure that local logging services cope with structured data (already available for a long time now)
- we need to define a mechanism to store messages in a structured form and a way query them
- last, but not least we need to define a naming scheme for event data which CEE can bring to the table
Most of these is already possible by using a combination of tools and proper configuration, however learning how to do this is not a trivial undertaking for those who only want to develop or use applications.
Changing that is the primary aim of Project Lumberjack. If you are interested in logging, make sure to check that out.
syslog-ng and the journal
There’s an ongoing project to create a new logging subsystem for Linux, called the journal, by Lennart Poettering of PulseAudio & systemd fame. It is implemented as a core component of systemd, thus has a good chance to be integrated to all distributions that carry systemd: Fedora, openSUSE, and probably others.
The vision and design is described in a paper here.
The reactions to the idea were mixed: there are some good features behind the idea, however it changes a couple of fundamental UNIX traditions. See article and comments here.
Since syslog-ng is also in the logging sphere, the logical question arises: how does this new project affect syslog-ng in the long run?
The short answer to that question is that it’ll probably help syslog-ng, but please read on.
Journald now & future
Right now journald is a very limited syslog implementation that only focuses on local logging: it collects local syslog messages, converts them into name-value pairs, then adds some trusted ones (like the pid, uid and gid) and writes these into journal files for storage.
The idea of working with structured messages in journald is currently limited because of the required application changes: only traditional syslog fields are available, the application message is stored within a single field. The vision here is to add structured logging to applications.
Other sources of local logs are to be integrated too: stuff like login/logout records (wtmp), audit logs and firmware (ACPI) logs.
The file format is interesting, although it is the source of most of the negative feelings: it is a binary format. It is undocumented (for now) with a library in the works to read & write them. The problem most people see that in emergency situations the file may become corrupt, and crucial information can be lost to diagnose the problem that caused the corruption in the first place.
As far as I understand, if applications were to support journal, they’d have to write records to the journal through the API without involving journald itself. This means that the file structure must support inter process synchronization, or that each application would log to different files to avoid that (using UUIDs for instance).
Network transport of the journal doesn’t exist yet, the vision seems to be that the journal is structured on the disk so that journals from multiple hosts can be merged simply by copying them to the same host. Ideas such as rsync or NFS mounts were floated as solutions to the transport problem. This is modeled after a bit like a git repository, which is great for storing source code, but may not be as great for log storage.
Further processing of logs is not in the cards, e.g. journald would take and store logs. In order to support higher level processing, applications would need to be modified and external tools to process logs would be needed.
As it seems, journald leans towards using a distributed model: each host has its own journals and whenever you want to look up records, you go back to the source and tries to address security concerns via cryptographic means (like using a chained HMAC for stored log messages).
Comparing to syslog-ng
syslog-ng has become much more than a syslogd in recent times. It supports structured messages similarly to journald (name-value pairs), can even extract such values from unstructured messages (db-parser), and can store stuff into much more than text files: MongoDB keeps the structure and provides indexed access, but output as JSON objects into simple text files is possible too
With syslog-ng (unlike journald), the user is in control: she can influence the storage policy, use whatever files, databases she pleases to store data. This is flexibility on one hand, but can be a problem if one wants to create tools that universally work with logs out-of-the-box.
Some users store messages in /var/log, others in /logs, yet others use MongoDB for their log storage. Writing a GUI application that works with log data out-of-the-box without having to specify where these are stored is next to impossible. The syslog model is to use specialized tools for each of these storage mechanisms and home-grewn scripts to do site specific processing. The reason is simple: use-cases are so different (from mail logs to financial transactions) and log data so voluminous (in the 100TB scale) that one size doesn’t fit all.
syslog-ng is more like an infrastructure for the actual, potential per-site log processing needs, journald is a complete system for a more limited use-case, which may just be enough for a number of users.
The security model of traditional syslog is to get logs off the potentially vulnerable system, leaving as small window for potential modification as possible. Certainly sending out a hash value periodically is possible with the journal too, however the actual log messages could be lost if the source system is compromised. This way syslog leans toward a centralized system, which itself can be distributed by using trusted intermediaries that store log data in need, but _independently_ of the source host.
Currently the only feature that journald has over syslog-ng is ‘trusted fields’, e.g. the fact that journald can determine the actual uid, gid, pid… values, making it more difficult to forge messages on the local host. Although these are handy, I didn’t see the need for these in practice in the past 13 years I’ve been maintaining syslog-ng, and I knew about the possibility to get this information from the kernel. Anyway, adding this to syslog-ng is not difficult, probably an hour or two and I may just do that to demonstrate. rsyslog has rushed to implement these probably for the very same reason: http://blog.gerhards.net/2011/11/trusted-properties-in-rsyslog.html
The other planned log sources are partially supported by syslog-ng too, for instance BSD process accounting logs are supported directly (http://bazsi.blogspot.com/2010/07/syslog-ng-and-process-accounting.html) and similar support can also be added for the others, if not already integrated with syslog.
What Next?
I think that journald can become great for computers where logging is not the primary function. If the user has never changed her syslog.conf file, journald would provide much more for the user out-of-the-box, than does the current syslog. These are:
- proper logging under the boot process
- integrated to the service manager, easing troubleshooting for failed services (saving stdout and stderr)
- GUI application for ad-hoc checking of logs
- the ability to programmatically query logs without having to care about site-specific policies (how log files are organized for instance)
For those cases where logging is important, mandated by regulations or operations for a heterogenous enterprise system journald will probably not be enough. Not enough even if the whole vision is accomplished. As I see these features will not be adequate in journald:
- off-system storage for a long period (1-5 years is mandated by various regulations)
- on-line log collection for getting the message off the potentially vulnerable system as fast as possible (being late at most a minute is acceptable, but hourly syncs are not)
- performance for on-line collection and storage (I’ve seen requirements for handling up-to 250k msg/sec)
- interoperability with syslog: not just receiving and storing but also preprocessing, normalization & classification.
- existing standards
- open for on-line integration with home-grewn processing tools and
- SIEMs
These are the benefits how syslog-ng can win from journald:
- structured logging in applications: if these would actually emerge, syslog-ng would be there to support them too
- syslog-ng as an application: currently syslog-ng is used as a system component, replacing syslogd, which drives some features that may not match the primary vision of syslog-ng itself. If local logging would be taken care of journald, syslog-ng could focus where it is best: collecting, preprocessing, normalizing & classifying logs, including the ones in journald.
Journald can mean that Linux boxes would probably be installed without a full-blown syslogd by default.
As long as interoperability with a syslog application is a goal for journald (and I fail to see it won’t be the case), syslog-ng can happily coexist with the journal and can itself leverage all the benefits that journald brings. Journald will only replace syslog and syslog-ng in a limited use-case, which is not a primary focus for syslog-ng.
Currently, syslog-ng is not a default choice for the majority of distributions, which means that right now one needs to explicitly install syslog-ng over the default. This will not change much by the introduction of the journal, except the fact that the current default is a more direct competition for syslog-ng. If journald replaces that role, the playing field would be leveled somewhat.
syslog-ng Open Source Edition 3.3.1 released
I’m proud to announce that syslog-ng 3.3.1 has been released.
It is an important release in a number of ways:
- syslog-ng architecture was completely revamped to scale properly in
today’s multi-core environments. I’ve seen it process 800000 messages /
second in some scenarios. - following the licensing change in 3.2, it incorporates contributions
(code, testing & feedback) from 54 individuals.
In total, the development of 3.3 took about a year, producing 580 patches and the following diffstat:
294 files changed, 20314 insertions(+), 7624 deletions(-)
That compared to the total number of lines in the git repository (81804 lines) is a significant change over 3.2
See the syslog-ng ohloh page for more statistics:
https://www.ohloh.net/p/syslog-ng
For the list of features and changes, please see the NEWS entry.
You can download the source code from:
https://www.balabit.com/network-security/syslog-ng/opensource-logging-system/download/syslog-ng-ose/3.3.1
A number of distributions already carry 3.3.1 packages thanks for our faithful package maintainers.
You can also get the source code from git, at
Happy logging.
netfilter.org says thanks
You may not know, but I used to contribute to the Linux packet filtering subsystem called Netfilter: I primarily worked on transparent proxying, the technology needed for our proxy based firewall/gateway product, Zorp. If I remember correctly, we completely revamped that thing about 3 times, until it got integrated into the kernel.
Because of my Netfilter work and my affiliation with BalaBit, BalaBit has been a sponsor of Netfilter Workshop, the annual gathering for Netfilter developers. This is a very enjoyable occassion, at least I do enjoy meeting with guys like Patrick McHardy, David Miller, Harald Welte and the others. I’ve learned a lot from them.
I was told that because of our past support BalaBit got included into the “Thanks” section of netfilter.org, I’m proud of being mentioned there, being a (small) part of something that is deployed on the Internet in such great numbers is a good feeling.
On CVE-2011-1951: bug or security issue?
There’s an ongoing debate on the Linux Kernel Mailing List, whether security issues need separate attention. While I agree that distributing available information on security relevance is a good thing, I can also understand the concerns about the “security circus”. Being the finder of a security bug has value in one’s reputation, and building that reputation is a priority for some. Positioning mere bugs as security problems is quite simple: any crash-bug can easily become ‘Denial of Service’.
Certainly, crashes in a piece of the critical infrastructure is a severe problem, and addressing those bugs is important, but pinpointing a single case over the other ones withdraws energy from other, perhaps more important efforts.
syslog-ng had a bug in its PCRE support code: whenever the PCRE engine returned an error code when performing a match on a global (/g) pattern, it entered into an infinite loop, eating up memory and CPU cycles until memory runs out or it gets killed. It is important to note that at this point the regular expression is already compiled, so triggering this condition is not as easy as specifying an incorrect regular expression in the syslog-ng configuration file.
But anyway, mishandling error returns is bad, especially as starting with PCRE version 8.12 always returns an error because of the bug in the glue code. This new version is popping up in various distros, and was the first hint to find this bug.
This means, that you always get the infinite loop with a recent PCRE version in case you have a subst() rewrite rule in global mode and there comes a message which does not match. I’d translate this as the most obvious case when testing one’s deployment.
And quite possibly this bug is never triggered in any other cases.
Is this bug serious? Well, for those affected this can certainly cause some trouble. Is it worth fixing? Of course. Is it a security problem on a production system, with a stable software and configuration environment? Well, doubtful.
But still, syslog-ng is being patched with feverish pace in distros, and probably by users with custom compiled packages. This effort could perhaps be used to test and provide feedback on the syslog-ng version in development: 3.3.
Behind the scenes: syslog-ng 3.3
I just wanted to let you know, that fixes are nicely coming into the 3.3 beta tree, although it might not be very visible from th outside.
So if you consider trying out 3.3, I’d suggest to try a git snapshot instead of the 3.3beta1 tarball.
I’m trying to release a beta2 or rc1 in the near future. The version number depends on how much feedback we get until then 
Maintenance Policy update for syslog-ng Open Source Editions
Dear syslog-ng users,
As discussed on the syslog-ng mailing list, the current versioning policies regarding syslog-ng Open Source Edition is confusing, and with the proliferation of syslog-ng versions, their maintenance is an increasing burden on the syslog-ng project. Currently three major versions are supported (3.0, 3.1 and 3.2) and a fourth one (3.3) is in active development.
A decision was made that the distinction between “feature” and “stable” releases for syslog-ng OSE releases will cease to exist: all releases will have the same status support-wise:
- they will be supported for a year, or
- until the next stable release is made
- whichever is longer
Also, the versioning of Open Source and Premium editions will become completely independent, and it is not possible to compare their functionality on the version number alone. The Premium Edition will always be based on a specific Open Source release, and provide additional functionality compared to the base version. OSE releases published after the PE release may provide additional functionality, not yet present in a PE release.
The changes above cause some of our currently supported versions to be deprecated. In order to provide a time window for migration to a newer release, the following EOL dates were set:
- syslog-ng 3.0: 30th, June 2011.
- syslog-ng 3.1: 30th, June 2011.
Everyone running these or earlier releases should upgrade to the latest 3.2.x release, which is currently at 3.2.2 (with 3.2.3 being prepared).
For more information:
- support periods for syslog-ng OSE versions:
http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/support - to download the latest syslog-ng OSE sources:
http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/downloads/upgrade
The syslog-ng project, on its own doesn’t provide syslog-ng binaries, except for a limited number of Linux distributions. It is expected that users compile syslog-ng on their own, or use the binary provided by the OS supplier.
This change doesn’t affect you, if
- you run a Premium Edition of syslog-ng,
- you have a support contract in place, that says otherwise
Happy Logging,
Bazsi
The 2nd alpha for syslog-ng 3.3 is released
I wanted to let you know that a new alpha version for the upcoming syslog-ng version has been published on our website. It is by all means an important step forward, because the new release brings us:
- It uses multiple threads to do its work, thus capable of scaling to multiple CPUs. Add ‘threaded(yes)’ to your option block and you’re good to go.
- Built in mongodb destination for folks who prefer NoSQL over traditional SQL.
- This version has a new and improved “pdbtool patternize” functionality
You can find the latest release here:
http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.3.0alpha2
Alternatively, you can also check the git repository at:
git://git.balabit.hu/bazsi/syslog-ng-3.3.git (browse online)
Known issues
I know for certain that it will not work on FreeBSD unless the following environment variable is set:
IV_EXCLUDE_POLL_METHOD=kqueue
Otherwise syslog-ng will crash at startup, immediately.
Stability
Certainly you wouldn’t want to run your production system on this version, however this code base had endured a couple of test rounds, so if it compiles and starts up for you (which can be a challenge I can admit), then the basic functions will most certainly work.
Features:
- Support for systemd activation added.
- Add support for customizable token delimiters in “pdbtool patternize”, which makes the resulting patterns to have much better quality.
- Added support for a –no-parse command line option for “pdbtool patternize” to avoid parsing the input as normal syslog messages.
- Added a new flag ‘ignore-errors’ to LogWriter based destinations (file, pipe, tcp, udp, syslog)
- Added support for specifying the suppress() option globally.
- Added a new macro named $CONTEXT_ID that expands to the current context-id in correllated rules.
Bugfixes
- Fixed a tcp()/udp()/syslog() destination driver issue that caused aborts during startup.
- Handle non-epollable devices like /dev/null normally by failling back to the regular file-like handling. Earlier these caused an abort inside ivykis.
- Makefile fixes to make it possible to compile syslog-ng from a “make dist” tarball.
- Added error messages about libdbi initialization failures, that would prevent the SQL destination from working.
- Make it possible to compile against PCRE not in a standard location (caused compilation failures on FreeBSD).
- Fixed several $(grep) related bugs:
- not to cause a segmentation faule when the filter expression supplied contains syntax error
- the template parsing code removed quotes required by filter expressions, with the current change it is possible to use quotes in the filter expressions directly:
$(grep (‘$username’ == ‘root’) $MSG) - fixed the handling of template function invocations within correllation that had multiple messages in its context.
- Fixed a segmentation fault in pdbtool match –debug-pattern in case there was no matching rule.
- Fixed “pdbtool test”.
Other changes
- Changed the default syslog-ng.conf version number to 3.3 to match the current version.
- The pipe() destination used to override the default value of flush_lines() by explicitly setting it to 0. This behaviour has been removed, now the pipe() driver will properly use the default, unless overridden explicitly.
- Ported the 3.2 linking changes to 3.3, which means that:
- libsyslog-ng-patterndb.so is gone, both the command line pdbtool command and the patterndb plugin (libpatterndb.so) links its contents statically, to improve portability on Cygwin.
- unit tests & command line utilities link properly in –enable-mixed-linking mode
- Added debug messages in the correllation code to make it easier to debug correllation rules.
- Added debug/verbose options to pdbtool.
Credits
Code, bugreports, testing, documentation suggestions and other improvements were contributed by:
- Arkadiusz Miśkiewicz (PLD Linux)
- Balazs Scheidler (BalaBit)
- Balint Kovacs (BalaBit)
- Corinna Vinschen (RedHat)
- Dalibor Toman (fortech.cz)
- Gergely Nagy (BalaBit)
- Laszlo Boszormenyi (lsc.hu)
- Marius Tomaschewski (Novell)
- Peter Czanik (BalaBit)
- Peter Gyongyosi (BalaBit)
- Tom Gundersen (jklm.no)
- Valentijn Sessink (blub.net)
- Zoltan Pallagi (BalaBit)
syslog-ng 3.2 in openSUSE
The adoption rate of syslog-ng 3.2 is marvellous. It was made available for Mandriva on the date of the release, and about a week later openSUSE Factory has a package, thanks to Marius Tomaschewsky. I also received a patch to include support for cygwin into the system() source, courtesy of Corinna Vinschen. FreeBSD ports still has a 3.2beta1, hopefully it’ll be updated soon.
I’m happy.
A Mandriva package of 3.2.1 is already available
Thanks to Guillaume Rousse, an RPM package for Mandriva Linux is already available from syslog-ng 3.2.1.
Let’s see how the other distros will react
syslog-ng 3.2.1 released
Hi,
After 2 alpha and a beta release I’ve decided to declare that syslog-ng OSE 3.2 is now stable, and thus I’ve released 3.2.1, the first in the 3.2.x series. This version has the largest list of features even since the syslog-ng project was born, so make sure you check out all the goodies.
The key features were already covered on this blog in earlier posts, the most important ones in my opinion are:
- licensing change to LGPL/GPL combo, without the need to sign a contributory agreement
- plugins
- log message correllation support in db-parser
- automatic pattern generation for existing log files with pdbtool patternize
- template functions
Read the changelog and/or the documentation for more information. The diffstat between 3.1 and 3.2 are quite big:
407 files changed, 50305 insertions(+), 36487 deletions(-)
Considering that OSE 3.2 has 70k lines (more specifically 70163) in its git repository, this is quite a lot.
Source code:
The list of changes are summarised in the changelog here:
https://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.2.1/changelog-en.txt
The updated documentation is also available here:
Happy logging!
syslog-ng OSE 3.2beta1 released
Dear syslog-ng users,
I’m happy to announce that an important milestone towards the new syslog-ng release has been reached.
syslog-ng OSE 3.2beta1 has been uploaded to our website. It contains a lot of important new functionality compared to 3.2alpha2 and also some care has been taken to shake out release-critical bugs.
As of know I don’t know about any Release Critical issues, but I do now that a number of such bugs have been fixed since alpha2. So if you had issues with the alpha release and didn’t report the issues, it still is probable that this release fixes them.
I think this beta is good enough for everyone to try out, so if you like the new features, I’d appreciate if you could report with either negative or positive feedback to the syslog-ng mailing list.
If you haven’t tracked 3.2 development so far, I’d recommend to read the 3.2alpha2 and 3.2beta1 announcements for a sumary of new features coming in 3.2.
You can find these here:
http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.2beta1/changelog-en.txt
http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.2alpha2/changelog-en.txt
This time in addition to the source code, I also release syslog-ng binaries for Linux/i386 and Linux/amd64. The binaries are not thouroughly tested and will probably never be. The canonical release format for syslog-ng is the source code. Anyway, if you feel there are problems with the installer/binaries, feedback is welcome, we’re willing to fix them.
Here’s the NEWS entry for the 3.2beta1 release:
3.2beta1
Mon, 11 Oct 2010 12:25:07 +0200
Changes and new features destined to the syslog-ng 3.2 release are
complete, and starting with this release, only bugfixes and minor
changes are possible. There's only one exception to this: the
correllation framework in db-parser() is still considered
experimental and is recommended for early adopters only.
This beta has gone through some testing and initial blocker problems
were fixed before the release. Right now I'm not aware of any
serious issues, but as always, testing is appreciated.
New features since 3.2alpha2:
* Added support for message correllation in db-parser. See the
relevant blog posts for more information:
http://bazsi.blogs.balabit.com/2010/10/syslog-ng-correllation-updated/
http://bazsi.blogs.balabit.com/2010/09/syslog-ng-correllation/
* Added "pdbtool patternize", which implements automatic patterndb
generation from a sample log file.
http://gyp.blogs.balabit.com/2010/01/introducing-pdbtool-patternize/
* Added template functions framework and some initial functions:
http://bazsi.blogs.balabit.com/2010/09/introducing-template-functions/
The new functions are: $(echo), $(grep) and $(if)
* Added support to process native syslog.conf file using the
syslogconf SCL plugin.
http://bazsi.blogs.balabit.com/2010/09/syslog-ng-now-supports-the-syslog-conf-file-format/
* Added support for comparison operators in filter expressions, e.g.
it is now possible to use "$FACILITY_NUM" < "5". String and
numeric operators are also provided, the same way as in perl.
* syslog-ng now automatically detects if an incoming message is in
RFC3164 or RFC5424 format. This means that the syslog driver can
be used to process both.
* Added pdbtool validation support, using the "pdbtool test --validate".
Requires an installed xmllint program.
* pdbtool is now able to merge patterndb XML files recursively in
order to make it easy to use the results of the patterndb project.
* db-parser() automatically assigns class-specific tags to messages,
this means that a message classified "system" will get a
".classifier.system" tag in addition to storing the class in a
name-value pair named ${.classifier.class}* It is now possible to use multiple program name patterns for a
single ruleset in patterndb.
* Added $(ipv4-to-int) template function to convert an IP address to
its numeric representation.
Bugfixes since 3.2alpha2:
* Fixed a possible infinite loop in "pdbtool test" in case
program/message was missing from the sample message.
* SQL: revert don't require the current CVS version of libdbi
* Don't report "this config file version is too old" multiple times.
* Underscore and dash are assumed to be equivalent in plugin names.
* Various memory leaks were plugged.
Removed functions:
* Removed the use_time_recvd() global and per-destination option,
deprecated since 3.0. Can be substituted with $R_ prefix in macro
names.
Other changes:
* Restructured the source tree in order to make compilations of
independent plugins easier and faster. Modules go to modules/
subdirectory, the core lives under lib/ and the main executables
go into syslog-ng/
* SCL paths are determined relative to ${datadir} instead of ${prefix} to make distribution packaging easier.* Pass -avoid-version when linking modules.
* syslog-ng now requires bison 2.4, this is also checked by the
configure script.
Credits:
I’d like to thank the following people who made this release possible.People in no particular order:
- Péter Gyöngyösi for his patternize work
- Márton Illés for pdbtool test
- Róbert Fekete for his feedback on various functions/ideas
- Csaba Major for listening to my stupid ideas
- Martin Grauel for giving a talk on syslog-ng 3.2 and for his number of useful comments
- Bálint Kovács for trying patternize, giving me bugreports and for his patternize related ideas
- Péter Czanik for asking distributor maintainers to package syslog-ng 3.2 and to return feedback on the build process
- Matthew Hall, Martin Holste, Patrick H. for his patterndbfeedback and their support on the mailing that gave me time towork on syslog-ng instead of doing syslog-ng related mailing
Hopefully I haven’t forgotten anyone. I’d like to open the 3.3 branch of syslog-ng now and I’d appreciate if distributions would start adopting the 3.2 package. It comes with big build changes, so maintainers
please expect some more time with this release.
I’m trying to fix bugs fast as they appear to get a stable 3.2 by the end of this year.
Happy logging!
syslog-ng Open Source Edition roadmap updated
I’ve updated the syslog-ng OSE roadmap in the BalaBit webpage. You can find it here http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/features/roadmap. The Premium Edition roadmap has also received an update some weeks back, check this page if you are interested in that.
multi-threading coming in syslog-ng OSE 3.3
As posted on the mailing list already, I’m planning to turn syslog-ng into a fully-multi-threaded application in order to improve performance on multi-core systems. Since I don’t want to start destaibilizing 3.2 (rather to push dub that as stable soon), this will become part of OSE 3.3.
During my holidays I’ve worked a little on this to get some actual numbers how much it improves performance. Thus I’ve added the bare-bones of threading to all input/output state machines, which were previously all running in the same “main” thread.
The code is _very_ experimental, it wouldn’t work with more than a single client, but this already shows that using a single client and a single destination file, it gives us about 50% performance boost. On my development laptop with X and every desktop shinyness running, it increased performance of syslog-ng from about 65k/sec to 100k/sec with flow control disabled and 90k/sec with flow control enabled. (the extra 10k was probably dropped without flow control because the output thread was probably slower writing out the stuff to disk, but I’m not sure).
If anyone wants to see what I’ve done so far, here is the commit:
http://git.balabit.hu/?p=bazsi/syslog-ng-3.2.git;a=shortlog;h=multi-thread
syslog-ng & open core, why we think it is different
As you may have seen in my last post, syslog-ng was quoted as an example of “open core” development model, which has problems in the eyes of some Open Source purists. I was trying to do my homework and understand their point of view and see what we can do about it. If you are interested, you can read these articles:
- Dave Neary’s Rotten to the Open Core
- Simon Philip’s Open Core is Bad For You
- Henrik Ingo’s So if I don’t call myself ‘open source vendor’, then everything is fine? (yes)
People who know me either personally or through the syslog-ng mailing list would probably know that I by no means want to publish something substandard or cripple the syslog-ng open source functionality in any way. I wanted to start with this statement as the way I see there’s only a small and subtle difference between an “open core” and a “true open source” project. I’d like to explain why I feel that syslog-ng is still a “true open source” project.
The most to the point explanation why “open core” is bad I saw thus far was: “The question this boild down to is: Does the Open Source version exist because it’s a nice selling point or does it exist because someone believes in the values of Free Software?” [link]
I’d say that both is true for BalaBit: we do believe on Free Software and we do publish a lot of that, outside syslog-ng as well (our TProxy work was integrated in kernel 2.6.28, various libdbi/dbi-drivers patches, iptables-utils, jailer, restrict and a couple of others, read this page for more). Some of these are more successful, others less so, but I guess the same applies to other free software hackers too.
So why I think there’s only a subtle difference between an “open core” and “true open source” development? Because at the fact-side, they look _very_ similar. Some quoted “true open source” projects (Apache, rsyslog, JBoss, Tomcat) and probably others support commercial development on top of them: their “core” license is compatible with commercial licenses (either LGPL or something else). The difference seems to be the very existance of a commercial offering from the same author.
So while syslog-ng 3.2 will have the same licensing structure as rsyslog, some still say that syslog-ng is not a true open source project. Again, the root cause for that is the existence of the syslog-ng Premium Edition package. I was trying to understand the logic behind this stance and came up with the following list of reasons (not necessarily applying to syslog-ng, but more on that later):
- with “open core” development, products usually fork the Open Source part of the project for their internal development and the flow of bugfixes and features on the open source portion happens in a closed way as a code dump
- the development is driven by the commercial edition, changes are trickling to the OSE version with a delay
- because of this setup, the community has no say in the features or in the way those features are developed
- because of this setup & without extra effort the open core part can quickly deteriorate
- the neglection of the open source portion can go as far as being completely unusable (that’s why the word crippleware)
- this process may not be visible right from the start, Open Core development may look like a true open source at first, but this can happen with time, effectively pushing users to pay money when the OSE version deteriorates and the replacement of a core component of a system would be more expensive
- when a competing open source solution is contributed, such a contribution is not accepted into the upstream project, because that would compete with the commercial edition.
The way I see it, the line between “open core” / “true open source” is very thin and also difficult to judge for an outsider. These are the reasons why I think that although syslog-ng does have a commercial edition, it is still an open source project.
Code base fork
Right now the code base for the OSE/PE editions are in different repositories. However one reason of the licensing change in 3.2 is to eliminate this and use the same core in both the PE and the OSE editions.
It must also be added, that the current setup was created because of community needs: in the early days, only a single, internal repository was used and the OSE edition was exported as a tarball every night as a snapshot. As the lack of a real version control system was a problem for the community we came to the currently used setup: two git trees, patches synchronized between them. This is of course much more work, but we volunteered to do that.
Development drive
Right now, the development on syslog-ng happens in three parallel teams. They each create original work. These are:
- the syslog-ng Store Box team,
- the syslog-ng Premium Edition team and
- the Open Source team.
The method used to synchronize work is as follows:
- bugfixes are done in the code base the bug report is coming from
- bugfixes in parallel branches are synced as the first task of every maintenance release
This means that it is true, that with PE related features OSE is behind, but this is also symmetric: OSE developed features take their time to propagate into the Premium Edition.
One noteworthy thing is that there are several people inside BalaBit who contribute their free time to syslog-ng development, you can find their git trees on git.balabit.hu.
And the features in the OSE edition are not worthless, they are really important in the future of syslog-ng. Some examples:
- db-parser to parse log messages (in 3.0)
- change from syslog-only implementation to a syslog-independent one (nvtable in 3.1)
- the new plugin architecture (in 3.2)
- syslog-ng configuration library (in 3.2)
- first support to non-syslog messages (3.2)
- and so on.
Community involvement in development decisions
Well, this is a though question. syslog-ng may not attract that many contributors or I myself suck at this, but it has nothing to do with the current PE/OSE division. Even if I was directly asking, I received only little feedback and this was in no way different in the first 8 years (1998-2006) of syslog-ng when there was no commercial fork. In fact, the community seems to be more active these days.
But anyway, since I myself develop the OSE version and it is out in the open (as the git history and my blog posts prove), the community would have a chance, at least in the features that BalaBit itself has chosen to implement. Also, small feature requests on the mailing list are quite rapidly implemented (the last thing I remember from the last couple of weeks is the “condition()” option to rewrite expressions).
With the current architecture/license change, I plan to go back and seek for contributors who didn’t want to sign the CLA whether they’d be interested in contributing their code as a plugin, or perhaps directly into syslog-ng.
Open Core deterioration
I myself worked very hard for this not to happen, and if you look at the mailing list archive, a lot of this time actually was spent from my free time.
Since the project/release overhead of a fix is smaller in the case of the Open Source development, it quite often happened that the fix was first made available as a git commit in the OSE branch. Then integrated tested and released in the Premium Edition as well with a slight delay.
Forcing users to use the Premium Edition
I always said and I believe that the Open Source Edition is a full featured syslog implementation. I’d say nothing proves my word better than that it became and was proposed as the default syslog implementation of various distributions. The existance of the Premium branch even created further features in the OSE, which may or may not have happened without that.
It is perfectly acceptable for us, that someone will stick to the Open Source edition. We work hard to update syslog-ng packages in distributions to its latest version, even though that competes with our commercial offering.
So I wouldn’t say we force them, not on the feature and neither on the quality side of things. The Premium Edition offsers the following added value compared to the Open Source Edition:
- support for a wide variety of Opearing System and CPU combinations (about 27 right now, but about 10 new combinations are in the pipeline for 4.0)
- longer support cycles
- support services
- maintenance & bugfixes
- …. and additional features.
Right now, our feeling is that the additional features is a selling point and without it, we’d be earning less money. With less money, we could allocate less time for syslog-ng. And I’m sure that’d affect syslog-ng, both Open Source and Premium Editions negatively.
Competing features
One more complaint against the “Open Core” business model is that there’s a conflict of interest on the maintainer of the Open Source portion whenever the community is contributing a feature that is present in the commercial edition.
The way I see it and with the advent of the new plugin based architecture, this is not necessarily a problem, since a plugin doesn’t have to be distributed with the main distribution tarball. In fact I expect several plugins that wouldn’t be added to the distribution. For example gyp’s Twitter destination is a nice hack, but I probably wouldn’t include it. Also, a Linux distribution is in the position that a plugin/functionality can easily reach much more users than the syslog-ng packages that we ourselves publish.
But anyway, once a feature is in the scope of syslog-ng (and being a feature in the commercial edition certainly proves that point) and a working implementation is contributed on the syslog-ng mailing list, then I hereby state that we are willing to include it in the main syslog-ng distribution. Of course technical and quality issues do still apply, but if there’s enough support from the community (e.g. on the syslog-ng mailing list) and if at least the plugin names are different from the commercial edition (this is purely a technical reason to make the maintenance feasible and user confusion low).
Since I feel that this topic is one of the strongest arguments against the “freeness” of syslog-ng, but with the above statements we are at the same level as any other projects “true open source project”: you need to take our word for it. I hope that with this post and with our last 12 years of open source work, we earned this trust. And if we don’t keep this promise we can be blamed.
But until that happens, please don’t judge syslog-ng’s freeness.
Conclusion
I agree that trying to balance between the Open Source and Premium Editions of syslog-ng is a difficult task, one that requires a lot of work too. We may have made some mistakes, but our intention is clear: OSE is an independent entity, standing on its own feet.
I hope this blog post clears up some of the confusion around this.
PS: Thanks for reading until this far.
LWN: syslog-ng rotten to the (Open) Core?
This was first posted as a comment under an article on lwn.net, but I thought it was important enough to post it here for others not reading lwn. Please go ahead and read the original article which is about the “Open Core” business model and its problems from the Free Software community point of view.
A commenter thought that syslog-ng was an example, which only exists as a marketing tool for the company’s commercial offering. Anyway, here’s my post:
First of all, I want to make it clear that I’m biased on the syslog-ng case, but still wanted to express my opinion here. I’m biased as I’m the primary author of syslog-ng.
I think syslog-ng is a completely different case from the one described by Neary. The GPL version is not crippleware, it was never published for marketing purposes only and for the majority of syslog-ng’s existence only the Open Source stuff existed. The Premium Edition is only about 3 years old and syslog-ng started in 1998.
We never removed features from the OSE version, the Premium Edition only included _additional_ features, and a lot of those are already available in the OSE.
Some examples:
* TLS support (became available in 3.0, almost 2 years ago)
* SQL destination (became available in 2.1, 2.5 years ago)
* performance improvements (3.0)
* etc.
In the other direction, we usually receive bugfixes and it is a pure technical reason that we used to require copyright assignment: I wanted to keep the two branches as close as possible (which if not done is the reason #1 why Open Core products become crippleware fast). _And_ since we heavily invested in automatic testing and our customers report bugs directly to us, we fix way more bugs in the OSE version than the community.
But anyway, I didn’t think that the dual license model was so problematic at the time we made this decision 3 years ago. Our efforts have never been “Rotten to the Open Core”. If you don’t believe that, check out the git repository or read the mailing list archive and see it yourself.
And this whole mess is the past, OSE 3.2 has been relicensed, and it is true that we’re going to publish non-free plugins, but anyone else is welcome to join and do the same.
syslog-ng 3.2alpha2 released
I’ve just uploaded syslog-ng 3.2alpha2 to the release directory. The last alpha release didn’t compile on all supported platforms and the automatic test-suite was disabled, because it only worked if syslog-ng got installed first.
These obstacles have been overcome and together with some fixes and a couple of new features, 3.2alpha2 is now available. I’ve also forward ported all bugfixes from syslog-ng 3.1.2.
For those who are starting to experiment with the 3.2 branch, here’s the list of new features compared to 3.1. Those who tried 3.2alpha1, the list of changes compared to 3.2alpha1 is at the end of this post.
Since the documentation of syslog-ng is not yet up-to-date with the new features introduced, I’ve tried to also include URLs for the best known descriptions. The references may not be 100% accurate, but should give anyone interested an idea how to start experimenting.
Also, please note that although this is an alpha release, the bulk of the changes are in the configuration parser, so once your configuration was parsed properly and syslog-ng starts up, an almost unchanged code is processing it. This means that this release should be good enough to start playing with. And feedback about what kind of syslog-ng.conf parsing errors you encounter on real-life configuration files is more than welcome.
Code quality & functionality wise, this could be a beta release, I only expect “procedural” changes, like cleaning up the plugin names, which wouldn’t be nice to do in a beta release (though not unheard of
New features in 3.2:
- Plugins: the new architecture replaces the old monolithic one, all syslog-ng functionality is loaded from external plugins when needed. It is possible to write plugins to extend syslog-ng functionality in the following areas: sources, destinations, filter expression, parsers, rewrite ops, message format.
http://bazsi.blogs.balabit.com/2010/07/syslog-ng-contributions-redefined.html
- The framework for a “syslog-ng configuration library” (aka SCL) a collection of configuration snippets installed along syslog-ng, simplifying the authoring of syslog-ng configuration files.
http://bazsi.blogs.balabit.com/2010/07/syslog-ng-contributions-redefined.html
- pdbtool match is now able to read a file containing syslog messages and apply patterndb and a filter expression on the contents.
- pdbtool test is now able to perform pattern testing automatically based on the supplied example log message.
- Persistent state containing the current file position for file sources is now continously updated during runtime, instead of updating it only at exit, which makes it much more reliable in case syslog-ng doesn’t terminate normally.
- Better syntax error reporting in the configuration file.
- Support for reusable configuration snippets, similar to macros with parameters, named “blocks”.
- Added a confgen plugin that includes the output of a program into the configuration file, making it possible to generate configuration file snippets dynamically.
- Support for BSD-style process accounting logs via the pacct() source driver defined in by SCL and the underlying pacctformat plugin.
- Support for explicit COMMITs in the SQL driver, this speeds up SQL INSERT rate significantly if flush_lines() is non-zero.
- It is now possible to supply a filter to rewrite expressions and only apply the rewrite rule in case the filter matches.
- It is now possible to use multiple parser expressions in a single parser object, similar to rewrite rules.
- Added support for using the include statement from anywhere in the configuration file, instead of only at top-level. Also introduced syslog-ng “global values” that can be defined and the substituted anywhere in the configuration file.
- Default configuration file supplied as part of SCL.
Incompatible changes:
- syslog-ng traditionally expected an optional hostname field even when a syslog message is received on a local transport (e.g. /dev/log). However no UNIX version is known to include this field. This caused problems when the application creating the log message has a space in its program name field. This behaviour has been changed for the unix-stream/unix-dgram/pipe drivers if the config version is 3.2 and can be restored by using an explicit ‘expect-hostname’ flag for the specific source.
Changes since 3.2alpha1:
- Now compiles on all platforms and the unit/functional tests also run. (tested: AIX, HP-UX, Solaris, FreeBSD, Linux, Tru64)
- Fixed pdbtool match –debug-pattern output for ESTRING parsers.
- Fixed a possible memory leak in the lexer, which would accumulate in case SIGHUPs.
- Fixed Solaris STREAMS device support.
- For
ward ported all bugfixes from syslog-ng OSE 3.0 & 3.1 - Disable process accounting module by default as it doesn’t compile on non-Linux platforms.
- Added “pdbtool match –file” option to read and parse an existing logfile.
- Added “pdbtool test” to check the log samples in the patterndb file.
- Added “dont-create-tables” flag for the SQL destination to inhibit automatic table creation.
- Added “condition()” support for rewrite expressions, which makes it possible to skip rewrite rules that do not match a filter expression.
- Added “–module-path” command line option to control where modules are loaded from from the command line.
Happy logging!
syslog-ng OSE 3.2 caveats
Starting with syslog-ng OSE 3.2, syslog-ng became plugin based, which has some consequences that even experienced syslog-ng users may stumble into.
The most obvious one, is that syslog-ng now produces a series of .so files loaded at runtime, instead of being a monolithic executable. If a given .so is not not or not loaded, some of the functionality may be missing. This usually manifests itself by a syntax error when parsing the configuration file.
Second: if you compile syslog-ng from source, the unit/functional test programs also want to load plugins, and they try to do that from the install directory. This means that you first have to install syslog-ng using “make install” before running the testsuite. This is not an ideal solution, but should work for now.
Plugins are loaded from $prefix/lib/syslog-ng by default, however this can be changed using the `module-path` global, which contains the list of directories where syslog-ng should look for modules. You can change this using the syntax:
@define module-path `module-path`:/usr/local/lib/syslog-ng-plugins
I think that’s it for now.