ABOUT
Project Lumberjack to improve Linux logging
In a lively discussion at the RedHat offices two weeks ago in Brno, a number of well respected individuals were discussing how logging in general, and Linux logging in particular could be improved. As you may have guessed I was invited because of syslog-ng, but representatives of other logging related projects were also in nice numbers: Steve Grubb (auditd), Lennart Poettering (systemd, journald), Rainer Gerhards (rsyslog), William Heinbockel (CEE, Mitre) and a number of nice people from the RedHat team.
We discussed a couple of pain points for logging, logging is usually an afterthought during development, computer based processing, correllation of application logs is nearly impossible. We roughly agreed that the key to improve the situation is to involve the community at large, initiate a momentum and try to get application developers on board and have them create structured logs. We also agreed that this will not happen overnight and we need to take a gradual approach.
To move into that direction, the benefits of good logging needs to be communicated and delivered to both application developers and their users.
We also talked about what kind of building blocks are needed to deliver a solution fast, and concluded that we basically have everything available, and even better they are open source. The key is to tie these components together, document best practices and perhaps provide better integration.
Thus project Lumberjack was born, hosted as a Fedora project at https://fedorahosted.org/lumberjack/.
The building blocks that need some care are:
- some applications already produce logs in structured format, those should be integrated (auditd for instance)
- we need to define a mechanism to submit structured logs to local logging services for further processing (ELAPI and some enhanced syslog)
- we need to make sure that local logging services cope with structured data (already available for a long time now)
- we need to define a mechanism to store messages in a structured form and a way query them
- last, but not least we need to define a naming scheme for event data which CEE can bring to the table
Most of these is already possible by using a combination of tools and proper configuration, however learning how to do this is not a trivial undertaking for those who only want to develop or use applications.
Changing that is the primary aim of Project Lumberjack. If you are interested in logging, make sure to check that out.
[...] may remember the Lumberjack project I wrote about earlier. It is an attempt to improve system logging by creating conventions and standards to cover [...]
[...] How project Lumberjack can improve logging [...]
Bambano wrote:
> http://www.change.org/petitions/lennart-poettering-stop-writing-useless-programs-systemd-journal
>
> mindent elmond.
Marmint?
Bambano wrote:
> Miért van rossz érzésem…
> a unix alapvetően sor alapú ascii szövegfájlok
> feldolgozására épül. amikor ilyen strukturált
> logbejegyzést olvasok, mindjárt egy volt kollégám
> jut eszembe,aki képtelen volt elfogadni, hogy
> xml-en kívül más fájlformátum is létezik, pláne azt,
> hogy xml-en kívül létezhet fájlformátum, amit
> sokkal könnyebb értelmezni és felhasználni.
Ebben egyet is ertek veled. Az XML-t konnyu automatiusan feldolgozni, de kezzel irni es olvasni nehez. Ezert van mar YAML es ezert van JSON, mindketto egy picit egyszerubb es atlathatobb.
Mindenesetre a strukturalt logolas nem feltetlenul jelent XML-t, bar jelenthet azt is. A CEE project kapcsan a cel az volt, hogy az uzenet szemantikajat definialjuk, utana a reprezentacio masodlagos. Az lehet akar XML akar JSON, akar egy jol formazott WELF (nev=ertek parok sorozata) is.
Igazabol a strukturalt adatbol a szovegeset is elo lehet allitani, ld a syslog-ng template funkciojat. Gyakorlatilag valami ilyennel tudsz barmilyen formatumot eloallitani _miutan_ eloalltak a strukturalt adatok:
“$DATE $HOST $MSGHDR$MSG\n”
Talan ismeros lehet a UNIX shell-bol, mivel a minta onnan szarmazik. A valtozok pedig a “struktura”.
Talan nem is annyira idegen a UNIX-tol.
>
> Szóval ha egy benzinmotors autóban meghekkelik
> a motort, hogy gázolajjal működjön, gyújtás nélkül,
> akkor az már nem benzinmotoros autó. Az a unix, > amelyik (például) xml-be logol, az már nem unix.
> Felesleges munkára kényszeríteni azokat, akiknek
> tökéletesen megfelel a mostani logolási rendszer,
> nem szép dolog.
Megnyugtatlak, XML-be nem fogunk logolni
> Jó lesz, ha ez a projekt hamvába hal, mielőtt túl
> sok energiát pazarolnak rá.
Nagyon sok duplikacio tortenik es felesleges energia megy el minden program sajat logolasi rendszerenek a kialakitasara. Miert logol az apache, a postfix az exim, az auditd, a squid (es meg sorolhatnam) sajat file-ba?
Ha valakinek az _osszes_ logbol kell adatot kinyernie, azokat korrellalnia, elemeznie, akkor hol vannak ezek a logok? Nem egy trivialis kerdes egy mai rendszeren.
Tritzt wrote:
> Something I worry about when we have RedHat as
> the driving force here is that we will have a logging
> system which has gnome-dependency and which in
> the end will be like microsofts eventviewer.
I have never said that lumberjack aims at using journald, perhaps we might make use of some of its functionality, but for sure as an option. The key here is that we want something out-of-the-box, but you can configure syslog-ng as you wish
It won’t go away.
Some people simply don’t want to mess with an SQL or MongoDB just to get logging going.
>
> Of course I wish I could configure every log source
> (independely of source application) and redirect
> the logs to where I wish and maybe make some
> logs more human readble than they are today.
Well, today most logs are syslogs, the only natural recipient for those is a human, at least it is very difficult to process using automatic tools.
But certainly you won’t lose any of the configurability of syslog-ng just because we would supply a default configuration that would just “work”.
Steveo wrote:
> Is it your intention to make systems development a ‘trivial undertaking’?
I didn’t (or at least didn’t mean to) refer “systems development” that would become trivial. I wanted to say that configuring a system the processes structured logs with all the existing tools is possible, but far from trivial. The aim is to make that simpler.
E.g. by making it simple to:
- use a structured logging API for local applications (on top of syslog)
- supply a default configuration for syslog-ng to process such logs in a meaningful manner
- supply a default configuration for a storage engine (mongodb or perhaps the journal, but as a choice nevertheless) to store structured logs
- provide a way (API or scriptable interface) that makes it simple to query structured logs without having to know site-specifics.
The biggest hurdle with using syslog as an infrastructure for applications that actually want to process their own logs is that multitude of parameters that are needed to access system logs (file name, timestamp format, perhaps customized templates).
See for instance Apache, auditd, squid, postfix or anything that actually _reads_ its own log files, those are using their own solution for the problem, instead of relying on syslog. Certainly they can be configured to use syslog, but that’s not their default behaviour, and I guess most of their log processing tools break in that configuration.
http://www.change.org/petitions/lennart-poettering-stop-writing-useless-programs-systemd-journal
mindent elmond.
Embarassment because of the name mistake. I was so sure about his name that I didnMt validate. Updated he article.
Hi,
should that be s/Steve Gibbs/Steve Grubb/ ?
I have been working with logs for a long time, extending MRTG to work with any log file basically. These days there is a lot of data that can be kept in trees or in file systems… how we abstract it using parallel processing and distributed systems can help us perform predictive analytics on system quality attributes logged over long periods of time.
[...] his blog, syslog-ng developer Balázs Scheidler writes about the birth of Project Lumberjack, which is an effort to improve Linux logging. "In a lively [...]
Miért van rossz érzésem…
a unix alapvetően sor alapú ascii szövegfájlok feldolgozására épül. amikor ilyen strukturált logbejegyzést olvasok, mindjárt egy volt kollégám jut eszembe,aki képtelen volt elfogadni, hogy xml-en kívül más fájlformátum is létezik, pláne azt, hogy xml-en kívül létezhet fájlformátum, amit sokkal könnyebb értelmezni és felhasználni.
Szóval ha egy benzinmotors autóban meghekkelik a motort, hogy gázolajjal működjön, gyújtás nélkül, akkor az már nem benzinmotoros autó. Az a unix, amelyik (például) xml-be logol, az már nem unix. Felesleges munkára kényszeríteni azokat, akiknek tökéletesen megfelel a mostani logolási rendszer, nem szép dolog.
Jó lesz, ha ez a projekt hamvába hal, mielőtt túl sok energiát pazarolnak rá.
Hogy mit írtam ide, nem tudom, mert valaki úgy gondolta, hogy vakító fehér alaplapon nagyon jó ötlet sötétszürke keretben feketével írni. Szerintem nem jó ötlet látássérültet alkalmazni webmesternek, mégha rendkívül pc is.
Something I worry about when we have RedHat as the driving force here is that we will have a logging system which has gnome-dependency and which in the end will be like microsofts eventviewer.
Of course I wish I could configure every log source (independely of source application) and redirect the logs to where I wish and maybe make some logs more human readble than they are today.
[...] Um dieses Problem anzugehen, trafen sich jüngst in den Red-Hat-Räumlichkeiten in Brno einige interessierte Entwickler, darunter Steve Gibbs (auditd), Lennart Poettering (systemd, journald), Balázs Scheidler (syslog-ng), Rainer Gerhards (rsyslog), William Heinbockel (CEE, Mitre) sowie einige Red-Hat-Entwickler. Davon berichtet Scheidler, der Gründer von BalaBit, jetzt in seinem Blog. [...]
[...] Via | Balázs Scheidler [...]
[...] Via | Balázs Scheidler [...]
[...] Project Lumberjack to improve Linux logging In a lively discussion at the RedHat offices two weeks ago in Brno, a number of well respected individuals were discussing how logging in general, and Linux logging in particular could be improved. As you may have guessed I was invited because of syslog-ng, but representatives of other logging related projects were also in nice numbers: Steve Gibbs (auditd), Lennart Poettering (systemd, journald), Rainer Gerhards (rsyslog), William Heinbockel (CEE, Mitre) and a number of nice people from the RedHat team. [...]
Is it your intention to make systems development a ‘trivial undertaking’?
Good luck.
[...] weitere Hintergründe finden sich in einem Blog-Eintrag von Balázs Scheidler, dem leitenden Entwickler des bei einigen Mainstream-Linux-Distributionen [...]
[...] | Balázs Scheidler (1 Voti | Media: 5 su 5) 0 [...]
[...] Via | Balázs Scheidler [...]
[...] Via | Balázs Scheidler [...]