[logs] Syslog - monitoring the bigger picture

[Tina Bird]

> Just curious, what is wrong with all other past logging projects, that
> seek to accomplish just about the same?

now *there's* a question for the ages.

what i usually tell my students:

"some junior sys admin who just took a perl/C/lisp/whatever class is tasked
with centralizing and monitoring system logs at her job. after getting all
the data centralized, she begins to look at the data and to try to
understand what's important and what isn't.

once she starts figuring out what she needs in terms of reports and alerts,
she hits the net, because *surely* this is a solved problem. she gets a lot
of google hits for swatch, so she starts there -- but it doesn't handle
thresholding traffic, which is critical for the firewall, and the job of
coming up with all the right keywords is less than exciting. so she asks
around a bit, and someone mentions logsurfer, cos it lets you deal with
multi-line messages and context, so she can get the contexting she needs.

but logsurfer is pretty complicated for what she needs, and she can't tell
whether it's still being maintained, or, for that matter, if it *matters*
whether or not it's still being maintained. and her manager keeps asking for
progress reports.

so finally she throws together a little script/program/spell that does
*exactly* what her organization needs. it may be tweaked for a particular
vendor's products, or the specific reporting requirements of her industry,
or whatever. but it does the trick for her and her manager is happy and she
can finally get onto something more interesting.

then she posts it on her website, and (with any luck) pretty much forgets
about it for the rest of her life."

...lather, rinse, repeat.

let me pre-emptively apologize to all list members who have contributed to
the log analysis and management tools that are out there. it's clear that
many of the folks involved in these projects take a much more big-picture
view than i've painted here, and i don't mean to smear them with the same
brush.

but wow, after 5 years of trying to maintain an up-to-date list of log
parsing and analysis tools -- which included attempting to figure out how
they differed from each other -- this is the only explanation i've been able
to come up with.

cheers -- tbird