[logs] Error messages from syslogd

[Amiran Alavidze]

Hi,

In fact the parsing tree you've mentioned and the regular expressions
can be used together - OSSEC, for example, implements hierarchical
rules that are based on regular expressions. So I guess the
difficulties you mentioned are not RE-related but rather list vs tree
parsing methods related.

http://www.ossec.net/ossec-docs/auscert-2007-dcid.pdf
(see page 32)

Kind regards,
Amiran Alavidze

On 7/12/07, Marcus J. Ranum <mjr at ranum.com> wrote:

> Briefly, here's what needs to be done:
> Instead of trying to build sequences of fall-through matching rules, you
> need to build an acyclic left-to-right parse tree that completely defines
> all the message forms. Abe and I did some experimenting with this
> a few years ago and it turned out there are only about 50,000 variant
> forms of messages - well - 60,000 if you count all the new ones the
> OpenBSD guys appear to have added to syslogd. Obviously, you
> don't need them all at once; you just need the ones you're seeing.
> What's funny is that executing against a parse tree is going to be
> oodles faster than a list of regular expressions once you go above
> a certain number.
>
> I offer as evidence an evil hack I did for a buddy last year, which
> simply builds a nested sequence of calls to sscanf, keeping
> track of the farthest-right point of matching, and walking left
> to right. I  forget the exact number but it was handling something
> like 400,000 log lines per second at 2% CPU utilization (I.e.: it
> was I/O bound) - I am not necessarily suggesting that someone
> build a tool that outputs a C-coded recognizer... but consider
> for a second that if you did, your iPod could handle log data
> rates that make the largest commercial SIMs eat their own
> intestines in sheer agony.