[logs] regexless parsing, again?

[Mordechai T. Abzug]

On Tue, Sep 18, 2007 at 12:51:13PM -0400, Marcus J. Ranum wrote:

> >> There have been some amazing advances in hardware to do PCRE.

> That'd the the "hardware turbo-charged lipstick on a pig" option.

Even parsing is lipstick on a pig.  The correct solution is to
redesign all log systems so log events are structured, and come with
machine-readable catalogs, a la SNMP trap.  [Yes, SNMP sucks in
general, but the SNMP community dodged most log analysis problems 10+
years ago, such that SNMP analysis solutions have been ahead of syslog
analysis solutions for many years, IME.]

Too bad unstructured syslog isn't going to go away any time soon.  Any
attempt to analyze unstructured data is "lipstick on a pig" compared
to importing vendor log catalogs in a machine-readable, vendor-neutral
format (such as *spit* ASN.1 *spit*).  Whether you use regexes,
parsing, hybrid multi-layer techniques, or a rigidly structured format
with event catalogs, it's all a trade-off between log analysis system
design complexity, log analysis system performance, log analysis user
burden, log compatibility with "legacy" log systems, and comfort of
the developers generating the logs.

[Cue a bunch of starry-eyed innocents talking up CEE, CEF, and other
"standards" that are supposed to make unstructured logs go away Real
Soon Now.]

- Morty