[logs] regexless parsing, again?

[Rainer Gerhards]

Hi Morty,

I jump in this thread without having read all before (was busy with
other things the past days).

I basically agree, but, as you say, the problem is there will be no
standard soon. The IETF netconf working groupd might be seeding a new,
XML based approach, which in the looooong term could become such a
standard. But even there, data modeling efforts have a very hard start.

Rainer

> -----Original Message-----
> From: loganalysis-bounces at loganalysis.org [mailto:loganalysis-
> bounces at loganalysis.org] On Behalf Of Mordechai T. Abzug
> Sent: Thursday, September 20, 2007 2:59 AM
> To: Marcus J. Ranum
> Cc: Desai, Ashish; loganalysis at loganalysis.org
> Subject: Re: Re: [logs] regexless parsing, again?
> 
> On Tue, Sep 18, 2007 at 12:51:13PM -0400, Marcus J. Ranum wrote:
> 
> > >> There have been some amazing advances in hardware to do PCRE.
> 
> > That'd the the "hardware turbo-charged lipstick on a pig" option.
> 
> Even parsing is lipstick on a pig.  The correct solution is to
> redesign all log systems so log events are structured, and come with
> machine-readable catalogs, a la SNMP trap.  [Yes, SNMP sucks in
> general, but the SNMP community dodged most log analysis problems 10+
> years ago, such that SNMP analysis solutions have been ahead of syslog
> analysis solutions for many years, IME.]
> 
> Too bad unstructured syslog isn't going to go away any time soon.  Any
> attempt to analyze unstructured data is "lipstick on a pig" compared
> to importing vendor log catalogs in a machine-readable, vendor-neutral
> format (such as *spit* ASN.1 *spit*).  Whether you use regexes,
> parsing, hybrid multi-layer techniques, or a rigidly structured format
> with event catalogs, it's all a trade-off between log analysis system
> design complexity, log analysis system performance, log analysis user
> burden, log compatibility with "legacy" log systems, and comfort of
> the developers generating the logs.
> 
> [Cue a bunch of starry-eyed innocents talking up CEE, CEF, and other
> "standards" that are supposed to make unstructured logs go away Real
> Soon Now.]
> 
> - Morty
> _______________________________________________
> LogAnalysis mailing list
> LogAnalysis at loganalysis.org
> http://www.loganalysis.org/mailman/listinfo/loganalysis