[logs] open source artificial ignorance-like systems

[Safier, Adam *]

What I was thinking was that the AI language rules would parse the log
records into a common storage format.  A database would be hard to
modify for every new definition but many components are the same - date,
time, #, string message.  Maybe moving to something like XML where you
could add definitions would work, though I don't know how you would
store large data volumes.

Probably a combination of tools would be necessary:  simple ignore rules
to knock down the easy 80%+, then convert to XML so every component of
the log line gets a label or gets kicked out to the rule maintainers,
then check alarm rules and do event correlation on all the items that
now have nice labels.

It's been way too long since I touched code, so I'm not going to think
about performance.

Adam - sliding back into lurk mode.

-----Original Message-----
From: loganalysis-bounces at loganalysis.org
[mailto:loganalysis-bounces at loganalysis.org] On Behalf Of Marcus J.
Ranum
Sent: Tuesday, April 17, 2007 8:30 PM
To: Safier, Adam *; loganalysis at loganalysis.org
Subject: RE: [logs] open source artificial ignorance-like systems

Safier, Adam * wrote:
>I have to wonder if anyone has approached the Artificial Intelligence 
>crowd with log analysis questions like this.  It seems this would be 
>right up their alley.  I would think that LISP might be decent at 
>processing something like a log file, though I admit I don't know 
>enough to know which AI tree to bark up.


The problem is that logs are really a form of communication - which
means they're a language problem. AI can do some interesting things with
language but having a grammar (even approximate) for the language is
critical. UNfortunately for us log analysts there isn't anything like an
actual logging language. There's a vocabulary, but the vocabulary isn't
used consistently. :(
I.e.: what happens when you have a host named "root"?
You find that any word in the log vocabularly can mean anything, which
means that they all mean nothing.

mjr. 

_______________________________________________
LogAnalysis mailing list
LogAnalysis at loganalysis.org
http://www.loganalysis.org/mailman/listinfo/loganalysis