[logs] open source artificial ignorance-like systems

[Joe_Wulf]

I'd like to offer another perspective.......... what about a 'dictionary'?
All the OS's have 'some' elements of commonality.  Each OS vender has common
areas.
Create an analysis of log information that is common and stabilize its 'format',
even
to the 'bit' level where applicable, and document it.  That is then standard
parse able.

R,
-Joe Wulf, CISSP, USN (RET)
 ProSync Technology Group, LLC
 ProSync 
 www.prosync.com
 Senior IA Engineer

-----Original Message-----
From: loganalysis-bounces at loganalysis.org
[mailto:loganalysis-bounces at loganalysis.org] On Behalf Of Safier, Adam *
Sent: Wednesday, April 18, 2007 21:26
To: Marcus J. Ranum; loganalysis at loganalysis.org
Subject: RE: [logs] open source artificial ignorance-like systems

What I was thinking was that the AI language rules would parse the log records
into a common storage format.  A database would be hard to modify for every new
definition but many components are the same - date, time, #, string message.
Maybe moving to something like XML where you could add definitions would work,
though I don't know how you would store large data volumes.

Probably a combination of tools would be necessary:  simple ignore rules to knock
down the easy 80%+, then convert to XML so every component of the log line gets a
label or gets kicked out to the rule maintainers, then check alarm rules and do
event correlation on all the items that now have nice labels.

It's been way too long since I touched code, so I'm not going to think about
performance.

Adam - sliding back into lurk mode.

-----Original Message-----
From: loganalysis-bounces at loganalysis.org
[mailto:loganalysis-bounces at loganalysis.org] On Behalf Of Marcus J.
Ranum
Sent: Tuesday, April 17, 2007 8:30 PM
To: Safier, Adam *; loganalysis at loganalysis.org
Subject: RE: [logs] open source artificial ignorance-like systems

Safier, Adam * wrote:
>I have to wonder if anyone has approached the Artificial Intelligence 
>crowd with log analysis questions like this.  It seems this would be 
>right up their alley.  I would think that LISP might be decent at 
>processing something like a log file, though I admit I don't know 
>enough to know which AI tree to bark up.


The problem is that logs are really a form of communication - which means they're
a language problem. AI can do some interesting things with language but having a
grammar (even approximate) for the language is critical. UNfortunately for us log
analysts there isn't anything like an actual logging language. There's a
vocabulary, but the vocabulary isn't used consistently. :(
I.e.: what happens when you have a host named "root"?
You find that any word in the log vocabularly can mean anything, which means that
they all mean nothing.

mjr. 

_______________________________________________
LogAnalysis mailing list
LogAnalysis at loganalysis.org
http://www.loganalysis.org/mailman/listinfo/loganalysis



_______________________________________________
LogAnalysis mailing list
LogAnalysis at loganalysis.org
http://www.loganalysis.org/mailman/listinfo/loganalysis