[logs] open source artificial ignorance-like systems

[Tom Le]

The problem with most AI approaches is you have to train the network, and in
order to do that you have to provide accurate examples (not to mention all
of the usual issues of normalizing data).

There are some limited things you can do with vary narrow scopes, like
building AI models are just srcIP, dstIP, and dstPort activity.  This can be
done with firewall traffic logs, netflow, router ACL records, etc.

But when you start talking about host based messages or IDS/IPS events, it's
just impossible to train the network on all the possible combination of
events and criteria (e.g. ignore these types of events under these
conditions... aka "tuning").

Almost all correlation today is either focused on anomaly/heuristics type
detection or a priori pattern matching of some kind.  Pattern matching works
great, if you believe the vendor has accurate pattern matching and can
factor in your tuning requirements (e.g. events and assets to white list,
black list, etc.).  But to answer the OP's question, none of the good stuff
is open source.  You can check out OSSEC which does some correlation between
different types of events.


On 4/17/07, Safier, Adam * <adam.safier at fda.hhs.gov> wrote:
>
> I have to wonder if anyone has approached the Artificial Intelligence
> crowd with log analysis questions like this.  It seems this would be
> right up their alley.  I would think that LISP might be decent at
> processing something like a log file, though I admit I don't know enough
> to know which AI tree to bark up.
>
> Adam
>
> -----Original Message-----
> From: loganalysis-bounces at loganalysis.org
> [mailto:loganalysis-bounces at loganalysis.org] On Behalf Of Chris Buechler
> Sent: Tuesday, April 17, 2007 2:45 PM
> To: loganalysis at loganalysis.org
> Subject: [logs] open source artificial ignorance-like systems
>
> Likely everyone here has read MJR's artificial ignorance guide, but just
> in case you haven't:
> http://www.ranum.com/security/computer_security/papers/ai/index.html
>
> What I've spent a lot of time looking for, with no success, is an open
> source system that will implement something similar but smarter than a
> simple grep from cron. What I ideally want is something that reads logs
> in real time, and has a list of "rules", if you will, which would be a
> list of regex's with an action for each. Like always ignore xyz.*,
> ignore abcd.* unless it happens more than X times in Y time frame
> (minutes, hours), etc. and a default alert rule for anything that
> doesn't match any of the previous rules, since it's something I'm not
> expecting.
>
> Seems pretty simple, but a lot of searching and browsing loganalysis.org
> and similar sites has left me with nothing. I could hack together
> something with standard Unix tools that would meet most of these
> requirements. But I figure as simple and useful as this is, somebody has
> probably already put together an open source package that does what I
> describe, I just can't find it.
>
> Any suggestions would be appreciated. OS, if it matters, could be either
> FreeBSD or Linux.
>
> Cheers,
> Chris
>
> _______________________________________________
> LogAnalysis mailing list
> LogAnalysis at loganalysis.org
> http://www.loganalysis.org/mailman/listinfo/loganalysis
>
>
>
> _______________________________________________
> LogAnalysis mailing list
> LogAnalysis at loganalysis.org
> http://www.loganalysis.org/mailman/listinfo/loganalysis
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.loganalysis.org/pipermail/loganalysis/attachments/20070417/e7d40a7e/attachment-0001.html