[logs] Analyzing tons of logs

[louie bounassif]

Hi,
Logpp is a tool for preprocessing event logs and feeding relevant data
to other programs for storing or in-depth analysis. Logpp reads lines
appended to input files, matches the lines with patterns, and writes the
results to given destinations. 

It completely written in C and uses the PCRE library.
I have tested it and achieved outstanding results but I can't recall
exactly what they were! 

Remember the Code is still Alpha code. 

Here is the link to the tool:
http://prdownloads.sourceforge.net/logpp

Its written by Risto Vaarandi the same guy who wrote SEC (Simple Event
Correlator)... in fact you may event want to check that out here
http://www.estpak.ee/~risto/sec/

SEC (written in perl) is a wonderful tool for correlating data, however
recently there has been continuous discussion regarding SEC's
performance...from these discussions Logpp was born as a high
performance log analysis plugin to SEC mainly but it can be used
separately actually. 


Regards,
louieb


-----Original Message-----
From: louie bounassif 
Sent: Thursday, 29 March 2007 2:31 PM
To: 'Anton Chuvakin'; Chetan Gupta
Cc: loganalysis at loganalysis.org
Subject: RE: [logs] Analyzing tons of logs

Hi All,
Try http://prdownloads.sourceforge.net/logpp/logpp-0.12.tar.gz
Its still Alpha code but its worth a try.

Regards,
louieb

-----Original Message-----
From: loganalysis-bounces at loganalysis.org
[mailto:loganalysis-bounces at loganalysis.org] On Behalf Of Anton Chuvakin
Sent: Thursday, 29 March 2007 1:20 PM
To: Chetan Gupta
Cc: loganalysis at loganalysis.org
Subject: Re: [logs] Analyzing tons of logs

Chetan and all,

> How do we go about log analysis if we have tons (maybe in trillions)
of logs
> from lets say tcpdump (raw logs) or some firewall (like netscreen or
pix)?
> What would be the best way to normalize and analyze these logs in the
> shortest possible time?

Let's see here: assuming 1 trillions records of 200 bytes (typical
PIX, way too small for a packet), we are looking at roughly 180TB of
data. To analyze... not just to store.

So, I have a sneaking suspicion that ALL the mentioned solutions will
fail miserably, albeit without embarrassing their creators (cause
that's a looooooooooot of data!). I have to admit that Jose is
probably right: you might need to write some purpose-specific code
here. Look up some old posts by Marcus Ranum (here
http://www.andrews.hu/guru/msg583.html and around) for useful tips on
super-fast but purpose-specific log processing.

Best,
-- 
Anton Chuvakin, Ph.D., GCIA, GCIH, GCFA
      http://www.chuvakin.org
  http://chuvakin.blogspot.com
    http://www.info-secure.org
_______________________________________________
LogAnalysis mailing list
LogAnalysis at loganalysis.org
http://www.loganalysis.org/mailman/listinfo/loganalysis