[logs] Analyzing tons of logs

[Abhimanyu Chitoshia]

Hi,

Chetan, if you have a few trillions of logs, manual screnning cannot work.
All the date changes and other normalization activities need to be
automated. Also, since raw traffic dumps are also involved, the field
becomes vast.

But as a consultant you can probably tell the client abt paid stuff that
can analyse logs for forensics, ids, ips, compliance like arcsight, net
forensics... there are cheaper ones like NetIQ.

My favorite though is OSSIM, its an open source security information
manager, just like the paid stuff i mentioned earlier. It will manage a
lot of disparate logs, as an IPS, with forensics capabilities  for
complaince purposes. Will also act as a store house for data.

For more check out the website www.ossim.net or just feel free to ask.

Regards
Abhimanyu


> The million dollar question ;-) I guess you guys finally have a big case
> of someone stealing something and the company
> wants to know the damage, etc.
>
> See if http://www.splunk.com  works.  You might want to dump your binary
> files tcpdump, pix logs into text and then run splunk over it.
>
> You are going to have to brute force this assuming you don't have the
> luxary of time to build a database, etc. That take time.
>
> 1. Setup a good filesystem structure.  That helps a lot
> /YYYY/MM/DD/datasource might be a good start
> 2. Dump all the files in the directory system. Then extract the text
> parts into the same directory.
> 3. Then start running "fgrep" to narrow down the files files you really
> need.
> 4. Send those files to splunk  or a database if you feel like it.
>
> If you have money go buy this http://www.sun.com/servers/x64/x4500/
> You will get 24Tb of storage for $60K and a very fast search capability.
>
>
> Have fun.
>
> Ashish Desai
> Internet Channel Security
> Fidelity Investments
>
>
>
>
>   _____
>
> 	From: loganalysis-bounces at loganalysis.org
> [mailto:loganalysis-bounces at loganalysis.org] On Behalf Of Chetan Gupta
> 	Sent: Wednesday, March 28, 2007 8:10 AM
> 	To: loganalysis at loganalysis.org
> 	Subject: [logs] Analyzing tons of logs
>
>
>
> 	Dear List Members,
>
> 	I am looking for opinion from the experts for a particluar
> problem.
>
> 	How do we go about log analysis if we have tons (maybe in
> trillions) of logs from lets say tcpdump (raw logs) or some firewall
> (like netscreen or pix)?
> 	What would be the best way to normalize and analyze these logs
> in the shortest possible time?
> 	Import them into a database? Use a commercial application like
> arcsight? loglogic? simple text editor like editplus?
> 	Any suggestions/comments would be appreciated.
>
> 	Regards,
>
> 	Thanks and Regards,
> 	ERNST & YOUNG (r)
> 	Ernst & Young Pvt. Ltd
>
> 	Chetan Gupta
> 	Consultant
> 	Risk and Business Solutions
> 	FIDS
> 	_______________________________________________________
>
>
> 	Mobile:      +91 - 9810718489
> 	Fax:          +91 - 11 - 2661 1012
> 	URL:          http://www.ey.com/in
> 	_______________________________________________________
>
>
> 	----------------------------------------------------------
> 	The information contained in this communication is intended
> solely for the use of the individual or entity to whom it is addressed
> and others authorized to receive it. It may contain confidential or
> legally privileged information. If you are not the intended recipient
> you are hereby notified that any disclosure, copying, distribution or
> taking any action in reliance on the contents of this information is
> strictly prohibited and may be unlawful. If you have received this
> communication in error, please notify us immediately by responding to
> this email and then delete it from your system. Ernst & Young is neither
> liable for the proper and complete transmission of the information
> contained in this communication nor for any delay in its receipt.
>
>
> _______________________________________________
> LogAnalysis mailing list
> LogAnalysis at loganalysis.org
> http://www.loganalysis.org/mailman/listinfo/loganalysis


Thanks & Regards
Abhimanyu Chitoshia
Associate Consultant
Mahindra Special Services Group