[logs] Analyzing tons of logs

[Desai, Ashish]

The million dollar question ;-) I guess you guys finally have a big case
of someone stealing something and the company
wants to know the damage, etc.
 
See if http://www.splunk.com  works.  You might want to dump your binary
files tcpdump, pix logs into text and then run splunk over it.
 
You are going to have to brute force this assuming you don't have the
luxary of time to build a database, etc. That take time.
 
1. Setup a good filesystem structure.  That helps a lot
/YYYY/MM/DD/datasource might be a good start
2. Dump all the files in the directory system. Then extract the text
parts into the same directory.
3. Then start running "fgrep" to narrow down the files files you really
need.
4. Send those files to splunk  or a database if you feel like it.
 
If you have money go buy this http://www.sun.com/servers/x64/x4500/
You will get 24Tb of storage for $60K and a very fast search capability.
 
 
Have fun.
 
Ashish Desai
Internet Channel Security
Fidelity Investments
 
 


  _____  

	From: loganalysis-bounces at loganalysis.org
[mailto:loganalysis-bounces at loganalysis.org] On Behalf Of Chetan Gupta
	Sent: Wednesday, March 28, 2007 8:10 AM
	To: loganalysis at loganalysis.org
	Subject: [logs] Analyzing tons of logs
	
	

	Dear List Members, 
	
	I am looking for opinion from the experts for a particluar
problem. 
	
	How do we go about log analysis if we have tons (maybe in
trillions) of logs from lets say tcpdump (raw logs) or some firewall
(like netscreen or pix)? 
	What would be the best way to normalize and analyze these logs
in the shortest possible time? 
	Import them into a database? Use a commercial application like
arcsight? loglogic? simple text editor like editplus? 
	Any suggestions/comments would be appreciated. 
	
	Regards, 
	
	Thanks and Regards,
	ERNST & YOUNG (r)
	Ernst & Young Pvt. Ltd
	
	Chetan Gupta
	Consultant
	Risk and Business Solutions
	FIDS 
	_______________________________________________________
	
	          
	Mobile:      +91 - 9810718489
	Fax:          +91 - 11 - 2661 1012           
	URL:          http://www.ey.com/in
	_______________________________________________________
	
	
	----------------------------------------------------------
	The information contained in this communication is intended
solely for the use of the individual or entity to whom it is addressed
and others authorized to receive it. It may contain confidential or
legally privileged information. If you are not the intended recipient
you are hereby notified that any disclosure, copying, distribution or
taking any action in reliance on the contents of this information is
strictly prohibited and may be unlawful. If you have received this
communication in error, please notify us immediately by responding to
this email and then delete it from your system. Ernst & Young is neither
liable for the proper and complete transmission of the information
contained in this communication nor for any delay in its receipt.
	

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.loganalysis.org/pipermail/loganalysis/attachments/20070328/22d4967a/attachment.html