[logs] RE: Analyzing tons of logs

[Jamie Tyler]

I apologize for replying to a late thread. I debated as to whether I 
should, but given the recent tests and research that I have been 
conducting for the US Gov I felt compelled to share my findings. My 
research has uncovered a product, previously developed by the US 
Government (as a database), but now licensed and enhanced for commercial 
use through a company called Nitro Security. My benchmarks, which I was 
highly skeptical of initially when presented as vendor claims, do show 
their back end technology to truly be between 950-980% faster then my 
standard Oracle and MySQL deployments (~80k inserts per second) that I 
oversee. I've had Oracle professional services in house to attempt to 
tweak their database to better fit my needs as a backend for ArcSight 
with no avail as well. What I've seen is that the feature set that Nitro 
provides, albeit, somewhat limited compared to ArcSight in the high 
level heuristic and reporting world (which is rarely all that accurate 
in enterprise products anyway) to provide data analysis of literally 
trillions of events and flows in a matter of seconds and minutes. The 
anomaly detection and correlated baselines are something that I haven't 
seen demonstrated in any other product. I don't want to sound like a 
shill for this company in any way, but simply want to report the unique 
performance and results that I have seen per my own testing. Despite the 
maturity of the product, the database foundation (which they also 
utilize custom solid state drives in some applications) enables 
functionality like I've never seen before. I really can't pass along to 
many detailse about my testing is it has occurred under contract, but I 
felt compelled to pass along the fact that it has resulted in some 
highly unique results. Thanks for the time.

Best Regards,
Jamie Tyler, CISSP, MCSE

-----

Dear List Members,
 
I am looking for opinion from the experts for a particluar problem.
 
How do we go about log analysis if we have tons (maybe in trillions) of
logs from lets say tcpdump (raw logs) or some firewall (like netscreen or
pix)?
What would be the best way to normalize and analyze these logs in the
shortest possible time?
Import them into a database? Use a commercial application like arcsight?
loglogic? simple text editor like editplus?
Any suggestions/comments would be appreciated.
 
Regards,
 
Thanks and Regards,
ERNST & YOUNG ®
Ernst & Young Pvt. Ltd
 
Chetan Gupta
Consultant
Risk and Business Solutions
FIDS
_______________________________________________________
 
         
Mobile:      +91 - 9810718489
Fax:          +91 - 11 - 2661 1012          
URL:          http://www.ey.com/in


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.loganalysis.org/pipermail/loganalysis/attachments/20070606/757a855d/attachment-0001.html