[logs] History of Log Analysis and Modern Search Engine

[Tom Le]

On 7/6/07, Anton Chuvakin <anton at chuvakin.org> wrote:
> When I think of log searching I imagine typing a keyword (or a set of
> keywords or maybe a regex) into a command line (web form, etc) and
> then seeing the log records that match (or NOT match) the above search
> expression.

This is just the first level search capability.  Obviously you first need
the ability to search indexed data in some useful way which just standard
syslog-ng and grep is very poor at without indexing.  A definition of
"search" needs to include deeper levels of search beyond just querying
indexed data.
To obtain useful data from logs - whether you are performing real-time
monitoring, producing compliance reports, or just ad hoc queries - requires
you to have the ability to search for more information than just available
via free form or regex queries.  For example, can you run a query for "all
successful logins for user 'administrator'" without just searching on
"administrator"?  You need the ability to apply metadata analysis to your
log data (there are many approaches with different levels of cost &
complexity).  You then need the ability to analyze the metadata from the
returned search query - e.g. run pattern analysis on event frequency, time
proximity, asset profile, known usage baselines, acceptable use policies,
etc.
.
I would also add that my definition of searching does not require the user
to be an expert in the content or syntax/grammar of the log data.  Your
security admin may know what a 'failed login' looks like for Windows, but
what about Unix, SAP, Oracle, AS400 or OS390?  The user should be able to
specify "show me failed logins for user X" in some easy to search
capability.

> Exactly - I think that you can SEARCH for clues later in the
> investigation (e.g. do we see the same log traces on other systems,
> etc), but won't really help me to analyze how/why the intrusion
> occurs.

I disagree.  There are many ways to search for direct or indirect evidence
of an attack by doing nothing but log searching not "later in the
investigation" but as the primary analysis step at the beginning of the
investigation.  A lot depends on what data is available and reliability of
the data (and your ability to detect/eliminate compromised log data).

I know this is nothing new for this group, but some examples of ways to use
search to analyze how/why/when the intrusion occurred:  You can find access
pattern anomalies with firewall, router, web server, and database logs.  You
can track user and resource access requests (successful and failed).  You
can "search" the objects on the system themselves for specific attributes to
detect known trojans, rootkits, viruses, and other signs of system
compromise (okay, I admit this isn't pure "log searching" but you can apply
the same search methodology to an "ls -laR" file output). You can
cross-reference IDS logs and logs from first-hop neighbors to the attacked
host.  You can cross-reference against honetpot data.  You can look at
vulnerability scan data or recommended patch reports to determine potential
attack vectors and focus your search there.  For example, you have a report
that says you have vulnerable version of openSSH or Apache - then focus your
search to logs and objects directly related to that attack vector.  You can
look at *all* openSSH and Apache logs on other systems for signs of
reconnaissance or failed access activity.

My point here is that there are many ways to find evidence of the attack
itself or the footprints of an attack before or after the incident occurred
by doing nothing by searching logs.
Tom

Disclaimer: I work for BT Counterpane.  We do all of the above on a regular
basis and do leverage products from companies with participants on this
list.  But commercial tools are not required; they just make what you do
easier, faster, and more scalable (in exchange for $).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.loganalysis.org/pipermail/loganalysis/attachments/20070706/f5524ce8/attachment-0001.html