[logs] History of Log Analysis and Modern Search Engine

[Greg Dotoli]

Anton,

I like the 1913 definition of search, I'd like to
understand how log searching from the early late 60s
and Early 70s, morphed into the algorithms that make
up the search engine base. 

I believe the earlier 60s/70s analysis was more
intrusion related  and existed in the server centric
model of the time; while the newer Client/Server
models of log analysis have morphed into business
intelligence. Our IDS systems are merely BI data-marts
seeking malicious traffic or host actions via
correlation and alert. 
The Internet search engine base locates information
via crawl engines, although when seen closely , all
log analysis or operating system is algorithmic
op-code running through a CPU.


My objective is to create a log analysis history in
Wikipedia for everyone.






Search

Search \Search\, v. t. [imp. & p. p. {Searched}; p.
pr. & vb. n.
   {Searching}.] [OE. serchen, cerchen, OF. cerchier,
F.
   chercher, L. circare to go about, fr. L. circum,
circa,
   around. See {Circle}.]
        
   Syn: To explore; examine; scrutinize; seek;
investigate; pry into; inquire.
        [1913 Webster]

	-- From The Collaborative International Dictionary of
English v.0.48

   Syn: Scrutiny; examination; exploration;
investigation; research; inquiry; quest; pursuit.
        [1913 Webster]

	-- From The Collaborative International Dictionary of
English v.0.48


search
  
2: an investigation seeking answers; "a thorough
search of the ledgers revealed nothing"; "the outcome
justified the search"
     3: an operation that determines whether one or
more of a set of items has a specified property; "they
wrote a program to do a table lookup" [syn: {lookup}]
     


127 Moby Thesaurus words for "search":
   analysis, angle for, ask for, beat, beat about for,
   beat the bushes, burrow, cast about, chase, check,
chivy, comb, delve, delve for, delve into, dig, dig
for, dig into, dog, domiciliary visit, dragnet,
enquiry, examination, examine, exploration, explore,
fan, fathom, fish for, follow, follow up,forage,
frisk, give chase, go after, go gunning for, go
into,go through, grub, gun for, hollo after, hound,
house-search, hunt, hunt for, hunt up, hunting,
indagate, inquire of, inquiry, inspect,inspection,
investigate, look, look around, look at, look for,
look into, look over, look round, look through, look
up, make after, nose around, peer into, perquisition,
perusal, plumb, poke, poke around, poke into, posse,
probe, prosecute, prowl after,pry, pry into, pursual,
pursuance, pursue, pursuing, pursuit, quest, quest
after, raise the hunt, rake, ransack, ransacking,
research, researching, root, rummage, run after, run
down, scan, scour, scouring, scout out, scrimmage,
scrutinize, scrutiny, search for, search into, search
party, search through,search warrant,
earch-and-destroy operation, searching, see to, seek,
seek for, seek out, seeking, shake down, sift,
sifting, skirmish, smell around, sound, stalk,
stalking, still hunt,still-hunt, study, take out
after, try to find, turning over



--- Anton Chuvakin <anton at chuvakin.org> wrote:

> > So how exactly are you defining "searching"?
> 
> Awesome question, actually. Let the discussion
> start.
> 
> When I think of log searching I imagine typing a
> keyword (or a set of
> keywords or maybe a regex) into a command line (web
> form, etc) and
> then seeing the log records that match (or NOT
> match) the above search
> expression.
> 
> > codes; and you'd probably have clues provided by
> the way in which the
> > compromise was discovered.
> 
> Exactly - I think that you can SEARCH for clues
> later in the
> investigation (e.g. do we see the same log traces on
> other systems,> etc), but won't really help me to
analyze how/why > the intrusion
> occurs.
> 
> > change the nature of my position. Whoever is
> paying me, I would still
> > consider artificial ignorance techniques to be
> search methods, tailored to
> > simplify the job of identifying significant
> events, as well as
> > never-before-seen entries.]
> 
> Ah, that is much broader definition than mine, for
> sure.
> 
> Mandatory (!) disclosure: I work for LogLogic, as I
> thought everybody
> knows :-) In any case, my corporate affiliation is
> publicly disclosed
> on my site http://www.chuvakin.org
> 
> Best,
> -- 
> Anton Chuvakin, Ph.D., GCIA, GCIH, GCFA
>       http://www.chuvakin.org
>   http://chuvakin.blogspot.com
>     http://www.info-secure.org
> _______________________________________________
> LogAnalysis mailing list
> LogAnalysis at loganalysis.org
>
http://www.loganalysis.org/mailman/listinfo/loganalysis
>