[logs] open source artificial ignorance-like systems

[Stefano Zanero]

Tom Le wrote:

> I don't think that at all.  I said the problem with most AI approaches
> (to log analysis) is that it requires training.

Well, actually you mentioned "networks" twice... but still, even this
sentence is only historically correct: most approaches have been focused
on supervised learning.

Where I disagree with you is on the following:

>  Unsupervised training
> can help with feature extraction and anomaly detection if you can
> provide the proper apriori context with regards to meta data.

UN-supervised learning is, well, unsupervised, meaning you don't give it
anything a priori.

> monitoring, the only way to produce meaningfuł actionable results is
> with supervised training.

Alt. What do you mean by "actionable" ? Seemingly, I and Anton Chuvakin
have two totally different conceptions of that word, so I may or may not
agree with you depending on the definition.

> I would welcome any examples of SEM for unsupervised training 

SmartSifter, for instance ?

It's not even "unsupervised training", it's a discounting learning
outlier detector, so it does not even need "training" in the strict
meaning of the word.

> Here the human is the supervisor even
> if the AI algorithm uses unsupervised training.

You are confusing two different meanings of "supervisor". Any
UNSUPERVISED system is attended by a human, but you should call that
human differently than supervisor :D

> I disagree with you.  My reality includes analyzing billions of log
> events per week in an SEM context.  I would appreciate examples of
> your position.

Analyzing billions of log events per week is an awful problem, but it's
not a scientific position by itself.

You may very well state that "there are no tools out there that use
unsupervised learning", and you would be correct. But you should not
imply that they cannot exist. They do, and if you look closely at
scientific literature it will be easy to see how they can be built.

And, as MJR said in an inspired answer... supervised learning isn't
exactly different from rule-based systems: still misuse detection, even
if in a more clever format perhaps.

Best,
Stefanoù