[logs] open source artificial ignorance-like systems

[Tom Le]

On 4/19/07, Stefano Zanero <zanero at elet.polimi.it> wrote:

> Tom Le wrote:
> > The problem with most AI approaches is you have to train the network,
> > and in order to do that you have to provide accurate examples (not to
>
> You seem to think that AI means neural networks: just to let you know,
> neural networks are a common instrument in data analysis, and are not
> even strictly part of AI anymore...

I don't think that at all.  I said the problem with most AI approaches
(to log analysis) is that it requires training.  Unsupervised training
can help with feature extraction and anomaly detection if you can
provide the proper apriori context with regards to meta data.

But if you are talking about AI with regards to security event
monitoring, the only way to produce meaningfuł actionable results is
with supervised training.

> Additionally, you seem to imply here that learning algorithms need to be
> supervised (i.e. have pre-labelled samples to train on): this is untrue
> as well.

I would welcome any examples of SEM for unsupervised training outside
of feature extraction techniques which require human intervention to
interpret the results anyways.  Here the human is the supervisor even
if the AI algorithm uses unsupervised training.

> > But when you start talking about host based messages or IDS/IPS events,
> > it's just impossible to train the network
>
> Once again, this is your perception, but it does not necessarily
> coincide with reality ;)

 I disagree with you.  My reality includes analyzing billions of log
events per week in an SEM context.  I would appreciate examples of
your position.

Tom