[logs] CEE - a new logging standard

[Tom Le]

| > Cost - For the vast majority of vendors (especially the big ones),
logging
| > is one of the least important aspects of their development process.
There
| > are different groups responsible for different components.  If
normalizing
| > logging internally adds cost and time to development, I don't see
vendors
| > fixing their existing logging issues and on top of that compiling with a
new
| > standard.
|
| This is only partly true. The entire regulatory landscape is changing
| things QUITE a bit! You need accountability and more and more
| applications HAVE to log and log review has to be in place.

The big vendors will benefit more from their own compliance solutions than
to make it easy for third party tools.  I just don't see Cisco changing
their entire IOS logging to a new standard unless it is in their own best
interests.

| And contrary to what you claim, I am dealing with approximately 20
| vendors who are implementing a standard logging format at this very
| moment. [http://www.arcsight.com/solutions_cef.htm]

CEF's biggest problem is that it is not truly vendor neutral.  I'm not
knocking the standard itself.  But I think adoption may be a problem.

Also, the biggest issue I have with CEF (and perhaps this is where the
disconnect is between my post and your replies below), is that CEF focuses
on format, not content.

| > vendors/partners, etc.  The only way to introduce a drastic change in
| > logging and maintain legacy support is to do both in parallel and allow
| > users to choose either logging format (i.e. this is how SNMPv2 and v3
| > adoption happened).
|
| SNMP is not a format. It's a transport! You could argue it's a format
| too, but at most a really ugly one ;)

Yes, of course SNMP is a transport.  But there is no standard today for SNMP
content.  Think about all the attributes of an SNMP trap.  They are all
implemented differently across all vendors with custom definitions that the
log derivative players have to build custom support.  The common format and
MIB documentation helps this process (much easier than other transports),
but this still requires an individual integration effort for every new
product.

| > Competitive advantages - Paul Melson accurately wrote in his blog (in
| > commenting on CEE) that "It costs SIM vendors a decent chunk of change
in
| > development and support cost to spin up support for new products.
Standards
| > make this proposition cheaper: one set of code to maintain, potentially
| > hundreds of products supported."  This is precisely the reason why you
WON'T
| > see the largest vendors evangelizing any new log standard.  It's a huge
| > barrier to entry (in their favor).
|
| Okay. Here I COMPLETELY disagree. ArcSight is pushing a standard called
| CEF and we want as many people as possible to adopt it. It is NOT a
| competitive advantage to support more devices. We rather have 500 more
| supported products that we can sell solutions, appliances, etc. It's all
| about solving customer use-cases and supporting a wide variety of
| use-case.

CEF is in ArcSight's best interest.  It may not be in other log analysis
vendors' best interests.

| > Even Anton's company and it's competitors
| > would be inviting competition if a new log standard was adopted
overnight.
|
| Why? The SIMs and Log aggregators all have unique capabilities that they
| market. You build better platforms, you are a player in the field. But
| believe me, it's not _that_ easy.

Obviously, companies must evolve and the value proposition must include more
than just device support.  But if you look at the log derivative industry
today, it is still young enough that a key barrier to entry is device
support.  Think about all the meta data tagging that a LogLogic, Splunk, or
Cisco MARS performs and how this meta data "content" is used with "analysis
intelligence".  LL makes a lot of money selling their compliance & reporting
applications.  If a new logging standard now addresses the *content* of
these logs, and say adds a taxonomy to these events, there goes a huge value
added service that is generating revenue.  Sure, these log analysis vendors
still have to deliver on everything else besides device syntax, grammar &
content mapping (i.e. "device support"), but today device support is a
differentiator in the marketplace.  I can provide you a lot of specifics
from my own experience offline as to keep my comments vendor neutral.

| > Imagine how easy it would be to build a new log-centric product or
service
| > if
| > you didn't have to spend as much time building support for a hundred
vendor
| > devices (e.g. an MSSP that support a million log events).
|
| MSSPs are more and more using COTS products and not going with their own
| solutions anymore. It's too expensive to maintain a product like this
| (i.e., a ESM tool). There is a reason why some of the SIM vendors employ
| over 200 people. These things don't come to live over night.

The "big 4" build their own device support from the COTS log output,
AFAIK.  This means even though they are using COTS they still have to build
syntax, grammar, and content support.  The MSSP I work for has over 1M
unique event rules.  It's a huge differentiator in the marketplace.

| > What is the incentive for
| > existing vendors (especially the big ones) to change their logging
format?
|
| Interoperability. Supporting existing log collection frameworks, SIMs,
| enabling solutions (e.g., a SOX solution from a SIM vendor). You are
| suddenly helping solve the SOX problem without doing much.

But what if a big vendor like Cisco or McAfee or ISS already have their own
centralized management platforms?  They invite competition by
standardizingtheir log output.  It's a fine line between making your
event data more
accessible to third party applications vs. inviting competition.

In my experience, if it's just access to the syntax and grammar of the data,
device vendors are more than happy to help with API's or sharing data
schemas (this covers log format, i.e. CEF).  But when it comes to the
*content* of the data, such as meta data and event taxonomy, the data is
usually not available.  As a log analysis vendor you have have to build that
yourself - and I say it's an important differentiator.  Look at what Cisco
MARS does today as an example.  What MARS has planned to the future is based
entirely on this huge competitive advantage of building support for event
content into a normalized framework.  Log standardization helps them support
more devices, but tears down this huge barrier to entry.

| > There is no incentive or mandate for existing device vendors to change
their
| > logging format.
|
| Absolutely there is! And it's not just the syntax or transport. We are
| also talking recommendations (see my blog post)!

I am specifically talking about content.  Normalizing and extracting content
from the syntax is a huge pain for log analysis vendors... and a huge
competitive differentiator as well.

| You don't call OpenView a log correlation tool, do you? They are a
| network management tool. SNMP is about as far as they really support log
| feeds. I consider the ESM products of the world and I know for a fact
| that they are already supporting these types of efforts!

No, not a log correlation or management tool, obviously.  I am referring to
OpenView specifically from a content perspective.  In order to enable HP
OpenView management, alerting, and monitoring capabilities, device vendors
have to build smart plugins so that OpenView can understand the content of
the SNMP events and the third-party products can interoperate.  OpenView is
the closest thing the world has seen to event-based content standardization
(albeit a narrow scope).

Without a log standard that addresses content, CEF or any standard won't
really change the world that much.  For most third-party log analysis
companies, the grammar and syntax is the *easiest* problem to solve.  Very
advanced parsing and indexing schemes can eat tens of thousands of events
per second on commodity hardware.  The challenge is in building the
intelligence around the content.

| Sure. You are saying it yourself. There will never be a world where
| everyone will support the standard logging format, so there will ALWAYS
| be space for the log management / correlation / SIM /ESM tools to build
| | connectors and agents and have better support for some feeds than their
| competition. The game will change though! Instead of supporting 300
| products, you can scale much better and support 3000 products, because
| 2700 of them are using a standard format!

I don't think you can achieve 10x scale with just log format
standardization.  You need content standardization.  You need a defined
taxonomy that is both specific and extensible.  I can speak from my own
experience that understanding grammar and syntax is very easy to do.  The
analysis of the content is where the value lives.  I'm sure we all agree on
this point.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.loganalysis.org/pipermail/loganalysis/attachments/20070423/dccb5793/attachment-0001.html