[logs] High performance syslog aggregation

[Steve Bernacki]

First of all, thank you to those who responded to my last message 
regarding syslog load balancing.

I'm currently researching how to best implement a high-peformance, high 
volume syslog aggregation.  In our current environment, we have many 
devices logging to a small set of "front end" syslog aggregators which 
run syslog-ng.   Currently, these front-end aggregators have a number of 
filters enabled, which negatively impacts thruput.  What I'm looking to 
do is place new systems in front of the existing systems that simply 
capture, queue, and forward messages based on a very limit set of 
searchable criteria (no regexes needed!).  These systems should also 
have the ability to queue incoming messages onto disk and replay them in 
the event that a receiver goes down or becomes temporarily overburdened.

My first thought was to implement an architecture similar the following:

Hosts --(UDP)--> (front end) --(TCP)-->(multiple receivers)

In researching my "free" and "nearly free" options for doing this, 
syslog-ng community edition comes the closest, however only the 
commercial version supports "store and forward" for TCP syslog streams. 
  rsyslog looks like a promising alternative option, although I haven't 
been able to confirm through its documentation whether it supports any 
type of "store and forward" mechanism.

What other tools and/or solutions have I missed?  I've considered using 
syslog-ng to log to a program which ultimately stores and forwards 
messages, although it seems like there must be a better way of doing 
this.

Thanks once again for your guidance,
Steve
-- 
Steve Bernacki, Jr
To date, the Pan-Massachusetts Challenge has raised 204 million
dollars for cancer research.  Get involved!  http://www.pmc.org/