[logs] High performance syslog aggregation

[Rainer Gerhards]

--
Looks like I screwed up with the email addresses, thus a re-send. My
apologies if you get duplicates. --Rainer
--
Hi Steve,

I am Rainer, the maintainer of the rsyslog project. Store-and-Forward is
not yet implemented, but there are "backup actions". They allow you to
switch to a different syslog server, database server, whatever if the
primary fails. Maybe that solves your problem. A more in-depth
description is in my blog:

http://rgerhards.blogspot.com/2007/08/finally-ability-to-automatically-s
witch.html

The "backup action" is the first feature from a set that will also
support Store-and-Forward. I suggest you also have a quick look at this
blog page:

http://rgerhards.blogspot.com/2007/08/syslog-worker-pools-future-hardwar
e-and.html

It primarily is about threading, but I think you get a lot of insight
from it, too. Besides, it hopefully tells you why I consider rsyslog a
superb alternative for a high-performing syslog server (and of "sales"
blurb ;)).

Rainer

> -----Original Message-----
> From: loganalysis-bounces at loganalysis.org [mailto:loganalysis-
> bounces at loganalysis.org] On Behalf Of Steve Bernacki
> Sent: Friday, November 30, 2007 9:36 PM
> To: loganalysis at loganalysis.org
> Subject: [logs] High performance syslog aggregation
> 
> First of all, thank you to those who responded to my last message
> regarding syslog load balancing.
> 
> I'm currently researching how to best implement a high-peformance,
high
> volume syslog aggregation.  In our current environment, we have many
> devices logging to a small set of "front end" syslog aggregators which
> run syslog-ng.   Currently, these front-end aggregators have a number
> of
> filters enabled, which negatively impacts thruput.  What I'm looking
to
> do is place new systems in front of the existing systems that simply
> capture, queue, and forward messages based on a very limit set of
> searchable criteria (no regexes needed!).  These systems should also
> have the ability to queue incoming messages onto disk and replay them
> in
> the event that a receiver goes down or becomes temporarily
> overburdened.
> 
> My first thought was to implement an architecture similar the
> following:
> 
> Hosts --(UDP)--> (front end) --(TCP)-->(multiple receivers)
> 
> In researching my "free" and "nearly free" options for doing this,
> syslog-ng community edition comes the closest, however only the
> commercial version supports "store and forward" for TCP syslog
streams.
>   rsyslog looks like a promising alternative option, although I
haven't
> been able to confirm through its documentation whether it supports any
> type of "store and forward" mechanism.
> 
> What other tools and/or solutions have I missed?  I've considered
using
> syslog-ng to log to a program which ultimately stores and forwards
> messages, although it seems like there must be a better way of doing
> this.
> 
> Thanks once again for your guidance,
> Steve
> --
> Steve Bernacki, Jr
> To date, the Pan-Massachusetts Challenge has raised 204 million
> dollars for cancer research.  Get involved!  http://www.pmc.org/
> _______________________________________________
> LogAnalysis mailing list
> LogAnalysis at loganalysis.org
> http://www.loganalysis.org/mailman/listinfo/loganalysis