[logs] High performance syslog aggregation

[Tom Le]

On Nov 30, 2007 12:36 PM, Steve Bernacki <loganalysis at f.copacetic.net>
wrote:
> My first thought was to implement an architecture similar the following:
>
> Hosts --(UDP)--> (front end) --(TCP)-->(multiple receivers)
>
> In researching my "free" and "nearly free" options for doing this,
> syslog-ng community edition comes the closest, however only the
> commercial version supports "store and forward" for TCP syslog streams.
> rsyslog looks like a promising alternative option, although I haven't
> been able to confirm through its documentation whether it supports any
> type of "store and forward" mechanism.

Have you considered building a Perl-based read & forward application?

The front-end servers just write syslog to a file (which you probably are
already doing anyways).  The Perl application lives on each front-end server
and does a "stateful tail" that handles log rotation (or use Marcus's retail
and pipe to Perl if you don't want to build your own). The Perl app sends
tcp or udp syslog to the "multiple receivers".  Make sure syslog-ng
front-end servers  is configured to capture the originating IP address. You
can add custom forwarding rules based on whatever you need: source IP,
facility/priority, service name, or even regex-based identification of
specific messages types.

In other words, you use the native syslog functionality of writing to a file
as your "store" mechanism, and then you just intelligently read and
forward.  You can forward in real-time or at whatever desired interval.  You
can even add code for throttling, depending on how fast your multiple
receivers can receive syslog message.

The beauty of this method is you can also perform replays whenever needed (
i.e. say you are testing a SIM or SIEM tool) and at whatever speeds desired.

Shouldn't take anymore than a few hours to build for a typical Perl
developer.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.loganalysis.org/pipermail/loganalysis/attachments/20071204/28133fba/attachment.html