[logs] Feedback Needed: Large Scale Syslog Management

[Clayton Dukes (cdukes)]

Hi Folks,
If you are managing syslog data in your large scale environment, I'd like to hear from you.
I need input on:
1. Number of devices you are managing logs for (large scale being over 10,000 devices)
2. What log levels you are sending from the devices (i.e. 0-6 for normal operation, 0-7 when troubleshooting?)
3. What log levels you are reacting on (if not all).
4. How many people are assigned to look at log messages
5. What program(s) (commercial or open source) are used to do log analysis (syslog-ng, php-syslog-ng, splunk, etc).
6. How are you analyzing the logs? Are you doing a baseline analysis (based on number of events per device) or are you reacting on every incoming message...or do you just ignore them because there are too many to look at, etc.
7. Anything I missed?


I need to gather this information ASAP for a customer asking similar questions (they have 35k devices) by the end of the day today. So please reply as soon as you can :-)
I have my own opinions on all of this, of course, but I want to gather data on what other folks are doing.

cdukes