Generic Log Parsing Tools

  • Digesting Log Data Part 1 & Part 2: Rolling your own log summary program in Perl.
  • ACID (Analysis Console for Intrusion Detection): A PHP-based analysis engine that searches and processes a database of incidents generated by security-related software such as intrusion detection systems and firewalls.
  • awk: Checking Your System Logs with awk For the ultimate in roll-your-own. From the author: "This piece serves as a brief introduction to the awk programming language, discusses how to use awk to process UNIX log files, and provides several example scripts for system monitoring. It is not a complete toolkit, but rather an approach that can be adapted for a variety of log analysis tasks."
  • Scansyslog - Uses code and ideas from "Tthe Practice of Programming" to look for a large number of semi-static patterns in system logs, and then prints out only lines which are not matched.
  • checksyslog
  • colorlogs
  • CyberSafe Log Analyst
  • GeekTool - a Macintosh OSX generic logging utility/parser
  • guard: Scans system logs for signs of intrusion in real time. Guard produces colored output on the tty, sends alerts and generates regular reports. Excellent database of suspicious logfile strings included.
  • IPFC: The Inter Protocol Flexible Control, a centralized system for collecting and correlating log data from firewalls, routers, and general purpose devices.
  • Kiwi Logfile viewer is a freeware application for Windows 9X, NT/2000
    and ME. Its purpose is to display log files created by Kiwi Syslog
    Daemon in an easy to read manner.
  • Lire: a suite of applications that creates custom reports based on logfiles. Currently supports exim, sendmail, qmail, postfix, BIND, BOA and some Apache logs. Reports are created as ASCII text, HTML or PDF. This is a batch processing tool. Contact LogReport Technical Support for more information.
  • log_analysis: log_analysis goes through several different kinds of logs (currently syslog, wtmp and sulog), over some period (defaults to yesterday), comparing each entry against a list of Perl regular expressions. If there's a match, a data-extracting rule is applied, and the appropriate information is recorded under the appropriate category. Unknown messages are stored separately.
  • LogSentry (formerly Logcheck): LogSentry (formerly Logcheck) is designed to automatically run and check system log files for security violations and unusual activities. It uses a program called logtail that remembers the last position it read from in a log file and uses this position on subsequent runs to process new information.
  • LogDog:(New and improved version of LogDog, with easier configuration and much more efficient use of system resources) a log monitoring tool that allows you to assign keywords to generate alerts, keywords to ignore, and a list of administrators to e-mail. According to the author, LogDog will also aggregate specified messages within a (user-configured) time period.
  • log_merge: Assembles a coherent time line from logs received from multiple sources, based on configuration file.
  • logmuncher
  • logsurfer
  • logtool: A command line program that parses syslog (and syslog like) logfiles into a more palatable format. Data will be crunched into one of the following formats for your viewing pleasure: ANSI, colorized for easy "at a glance" viewing; ASCII (for e-mailed reports, and terminals that don't support color); CSV (for importing into spreadsheets and databases); HTML (for Web-based distribution); and RAW (if you're fond of the unprocessed format). Maintained by A. L. Lambert.
  • logtools, a set of C++ applications written for logfile management and analysis, written by Russell Coker. The tools include clfmerge (merges HTTP Common Log Format output files in order without sorting, which is especially useful for huge Web access logs); logprn (similar to everybody's favorite tail -f, but after a configurable timeout period, will run a program and dump new data to it; funnel (pipes a single stream of data to several distinct files or processes); clfsplit (separates out Common Log Format data files by client IP address); and clfdomainsplit (separates out CLF data files by server domain.
  • LogWatch -- log parser and reporting tool. Based on off-line processing, not real time.
  • Microsoft Log Parser v2.0: Allows SQL-like queries against log data in any format
  • Modular Logfile Analyzer: a GPL'ed parser preconfigured to report on logfiles from 15 different servers.
  • New & improved syslog reduction tools
  • Private I
  • Remote auditing: audit is an encrypted and authenticated communications mechanism for centralized logging. Meant to be used in conjunction with the modular syslog package described above.
  • root-tail: places a transparent overlay of a text file (such as /var/log/messages) into an X11 root window. Great for keeping an eye on things unobtrusively.
  • SEC (Simple Event Correlator): " ...A free and platform independent event correlation tool that was designed to fill the gap between commercial event correlation systems and homegrown solutions that usually comprise of a few simple shell scripts."
  • SHARP (syslog Heuristic Analysis and Response Program): SHARP is a library interface for resident programs to receive and filter syslog messages. Using SHARP, programs can maintain state and operate with a higher level view of system messages. SHARP can be used to throttle alert messages, track user login patterns, react when a message is not received, or even correlate messages between many systems. Contact Matt Bing for more information.
  • SIDS (Statistics-based Intrusion Detection System): SIDS is a log-based anomaly detection tool. It's primarily focussed on HTTP server logs at the moment, but any predictably formatted single line log data is theoretically managable with this code. Contact Ryan Russell for more information.
  • SLAPS-3: James Finegan's project for summarizing and reporting on UNIX system logs. SLAPS-3 is a work in progress. Great tool, with good documentation for enterprise deployments and an emphasis on making information useful to system administrators.
  • SL2: A Perl tool to identify single-line log anomalies
  • SLCT (Simple Log Clustering Tool): Code designed to identify patterns occurring in a logfile more frequently than a given threshold.
  • swatch
  • syslogScan 0.32
  • syslog-summary: A Python script that summarizes the contents of a syslog output file, by displaying each unique line once (timestamps are not included in the determination of line uniqueness). This script also provides the number of times each unique line appeared in the given file. Lines are displayed in the order they occur in the input file. This code is GPL'ed; it's written and maintained by Lars Wirzenius.
  • tklogger: Monitors any plain text log file and identifies user-configurable events (not limited to syslog data). Application is well documented, and includes a sample startup script as well as a sample rule configuration file.
  • xlogmaster: A system monitoring tool that allows administrators to monitor everything that's happening on a system in a very quick and comfortable way. It allows reading logfiles, checking devices or running status-gathering programs, translating all available data, and displaying results with filters and associated actions (including highlighting or lowlighting lines, hiding data, or taking actions on user-defined events.

NEW!Learn
Log
Analysis!

(Bonus: Now You can download Tbird's SANS Webcast Slides)