[logs] naming multiple output files with syslog-ng

[Chris Wee]

Christian,

On a previous project in 2003, we used Mysql4 MYISAM tables as the binary
logging format. It stood (and still stands) a
good chance of being readable up to 2010. The overhead of the DB engine was
neglible w.r.t. to the volume of the log (high).

Today, I would use sqlite3 because it is endian-neutral, and I think given
the support it is receiving, it has a good chance
of being supported for 5-10 years. Your ascii-parser is written in some
language (perl, python, ruby, java) and that has
to be ported to 128-bit architectures in the future which is about the same
chance as sqlite3/4/5...

Note, the performance gains we saw from binary versus ascii was due to the
nature of the data -- non-string data that required
conversion from text <-> binary representations. Your mileage may vary.

-chris

On Dec 20, 2007 10:55 PM, Christian Folini <christian.folini at post.ch> wrote:

> <snip>
>
> Binary formats and csv logging. A great deal faster and a lot more
> storage efficient. However, this is not always a priority.
> My priority is, that the logfiles I am writing today be readable
> tomorrow. Possibly with standard tools. I am quite sure that any
> binary format would be the wrong format tomorrow. If these guys are
> smart, they will be able to get the data out of the DB again, tomorrow.
> But it's likely to cause headaches and meetings and stuff.
> Ascii logging is the safe path; and we can always import the data
> into a db later on or transform them into csv format. Actually I
> did some work on that lately. All that is bugging me is the time it
> takes to parse the ascii logfiles. It takes too long for my
> taste, but then it's also a lot of logfiles. ;)
>  <snip>
> Christian
>
> P.S. Marcus: Cool to read your message in this thread. I just
> wrote about artificial ignorance in one of the conceptual papers
> for this project. :)
> _______________________________________________
> LogAnalysis mailing list
> LogAnalysis at loganalysis.org
> http://www.loganalysis.org/mailman/listinfo/loganalysis
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.loganalysis.org/pipermail/loganalysis/attachments/20071221/a963d7c2/attachment.html