libee 0.4.1 released

Monday, April 16th, 2012

We have just released libee 0.4.1.

This is a bug fixing release, targeting a single bug that prevented building on many platforms.

Changes

Version 0.4.1 (rgerhards), 2012-04-16

  • fixed configure.ac in regard to math lib

 

Download:
http://www.libee.org/download/libee-0-4-1/

As always, feedback is appreciated.

Best regards,
Florian Riedl

libee 0.4.0 released

Tuesday, February 21st, 2012

We have just released libee 0.4.0.

This version most importantly provides the ability to decode JSON and create a libee representation out of it. This is vital functionality to build the latest rsyslog, which supports structured logging via JSON-enhanced syslog.

Changes:

Version 0.4.0 (rgerhards), 2012-02-21

  • rename convert tool to prevent name clash with ImageMagick’s tool
  • support for decoding json added
  • bugfix: ee_newFieldFromNV() did not work correctly
    value was set, but number of values were not corrected, this the
    value could not be used (looked like non-present)

Download:
http://www.libee.org/download/libee-0-4-0/

As always, feedback is appreciated.

Best regards,
Florian Riedl

libee 0.3.2 released

Tuesday, November 22nd, 2011

We have just released libee 0.3.2.
This release includes a new major feature.
Changes:

Version 0.3.2 (rgerhards), 2011-11-22

  • API enhancements:
    • added capability to enumerate tags inside a tagbucket
    • added capability to obtain tagbucket for an event
      –> ee_EventGetTagbucket()
    • added capability to add a string value to a field in one call
      –> ee_addStrValueToField()
    • added ee_getTag(), ee_setTags()
  • added additional parser
    • RFC5424Date
  • potentially problematic API change: in earlier versions, the function ee_TagbucketHasTag() was errornously name ee_BucketHasTag().
  • This has been corrected, potentially resulting in API incompatibility. We have accepted this, because we are at level 0.x AND know that potentially no current user has ever used this function, but instead the upper-layer equivalents. But if thinks break, you now know why ;)
  • flat tags are no longer encoded in the CEE encoders as CEE does not support flat tags. However, this can be turned on via context flags, if desired
  • bugfix: compile problems under Solaris
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=253
  • bugfix: ee_EventGetTagbucket() always returned -1 (error)

Download:
http://www.libee.org/files/download/libee-0.3.2.tar.gz

As always, feedback is appreciated.

Best regards,
Florian Riedl

libee 0.3.1 released

Monday, April 18th, 2011

We have just released libee 0.3.1.
This release includes a new major feature.
Changes:

Version 0.3.1 (rgerhards), 2011-04-18

  • API extensions
  • brought tag handling a bit inline with upcoming 0.6 draft CEE spec

Download:
http://www.libee.org/files/download/libee-0.3.1.tar.gz

As always, feedback is appreciated.

Best regards,
Florian Riedl

log classification with liblognorm

Wednesday, April 6th, 2011

Today, we have added support for so-called “tags” to liblognorm (and it’s base library libee). This new capabilities permits very easy classification of syslog message and log records in general. So you can not only extract data from your various log source, you can also classify events, for example, as being a “login”, a “logout” or a firewall “denied access”. This makes it very easy to look at specific subsets of messages and process them in ways specific to the information being conveyed.

All details can be found at log classification with liblognorm.

libee 0.3.0 released

Wednesday, April 6th, 2011

We have just released libee 0.3.0.
This release includes a new major feature.
Changes:

Version 0.3.0 (rgerhards), 2011-04-06

Download:
http://www.libee.org/files/download/libee-0.3.0.tar.gz

As always, feedback is appreciated.

Best regards,
Tom Bergfeld

libee 0.2.0 released

Friday, April 1st, 2011

We have just released libee 0.2.0.

This release includes some bug fixes and feature enhancements. (more…)

New Mailing List for Log Normalization

Thursday, January 13th, 2011

Thankfully, the interest in log normalization and the related libraries liblognorm and libee has increased. Up until now, we have handled discussions on this topics via the rsyslog mailing list. As conversations increase, this may be come an unnecessary burden for those only interested in rsyslog. So we have created a new mailing list named lognorm. We used this somewhat generic name, as we intend to use it for both libraries. This saves me some overhead, and we strongly assume that anyone interested in liblognorm will also be interested in libee (but to a lesser extent in the reverse direction).

Please subscribe to the new lists. Currently, it is a very exciting phase in log normalization development, so getting involved is a great way to shape things in the way you need it!

libee 0.1.0 has been released

Thursday, December 9th, 2010

Libee is an event expression library which is inspired by the upcoming CEE standard. Right now, it provides capabilities to generate and emit messages in a set of standard format and read a set of different input formats. Libee also comes with a handy conversion tool that provide format transformation without the need to program.

This is the initial public release.

You can download libee here.

normalizing Apache Access logs to JSON, XML and syslog

Wednesday, November 17th, 2010

We like to make our mind up based on examples, especially for complex issues. For a discussion we had on the CEE editorial board, we’d like to have some real-world example of a log file with many empty fields. An easy to grasp, well understood and easy to parse example of such is the Apache access log. Thanks to Anton Chuvakin and his Public Security Log Sharing Site we also had a few research samples at hand.

Apache common log format is structured data. So there is no point in running it through a free-text normalization engine like liblognorm. Of course it could process the data, but why use that complex technology. Instead, the decoder is now part of libee and receives a simple string describing which fields are present. It’s called like this:

$ ./convertĀ  -exml -dapache -D “host identity user date request status size f1 useragent” < apache.org > apache.xml

Options specify encoder and decoder, and the string after -D tells the convert field names and order. But now let’s speak the input and output for itself:

The converter works by calling the decoder, which creates an in-memory representation of the log format in a CEE-like object model. Then, that object model and the called-for encoder is used to write the actual output format. That way, the conversion tool can convert between any structured log format, as long as the necessary encoders and decoders are available. This greatly facilitates log processing in heterogeneous environments.

Note that liblognorm works similar and, from libee’s point of view, can be viewed at as an decoder for unstructured text data (or, more precisely, for text data where the structure is well-hidden ;)).