Segregating Logs From Different Log Files On A Centralized Log Server Using Syslog-NG

In this post, I will demonstrate a way to capture logs from a series of log files, and relay those logs to a central log server, where the logs will be separated into log files, as they existed on the original host.

Reading from files

Syslog-ng has the ability to pull log data from files, then treat those logs as any other source, like this:

source  s_file_abclog { file(“/var/log/abc.log” follow_freq(1));};

In this way, you can handle log data from applications that do not have native support for syslog.

Then, you can send that data to a central log server, like this:

source  s_file_abclog { file(“/var/log/abc.log” follow_freq(1));};

destination d_remote{udp(“192.168.0.3″ port(514)); };
log { source(s_file_abclog); destination(d_remote);};

So What’s the Problem?

This is really powerful, but it has a big limitation:  if you are collecting logs from multiple files, and sending them to a log server, once they get to the log server, they have lost the context of while file they came from.  If we have something like this on our server:

source  s_file_abclog { file(“/var/log/abc.log” follow_freq(1));};

source  s_file_deflog { file(“/var/log/def.log” follow_freq(1));};

source  s_file_ghilog { file(“/var/log/ghi.log” follow_freq(1));};

destination d_remote{udp(“192.168.0.3″ port(514)); };
log { source(s_file_abclog); destination(d_remote);};

log { source(s_file_deflog); destination(d_remote);};

log { source(s_file_ghilog); destination(d_remote);};

On our central syslog-ng server, we have have a config that looks like this:

source src { unix-dgram(“/var/run/log”);
unix-dgram(“/var/run/logpriv” perm(0600));
udp(); internal(); file(“/dev/klog”); };

destination d_fileabc { file(“/var/log/abc.log”); };

destination d_filedef { file(“/var/log/def.log”); };

destination d_fileghi { file(“/var/log/ghi.log”); };

log {source(src); destination(d_fileabc); };

But wait, how does syslog-ng know what logs to send to which file?   The answer is that it doesn’t know.  All of the logs will go to each file.  That is not what we want.

Adding a Tag To The End of Each Message

We can tell syslog-ng on the receiving end what log file each log is intended for by using templates and filters.  On the sending side, we add a template statement in each destination:

source  s_file_abclog { file(“/var/log/abc.log” follow_freq(1));};

source  s_file_deflog { file(“/var/log/def.log” follow_freq(1));};

source  s_file_ghilog { file(“/var/log/ghi.log” follow_freq(1));};

destination d_remote_abc{udp(“192.168.0.3″ port(514) template(“$ISODATE $HOST $MSG – File ABC\n.”) template_escape(no)); };

destination d_remote_def{udp(“192.168.0.3″ port(514) template(“$ISODATE $HOST $MSG – File DEF\n.”) template_escape(no)); };

destination d_remote_ghi{udp(“192.168.0.3″ port(514) template(“$ISODATE $HOST $MSG – File GHI\n.”) template_escape(no)); };

log { source(s_file_abclog); destination(d_remote_abc);};

log { source(s_file_deflog); destination(d_remote_def);};

log { source(s_file_ghilog); destination(d_remote_ghi);};

Essentially, we are simply appending a tag that says what file the log belongs to.  We could use any tag we wanted to, so long as it was unique enough not to appear normally in a log.  On the receiving side, we have this:

filter f_file_abc {match(“File ABC”);};

filter f_file_def {match(“File DEF”);};

filter f_file_ghi {match(“File GHI”);};

destination d_file_abc { file(“/var/log/abc.log”); };

destination d_file_def { file(“/var/log/def.log”); };

destination d_file_ghi { file(“/var/log/ghi.log”); };

log {source(src); filter(f_file_abc); destination(d_file_abc); };

log {source(src); filter(f_file_def); destination(d_file_def); };

log {source(src); filter(f_file_ghi); destination(d_file_ghi); };

And there you have it.  This example isn’t particularly useful without customizing to your specific needs, but it should give a good framework for how to solve the problem.

Summary

It’s not the most elegant way to solve the problem, and the logs end up with some extra crap at the end of them, which could be parsed out using sed or similar, but it gets the logs routed to the right files.