Reading Logs From A File In Syslog-NG

I had previously written a little snippet on how to pull logs in from a file, however there is a substantial amount more to consider when configuring syslog-ng to read from a file, so I have dedicated this post to reading logs from a text file.

The basic structure for reading logs from a text file looks like this:

source s_file {
file(“/var/log/app.log”);
};

destination d_messages{
file(“/var/log/messages”);
};

log {
source(s_file);
destination(d_messages);
};

The source file driver has a lot of options which can you help to tailor the import of logs into syslog.

One of the first, and most important settings to consider when reading logs in from the file is the “flags(no-parse)” option on the source file statement.  By default, syslog-ng expects the lines in the text file being read to be in the format of a syslog message, and will attempt to parse out the relevant fields, like date/time, severity, program, etc.  However, most applications that write custom log files don’t use a standard syslog format, and so we need a way to tell syslog-ng that the whole log record in our log file is the “log message”, and not a syslog-formatted message.  We accomplish this with the “flags(no-parse)” option, which looks like this in the config file:

source s_file {

file(“/var/log/app.log” flags(no-parse));

};

Note that there is no semi-colon after the closing parentheses.

Source Confusion

Consider, for example, an application running on a server that creates its own custom logs to a text log file.  We want to capture those logs and send them to a central log server for sorting, analysis and storage.  We are running syslog-ng on the application server, and have it also configured to send the operating system logs to our central syslog server.  There is no good way, once the logs are sent to the central server, to tell which logs came from the operating system and which came from the application, since they are all streamed together.  Certainly, most OS logs will have the program name field set, and we could figure out which are the application logs by the absence of that field being set, but that is not optimal.  If we had two applications running on the server sending logs, it would not be possible to differentiate the source, unless the application included some indication of the source in each log message.

Syslog-ng provides an option to the source file driver to append a custom “program name” to each message processed by it using function program_override().  The syntax of program_override looks like this:

source s_file {

file(“/var/log/app.log” program_override(“My Application Name”) flags(no-parse));

};

Here is an example of reading from two different log files, and labeling each with a unique program name:

source s_file_a {
file(“/var/log/appa.log” program_override(“Application A”) flags(no-parse));
};

source s_file_b {
file(“/var/log/appb.log” program_override(“Application B”) flags(no-parse));
};

destination d_messages{
file(“/var/log/messages”);
};

log {
source(s_file_a);
destination(d_messages);
};

log {
source(s_file_b);
destination(d_messages);
};

Severity Of Logs From a Text File

By default, syslog-ng assigns logs read from a file the severity of kern.emergency.  We can change this using the default-facility() and default-priority() options in the source file driver statement.  To assign imported messages a severity of local0.Notice, our source file statement looks like this:

file(“/var/log/appa.log” program_override(“Application A”) flags(no-parse) default-facility(“local0″)  default-priority(“notice”));

This would, for instance, let you take two different logs from the same application and apply a different severity to each, which allows for different actions/notifications to be taken at the central syslog server.

Applying Different Severity To Specific Logs From A Text File

Consider the situation where an application writes all of its logs to a text file.  Mixed in the log messages are potentially important logs that should be acted on.  The best we can do with the source file driver is apply the same program name and severity to all logs coming from that file, which doesn’t do much to help us differentiate the important events from the unimportant ones.  Some log monitoring programs will let you create custom signatures and apply specific rules and actions based on the content of the message, but many others simply rely on the severity of the log message to determine what to do.  For instance, I may have a script that sends an email to my phone for each log with a priority of emerg.  Using the source file driver, either all of the logs or none of the logs will have that priority, so we need to consider an alternative.

I had previously written an instruction on rewriting the priority for log messages, and we’ll use that as the mechanism to accomplish the our goal here.  First, we need to create a filter to distinguish the important message we want to change the severity on:

filter f_alarm {match(“alarm” value(“MESSAGE”));};

which matches all logs that have the word “alarm” in the content of the log message.  We also need a filter to get the converse:

filter f_not_alarm {not match(“alarm” value(“MESSAGE”));};

which matches all logs that do NOT have the word “alarm in the content of the message.

Now, we assemble the configuration like this:

source s_file {
file(“/var/log/app.log” program_override(“My Application”) flags(no-parse) default-facility(“local0″)  default-priority(“notice”));
};

destination d_net_alarm {
udp(“192.168.0.1″  template(“<144>$DATE $HOST $MSGHDR$MSG\n”; template_escape(no)));

};

destination d_net{
udp(“192.168.0.1″));

};

filter f_alarm {match(“alarm” value(“MESSAGE”));};

filter f_not_alarm {not match(“alarm” value(“MESSAGE”));};

log {
source(s_file);

filter(f_alarm);
destination(d_net_alarm);
};

log {
source(s_file);

filter(f_not_alarm);
destination(d_net);
};

This rewrites the severity of the message for logs containing the word “alarm” to local0.emergency as they are being sent to the central syslog server, 192.168.0.1, but keeps the severity at local0.notice for all others.

This was meant to serve as an example of how the different functions of syslog-ng can be assembled to create complex transformations on log messages, and will probably not solve anyone’s specific requirement.  If you do have a complicated requirement, post a comment below, or drop a question into the syslog-ng help forums.