How can I prevent my TS from being converted to $ISODATE?

[mb335]

Hello,

Is it possible to prevent the timestamps from being converted to the $ISODATE format when sending events on to a remote syslog device?

I have a number of devices sending syslogs to my syslog-ng server. These logs are then sent to RSA enVision.

enVision needs to receive syslogs in their original format for it to identifiy the source device type and categorize them accordingly.

However, when syslog-ng sends the logs it converts the timestamps to the $ISODATE format which enVision doesn't appear to recognise.

If I set my destinations to /data/log/messages it receives the original format.

The intention is to use syslog-ng to filter out unwanted messages before sending them to enVision.

Many thanks in advance for your help.

Martin

[mutex]

What version of syslog-ng are you using, and what does the date format you want to retain look like?

[mb335]

Hi,

I'm running syslog-ng version 3.0.

The original log format looks like this:

May 11 13:41:03 10.0.0.1 NETSCR01: NetScreen device_id=NETSCR01  [Root]system-notification-00257(traffic): start_time="2009-05-11 13:33:53" ...

However the syslog-ng output is this:

May 11 13:33:37 172.16.0.1 1 2009-05-11T13:41:03+01:00 10.0.0.1 NETSCR01 - - - NetScreen device_id= NETSCR01  [Root]system-notification-00257(traffic): start_time="2009-05-11 13:33:53" ...

I'm not worried about the additional timestamp prefix as enVision can remove this.

Also, as you can see from the 2nd string, " - - -" replaces a ':' after the hostname.

Many thanks,
Martin

[mutex]

That is interesting...  Do you get that extra date field when you have the syslog-ng relay log to a file rather than passing to envision?

Here is a way to fix the problem using templates:

Code:

source { ip(x.x.x.x); };
template t_rewritetime { template("${MONTH} ${DAY} ${HOUR}:${MIN}:${SEC} ${HOST}: ${MSG}\n");  template_escape(no); };
destination { ip(x.x.x.x); template(t_rewritetime); };

[mb335]

No, I don't see the extra date field in the file.

I've not got access to this network right now, but I'll try your template and post the outcome asap.

Many thanks for your prompt responses.

Cheers,
Martin

[mutex]

That is very interesting.  I am wondering if the envision application is adding the extra date.  I can't see syslog-ng appending an extra date when sending a log on to another host, but not appending the extra date when logging it to a file.

[mb335]

That was my initial thought, however I ran a packet capture on the enVision server and saw the additional field before enVision processed the syslogs.

Additionally I set a destination to a syslog server on my laptop and saw the same field inserted.

enVision can certainly handle it as there's an option to ignore headers inserted by a syslog server.

To me it looks as if syslog-ng is ok as it's correctly writing output to a file, however the timestamp conversion kicks in when the transfer takes place ... I'm not sure what that process is.

Ok, I tried running with your template config and it resulted in this ...

May 11 18:00:54 172.16.0.1 1 2009-05-11T18:08:21+01:00 10.0.0.1 NETSCR01 - - - 05 11 18:08:21 10.0.0.1: NetScreen device_id ...

I'm off-site now with no access to the infrastructure until 10am GMT tomorrow but I'll take a look at my syntax then.

Cheers,
Martin

[mutex]

It would be interesting to try the packet capture on the syslog-ng relay. I am going to attempt to recreate it later tonight (minus envision - don't have one of those laying around)

[mb335]

Thanks for taking the time to recreate the issue. I've inserted my syslog-ng.conf file at the foot of this if it helps. You may see something else that's causing the problem.

No, I don't have an enVision system (3 servers + NAS) lying around either! ;-) Unfortunately it's one of those products you just can't virtualise.

Thanks, Martin ...


@version: 3.0

# options
options         {
                chain_hostnames(no);
                use_fqdn(no);
                use_dns(no);
                keep_hostname(yes);
                };

# sources of syslog messages (both local and remote messages on the server)
source s_local  {
                internal();
                unix-stream("/dev/log");
                file("/proc/kmsg" program_override("kernel: "));
                udp();
                };

# templates
template t_rewritetime { template("${MONTH} ${DAY} ${HOUR}:${MIN}:${SEC} ${HOST}: ${MSG}\n");  template_escape(no); };


# destinations
destination d_messages  {
                        file("/data/log/messages");
                        syslog("172.16.0.1" transport("udp") port(514) template(t_rewritetime));
                        };

destination df_datalogmessages_template {
                                        file("/data/log/batch-syslog" template(template_1) );
                                        syslog("172.16.0.1" transport("udp") port(514));
                                        };

# filters
filter f_local6         {
                        facility(local6);
                        };

filter f_notlocal6      {
                        not facility(local6);
                        };

# log statements
log     {
        source(s_local);
        filter(f_notlocal6);
        destination(d_messages);
        };

log     {
        source(s_local);
        filter(f_local6);
        destination(df_datalogmessages_template);
        };

[mb335]

Ok ... one of my colleagues has taken a look and supplied the following inline destination string which works:

destination d_mesg { udp("172.16.0.1" port(514) template("${MONTH} ${DAY} ${HOUR}:${MIN}:${SEC} ${HOST}: ${MSG}\n") template_escape(no)); };

Mutex ... this is not far off what you suggested so maybe I just stuffed up your syntax previously.

Again, many thanks for all your input ... much appreciated!

Cheers,
Martin

[mutex]

That is good to hear.  I was not seeing the same result when I tried yesterday, so I'm not sure why that happened, but I', glad that you have a solution.