[News]

News:

[rhavens]

rhavens

rhavens

Show rhavens's last posts.

Show general stats for rhavens.

• 
• Newbie
• Posts: 2

It appears that carriage return and line feed (CR/LF) characters in Syslog messages are being converted to spaces when they are written to log files.  Ethereal shows that the syslog packets have the characters in them when they arrive, so it must be syslog-ng that is converting them.  Extensive reading and searching in the 1.6 reference manual, Bugzilla, these forums and the web (via Google) have not given me much information.

Is there a way to turn this feature off or allow for character exceptions?  I assume it's part of a feature that removes all non-printable characters, though this is only a guess.  We are hoping to use Syslog-ng as a target for the logs from our JEE applications, and the logged stack traces are very difficult to read without the carriage returns.

I am using the syslog-ng-1.6.8-20.18, which is standard with SUSE Linux Enterprise Server (SLES) 10 (SP1 and 2).

-Russel Havens
Novell IS&T
Datacenter Operations

[mutex]

mutex

mutex

Show mutex's last posts.

Show general stats for mutex.

• 
• Administrator
• Newbie
• Posts: 782

syslog-ng does intentionally strip out new line characters.

This article has a patch that appears to accomplish what you are looking for:
http://thread.gmane.org/gmane.comp.syslog-ng/6660/focus=6670

Gmane      
From: Balazs Scheidler <bazsi <at> balabit.hu>
Subject: Re: Multi-line Logs
Newsgroups: gmane.comp.syslog-ng
Date: 2008-07-03 09:19:22 GMT (6 weeks, 6 days, 15 hours and 21 minutes ago)

On Thu, 2008-07-03 at 11:17 +0200, Balazs Scheidler wrote:
> On Wed, 2008-07-02 at 18:31 -0400, Enigma wrote:
> > I have a host that sends mult-line messages (embedded newlines, not
> > separate syslog messages) from what I can tell syslog-ng strips out all
> > the newlines and replaces them with a space.
> >
> > Is there anyway to disable or modify (replace them with something else
> > that can be easily s/// in post-processing) this functionality without
> > changing the code and compiling from source?
> >
> > I have been through the syslog-ng manual and asked Mr. Google and I
> > cannot find anything on this topic.
>
> Newlines and stuff are incompatible with TCP transport. If you are using
> UDP, it could work, however as you point out syslog-ng removes all NLs
> from log messages in order not to ruin your logfiles.
>
> With my development snapshot the new syslog-protocol drafts are
> implemented, that too allows embedded NLs.
>
> This crude patch deletes the part that removes NLs from messages:
>
> diff --git a/src/logmsg.c b/src/logmsg.c
> index 139fb3a..adb9f2d 100644
> --- a/src/logmsg.c
> +++ b/src/logmsg.c
> @@ -522,11 +522,6 @@ log_msg_parse(LogMessage *self, gchar *data, gint length, guint flags, regex_t *
>        self->stamp.time.tv_sec = now;
>      }
> 
> -  for (oldsrc = src, oldleft = left; oldleft >= 0; oldleft--, oldsrc++)
> -    {
> -      if (*oldsrc == '\n' || *oldsrc == '\r')
> -       *oldsrc = ' ';
> -    }
>    g_string_assign_len(&self->msg, src, left);
>  }
> 
>
> I might add something more sophisticated if you or anyone else can help me with finding out a
> good idea how to handle NLs when they are written to logfiles.
>
> E.g. you have a template like this:
>
> template("$DATE $HOST $MSG\n");
>
> If there's an NL in $MSG it'd probably break a lot of log parsers. If syslog-ng would repeat the syslog header
>

sorry, sent too early. Would it be enough if syslog-ng would be capable
of repeating the $DATE $HOST part for each line produced because of NLs
in MSG?

How do you want to use multi-line messages?

--
Bazsi

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html

[rhavens]

rhavens

rhavens

Show rhavens's last posts.

Show general stats for rhavens.

• 
• Newbie
• Posts: 2

Thank you mutex.  The gmain.org thread covered exactly the issue I wanted to understand. 

It sounds like Bazsi is already on the case.  His generic fix of having a simple "process_newlines" flag will work fine for our situation, since these are application logs and will not need to be parsed by any sort of standard log analyzer. 

A more flexible mechanism would to allow the administrator to specify a map table of some sort, maybe with a reference to an external file (char_remap_table=/etc/syslog-ng/char_remap.table) and then to have that table have entries that look like oldchar=newchar where the characters could be specified in some sort of regex-compatible format (e.g.: \n= ) or UTF-8 hex (e.g. 0x000a=0x0020) with some aliases for standard sets (ASCII_CTRL_CHARS=0x0020 for ASCII 1-29, etc.).  A missing file would mean, don't do translation; a basic translation table could be available, either as a char_remap.table.sample, or as an in-place default, with performance implications documented as comments in the file.  However, I digress. 

Thank you again for pointing out the thread. As soon as I figure out when is fixed, I'll have to convince then convince my SUSE contacts to put the update into openSUSE and SLES distros (and update channels) so I don't have to one-off these servers for this particular patch.


-Russel Havens
Novell IS&T
Datacenter Operations

Insert Quote
syslog-ng does intentionally strip out new line characters.

This article has a patch that appears to accomplish what you are looking for:
http://thread.gmane.org/gmane.comp.syslog-ng/6660/focus=6670

[mutex]

mutex

mutex

Show mutex's last posts.

Show general stats for mutex.

• 
• Administrator
• Newbie
• Posts: 782

Happy to help