Reading Logs From A File In Syslog-NG

I had previously written a little snippet on how to pull logs in from a file, however there is a substantial amount more to consider when configuring syslog-ng to read from a file, so I have dedicated this post to reading logs from a text file.

The basic structure for reading logs from a text file looks like this:

source s_file {
file(“/var/log/app.log”);
};

destination d_messages{
file(“/var/log/messages”);
};

log {
source(s_file);
destination(d_messages);
};

The source file driver has a lot of options which can you help to tailor the import of logs into syslog.

One of the first, and most important settings to consider when reading logs in from the file is the “flags(no-parse)” option on the source file statement.  By default, syslog-ng expects the lines in the text file being read to be in the format of a syslog message, and will attempt to parse out the relevant fields, like date/time, severity, program, etc.  However, most applications that write custom log files don’t use a standard syslog format, and so we need a way to tell syslog-ng that the whole log record in our log file is the “log message”, and not a syslog-formatted message.  We accomplish this with the “flags(no-parse)” option, which looks like this in the config file: Read more…

Pot Of Syslog-NG Tricks Version 3

Retaining the original hostname of the origin of syslog messages through a Syslog-NG relay

In some environments, syslog messages are concentrated and relayed through an intermediate syslog server.  One of the big deficiencies of the stock syslogd that comes with many Linux/UNIX operating systems is that they don’t provide the ability to keep the hostname or IP address of the original sender of the syslog message as it is transmitted through the syslog relay server.  Syslog-NG provides an option that, when used on the relay server, retains the host name of the originating system.  A basic relay syslog-ng.conf file might look like this:

source s_net { udp(); };
destination d_net { udp(“1.2.3.4″); };
log { source(s_net); destination (d_net); };

Simply adding the keep-hostname(); statement to the source definition tells syslog-ng to retain the original host’s name in the syslog message as it is relayed through.  The resulting config file would look like this:

source s_net { udp(keep-hostname();); };
destination d_net { udp(“1.2.3.4″); };
log { source(s_net); destination (d_net); };

And that’s all there is to it. Read more…

Spam Attack Update

In a previous post, I described a spam attack the syslog forum was under. The attack intensified pretty dramatically after that post. This time, though, it was a focused attack by a bot-net registering dozens of accounts per hour. I had read that the CAPTCHA system in SMF, even at the highest setting, had been programmatically defeated and registration bot scripts that can decipher the CAPTCHAs are readily available, so I installed the reCAPTCHA package, which has worked well on other sites. Interestingly, the rate of bot-originated spam registrations increased after switching to reCAPTCHA. It would seem that reCAPTCHA is also broken. I found an anti-bot registration puzzle package on the SMF mod site and gave it a shot this morning.

Since installing the puzzle package I have not had a single spam registration.  It’s only been a few hours since implementing it, but that has saved me from deleting about a hundred accounts.  As I watch the web logs, I can see the bots still diligently trying to create accounts, but are very fortunately not being successful.

I am concerned for a few reasons:

• I am probably pissing off the owner of a bot-net, which could end up with my site being DDOS’d.  Hopefully, he will instead lose interest and pick on another site.
• The anti-bot puzzle package uses colors as one of it’s tests.  This makes the site inaccessible for those who are blind or color blind.  Hopefully this is a manageable problem because a reloading the registration page will likely present them with a different question that can be answered by the visually impaired.
• I suspect that this new obstacle is only going to be effective temporarily, until the scripts can be updated to handle the puzzles.  If a script can be made to decipher CAPTCHAs, one can certainly be made to break the relatively trivial anti-bot puzzles.

Other interesting observations:

The bot-net was not simply executing scripts.  The spam wave appeared to be controlled a scripted interface to normal browsers running on owned PC’s. All of the hosts reported either being IE 6 or IE 7, running on various versions of Windows and reporting different tool-bar plugins, leading me to believe that it was indeed actual browsers being scripted.  Another indication is that the bots were also viewing Adsense ads on the forum site, which incrementing the viewed ad count (sadly, that didn’t result in any extra income).   Most of the hosts were using generic ISP IP addresses from all over the world.

Forum Spammers Abound

I have managed the syslog.org site for over a decade now and I have seen a lot of spammers.  Fighting the spam battle used to be pretty straight forward on this low volume forum running the Simple Machines software.  When a forum only gets a few posts a week, it’s pretty easy to pick out the spam.  For a while, the spammers were hell-bent on submitting cleverly written posts with a signature that included a link to the site they were spamming.  I like to think of this period in forum spamming as “fitting in”.  They didn’t contribute to the conversation, but it wasn’t obnoxious, either.  My read is that the spammer hoped the comment would be ignored by moderators.  Legit people making real contributions to the forum were and are welcome to include a link in the signature, but everyone else got banned.

“Fitting in” morphed into a much more blatant kind of spam I call the “dump and run”.  In the “dump and run”, the spammer submits a post that is generally quite long in the form of a short story or informational paragraph about the topic of the site being spammed.  Key words in the post are linked back to the site being promoted.  Clearly these posts don’t fit the context of nearly any forum, and so are going to stick out as spam everywhere.  Here are my theories on this type of spam: Read more…

Determining What To Monitor

Earlier in my career, I was the IT director for a medium sized enterprise and had responsibility for information security, in addition to networking, server ops, help desk, etc.  I was fortunate to be able to start with a mostly clean slate and had the help of many talented and energetic thinkers.  The company was a manufacturer of network security software and our flagship product was an intrusion prevention (IPS) application, which meant cost wasn’t an option with determining how many IPS engines to place throughout the network.  One of the debates we had was what to do with and how to respond to attacks coming from the Internet.  We had implemented a very robust and restrictive firewall infrastructure that was very effective at keeping out attackers.  The IPS engines on the Internet side of the firewalls reported a constant barrage of scans, malware propagation attempts and nearly every other type of attack the engine could detect.

Full of good intentions, we prioritized attacks and tried informing abuse contacts for assistance with stopping them.  There were far too many to handle manually, and we quickly found that ISP’s and network owners were either overwhelmed with complaints, didn’t care or were complicit in the attacks.  Our well intentioned efforts did absolutely nothing.  I found that the volume of Internet-based attacks is directly proportional to the number of IP addresses the organization has, and nothing was going to change that.  The only one who cared about what happened to our network was us, and we needed to focus on only things we need to care about. And we decided that we could only care about attacks that were successful.  The firewalls were effective in their jobs, after all.  We could roughly monitor successful attacks by looking at IPS engine traffic on the inside of the firewalls.

As our budding IT shop matured, we entered the age of metrics.  We relied heavily on technology to keep us safe, and so didn’t have a lot of process related metrics.  To show the value of the security program, we began reporting the number of attack attempts and the number of intrusions.  Being a technology company, and a security-oriented one at that, these numbers were very interesting to the management team at first.  We would show reports with millions of attacks each month, and always zero successful intrusions.  At some point it occurred to me – I don’t, nor was I expected to, do anything different if the number of attack attempts was 100,000 or 100,000,000,000 in a month.  It was a truly useless metric.  The very few times there were security incidents, the management team was made aware and kept up-to-date real time, so even the successful intrusion metric was pretty useless.

I learned a lot as I matured alongside that IT organization, and one important lesson was only monitor things that will cause you to take action based on the resulting data.  Anything else is simply trivia and probably not contributing anything useful to the organization.  Syslog is very similar.  There is generally good value in archiving logs for reference in the future, but for active monitoring, an assessment should be made about whether the types of things being monitored would cause an action based on a specific event being detected.

Windows Syslog

Windows does not natively support either sending logs out as syslog messages.  There are a number of applications that will translate Windows Event Logs to syslog.  A partial list is:

• EventReporter
• Snare
• NTSyslog

Why Send Event Logs To A Syslog Server?

There are a few good reasons to export Windows Event Logs as syslog messages.  Syslog is a basic format and allows logs from many sources to be normalized, stored in a central repository and analyzed by a common system.  Many log analysis engines support the direct pulling of Event Logs, but the mechanism to do so is generally pretty clumsy, requiring a batch process that periodically connects to a share and transfers a copy of the entire log file.  Such a process is inefficient if the log files are large, and does not provide the benefit of having the logs moved to a log sever/analyzer real time.  Logs sent to a separate log server are not at risk of being lost in the event of software or hardware failure or logical attack on the Windows server in question.

Downside

The primary down side to exporting Event Logs to syslog is that Event Logs are structured sets of data and the structure is not cleanly retained as the events are converted into a string of plain text.  Generally, though, it is possible to parse out the data with some rudimentary analysis of the converted log messages.

Log Analysis and Log Correlation Basics

Log data can provide benefits beyond the obvious notification of system events and security happenings.  Aggregated logs from a system or from multiple systems can provide a more complete picture of problems when those logs are correlated together.  To any experienced administrator, this is obvious.  Consider the following environment:

Read more…

Using Trends In Logs To Define New Security Requirements For Internet Facing Hosts

I have a few servers at a colocation datacenter for running a number of sites, including this one.  I have written before about detecting brute force attacks in logs.   I have been watching the attacks continue in my logs, and have noticed a few things:

1. The attacks, as before, are coming from many different sources, nearly simultaneously.

2. It’s interesting that the brute force account and password guessing are so well coordinated – generally I see the same user name tried by multiple hosts sequentially, the moving on to new name, usually in alphabetical order.

3. The attacks are now coming in via multiple vectors.  Previously, the attacks were only carried out using ssh connections, but are not also using pop3, imap and ftp.

4. The attacks have started trying to intelligently guess the name of accounts on the server, based on the domain name.  In my case, I am using cPanel/WHM, where each domain has a shell account, and generally the user name is some derivation of the domain name.   Previously, I would only see a dictionary of names being sequentially tried, but now I see the user name “syslog” being tried many times.  From the perspective of an attacker, it’s much better to only have one variable to brute force (the password) rather than two (user name and password).

So, what can I infer about the attackers given the data I have seen?

1. There are some number of botnets being used to guess passwords of accounts, and the zombies are being very well coordinated by some command and control infrastructure.

2. The attackers are trying very hard to infect web servers.  Once an attacker has access to modify a site, he can do a number of things; from send spam to integrate browser exploit code into the normal site(s).  The latter is particularly interesting, since there is substantial money to be made from installing adware/spyware/other maleware on desktops.

What can be done to counter these threats?

1. Enforce minimum password complexity  (8 character minimum is a good start)

2. Choose user names that can’t be directly linked to a domain.  For instance, the user name “syslog” would be easily guessable for the domain www.syslog.org.  A better name might be “syslog22″.

3. If unique usernames are used as identified in #2 above, logs can be monitored proactively for instances where the real account name is being tried unsuccessfully.  If there is not an externally discernible way to identify the user name,appearances of such entries in logs could indicate that the system has been compromised in some other manner.

Pot Of Syslog-NG Tricks Version 2

Correcting bad or duplicate time and date stamps

Trying to accept logs from applications or devices into syslog-ng, but end up seeing two date and time fields in the resulting log coming out of syslog-ng?  This happens because syslog-ng is not able to understand the format that the date and time stamp arrive in.  Here’s an easy way to fix that problem.  Use a template to only use the “message” part of the incoming log:

source s_net { udp(); };
destination d_file{ file(“/var/log/application.log” template(“$MSG\n”)); };
log { source(s_net); destination(d_file); };

Including the facility & priority of a message syslog-ng output

To add the written form of the facility and priority to logs coming out of syslog-ng, such as “local1:info”, use the following template.

destination d_file {
file(“/var/log/messages”
template(“$FACILITY:$PRIORITY $MSG\n”; template_escape(no))); };

Alternatively, to add the numeric severity, use this:

destination d_file {
file(“/var/log/messages”
template(“$PRI $MSG\n”; template_escape(no))); };

$FACILITY and $PRIORITY produce the written name of the facility and priority, respectively.  The PRI macro produces the derived number that represents both the facility and priority.  To get the numeric value of the facility and priority, use the $FACILITY_NUM and $LEVEL_NUM macros.

Configuring syslog-ng as a relay

Syslog-ng is pretty straight forward to configure as a relay.

source s_net { udp(); };
destination d_net{ udp(“192.168.0.1″ port(514)); };
log { source(s_net); destination(d_net); };

Designing A Log and Event Monitoring Program

Ultimately, as with all IT security programs, log monitoring programs are designed to address risks to data confidentiality, integrity and availability.  Risks come in many types:

• Hardware failure
• System compromise
• User error
• Rogue administrator

An organization’s program around log & event monitoring needs to be based on the specific risks that exist in that organization.  Consider two these two scenarios:

Scenario 1: The IT department of a large public corporation manages a large number of critical business systems, including systems that contain the company’s financial information, payroll information and client data.  This company cares deeply about the confidentiality, integrity and availability of it’s applications.

Scenario 2: A small Internet-based retailer of custom soaps is owned by a family.  Dave, a member of the family that owns the business, is technically savvy and has colocated a server to run the retailer’s web site.  Dave manages the server and the web site.  Dave also cares deeply about the confidentiality, integrity and availability of his company’s web site. Read more…