Using Trends In Logs To Define New Security Requirements For Internet Facing Hosts

I have a few servers at a colocation datacenter for running a number of sites, including this one.  I have written before about detecting brute force attacks in logs.   I have been watching the attacks continue in my logs, and have noticed a few things:

1. The attacks, as before, are coming from many different sources, nearly simultaneously.

2. It’s interesting that the brute force account and password guessing are so well coordinated – generally I see the same user name tried by multiple hosts sequentially, the moving on to new name, usually in alphabetical order.

3. The attacks are now coming in via multiple vectors.  Previously, the attacks were only carried out using ssh connections, but are not also using pop3, imap and ftp.

4. The attacks have started trying to intelligently guess the name of accounts on the server, based on the domain name.  In my case, I am using cPanel/WHM, where each domain has a shell account, and generally the user name is some derivation of the domain name.   Previously, I would only see a dictionary of names being sequentially tried, but now I see the user name “syslog” being tried many times.  From the perspective of an attacker, it’s much better to only have one variable to brute force (the password) rather than two (user name and password).

So, what can I infer about the attackers given the data I have seen?

1. There are some number of botnets being used to guess passwords of accounts, and the zombies are being very well coordinated by some command and control infrastructure.

2. The attackers are trying very hard to infect web servers.  Once an attacker has access to modify a site, he can do a number of things; from send spam to integrate browser exploit code into the normal site(s).  The latter is particularly interesting, since there is substantial money to be made from installing adware/spyware/other maleware on desktops.

What can be done to counter these threats?

1. Enforce minimum password complexity  (8 character minimum is a good start)

2. Choose user names that can’t be directly linked to a domain.  For instance, the user name “syslog” would be easily guessable for the domain www.syslog.org.  A better name might be “syslog22″.

3. If unique usernames are used as identified in #2 above, logs can be monitored proactively for instances where the real account name is being tried unsuccessfully.  If there is not an externally discernible way to identify the user name,appearances of such entries in logs could indicate that the system has been compromised in some other manner.

Pot Of Syslog-NG Tricks Version 2

Correcting bad or duplicate time and date stamps

Trying to accept logs from applications or devices into syslog-ng, but end up seeing two date and time fields in the resulting log coming out of syslog-ng?  This happens because syslog-ng is not able to understand the format that the date and time stamp arrive in.  Here’s an easy way to fix that problem.  Use a template to only use the “message” part of the incoming log:

source s_net { udp(); };
destination d_file{ file(“/var/log/application.log” template(“$MSG\n”)); };
log { source(s_net); destination(d_file); };

Including the facility & priority of a message syslog-ng output

To add the written form of the facility and priority to logs coming out of syslog-ng, such as “local1:info”, use the following template.

destination d_file {
file(“/var/log/messages”
template(“$FACILITY:$PRIORITY $MSG\n”; template_escape(no))); };

Alternatively, to add the numeric severity, use this:

destination d_file {
file(“/var/log/messages”
template(“$PRI $MSG\n”; template_escape(no))); };

Configuring syslog-ng as a relay

Syslog-ng is pretty straight forward to configure as a relay.

source s_net { udp(); };
destination d_net{ udp(“192.168.0.1″ port(514)); };
log { source(s_net); destination(d_net); };

Designing A Log and Event Monitoring Program

Ultimately, as with all IT security programs, log monitoring programs are designed to address risks to data confidentiality, integrity and availability.  Risks come in many types:

• Hardware failure
• System compromise
• User error
• Rogue administrator

An organization’s program around log & event monitoring needs to be based on the specific risks that exist in that organization.  Consider two these two scenarios:

Scenario 1: The IT department of a large public corporation manages a large number of critical business systems, including systems that contain the company’s financial information, payroll information and client data.  This company cares deeply about the confidentiality, integrity and availability of it’s applications.

Scenario 2: A small Internet-based retailer of custom soaps is owned by a family.  Dave, a member of the family that owns the business, is technically savvy and has colocated a server to run the retailer’s web site.  Dave manages the server and the web site.  Dave also cares deeply about the confidentiality, integrity and availability of his company’s web site. Read more…

Running Syslog-NG on Windows

This post describes running syslog-ng as a server on Windows.  In another post, we describe how to send Windows Event Logs to syslog.

There are many great commercial syslog servers for Windows.  There are not many options for those looking for a free alternative.  One option is Aonaware.  Another option is to install syslog-ng through cygwin.  Cygwin is a Linux-like environment run inside a windows command shell.  Cygwin runs on all current desktop and server versions of Windows.  In this post, we will walk through setting up syslog-ng on a windows host.

1. First, visit the Cygwin website to download the setup.exe application.  Save, then run setup.exe.
2. Choose “Install From Internet”
3. Select the directory to install into and the user to install for (leave this as “all users”).
4. Enter the directory for local packages. Accepting the default location is fine.
5. Choose your Internet connection type (direct or proxy)
6. Select a site to download from.  Any one should be fine.
7. At the “install packages” window, type is “syslog” in the search box.  You will see “Admin” below.  Expand the admin section, and you will see syslog-ng.  Click the word “skip” until you see 3.0.1 (or whatever the latest supported version is).
8. Also choose the following packages:
   1. Admin/cygrunsrv
   2. Editors/VIM
   3. Gnome/glib
9. Finish the installation. Read more…

Pot Of Syslog-NG Tricks Version 1

Fixing Duplicate Date and/or Hostname Problems

Some devices send syslog messages with improperly formatted headers, which can cause syslog-ng to append a new set of header information, meaning that the host name and/or date appear twice in the logs.  A simple way to solve this is using a template:

source s_net { udp();};
destination d_file { file(“/var/log/file.log” template(“$MSG\n”)); };
log {source(s_net); destination(d_file); };

The template function in the destination definition tells syslog-ng to discard the header from the received message and only keep the actual message data, which is contained in the variable $MSG.   Syslog-ng inserts the proper header, using the date the log was received and the host the log came from, resulting in a normal syslog message. Read more…

Defining Log Management and Log Monitoring Objectives

System logs are good for more purposes that many people realize.  In this post, I’ll describe the four broad categories of log usage.

Forensic Record Keeping

Many organizations choose to archive log data for a period of time for future reference.  Generally, the usefulness of keeping archived logs comes from the situation where a system problem is discovered some time in the future, and archived logs can help to piece together the cause of what happened, and can often be used as evidence in court should such a thing become necessary.

Proactive Identification of System Health Problems

Alerting tools can be used to monitor logs streamed from devices and notify operators of a problem, such as a failing disk, low memory, crashing processes, etc.  A big advantage of this is that many serious problems can be remediated before they impact the availability or performance of the system, by removing stale files to add drive space, contacting a vendor for a preemptively hardware replacement and so on.

Proactive Activity Monitoring of Users and Administrators

This is one area of log management that many organizations unintentionally overlook, because the perceived risk is not very high, the administrators are unaware of the options, and more commonly, the view that monitoring the activity of administrators is futile because such monitoring could be bypassed by a skilled administrator.  In separate post, I describe a process to monitor the activities of administrators.

Intrusion Detection

This is arguably the most common use of system logs – looking for evidence of system compromise.  Like system health monitoring, there are many tools on the market that can take in log streams and notify operators or security personnel when an event is detected.

A Note on Protecting Logs

In all cases, it is important to send logs to another system for storage, either real time or by some frequent batch process.  Protecting logs from a failing hard drive, crafty hacker or disgruntled administrator can be achieved by using a hardened syslog server.

Configuring SUDO for Effective Activity Monitoring Via Syslog

I have discussed in previous posts the importance of administrators using SUDO to provide individual accountability.  SUDO provides command-by-command accounting of actions performed by administrators, with logs sent as standard syslog events looking like this:

Feb  4 19:23:23 bsd sudo:    jerry : TTY=pts/0 ; PWD=/usr/home/jerry ; USER=root ; COMMAND=/bin/ps -x
Feb  4 19:23:34 bsd sudo:    jerry : TTY=pts/0 ; PWD=/usr/home/jerry ; USER=root ; COMMAND=/usr/bin/vi /etc/passwd
Feb  4 19:23:59 bsd sudo:    jerry : TTY=pts/0 ; PWD=/usr/home/jerry ; USER=root ; COMMAND=/usr/bin/tail -100 /var/log/messages

We can see pretty clearly all the actions I took above: the user “jerry” performed a number of actions, including one that is potentially concerning: vi /etc/passwd.  The action on /etc/passwd requires some investigation.

First, we need to be sure that an administrator can’t cover his tracks by deleting logs.  This is best accomplished by streaming the logs to a hardened syslog server, where the administrator doesn’t have the ability to delete logs.  Read more…

Building A Program To Manage And Monitor Administrators

Monitoring the activities of privileged users or server administrators is becoming a common requirement in many organizations for a few reasons:

• Compliance with legal or regulatory requirements, such as PCI, HIPAA, etc
• Performing outsourcing services to clients who require controls to prevent the service provider’s employees from causing harm to the client.
• A recent experience where a trusted employee performed some malicious action

In this realm of managing administrators, there are two primary objectives:

1. Individual accountability
2. Proactive monitoring of actions taken

Many administrators have the opinion that once you allow a person to act as root, all bets are off.  That is true to a large extent, and will require a fundamental change in thinking for some.  Controls need to be implemented to manage the actions of these privileged users in a manner that is commensurate with the risk of the system(s), applications and data being managed.  Read more…

Segregating Logs From Different Log Files On A Centralized Log Server Using Syslog-NG

In this post, I will demonstrate a way to capture logs from a series of log files, and relay those logs to a central log server, where the logs will be separated into log files, as they existed on the original host.

Reading from files

Syslog-ng has the ability to pull log data from files, then treat those logs as any other source, like this:

source  s_file_abclog { file(“/var/log/abc.log” follow_freq(1));};

In this way, you can handle log data from applications that do not have native support for syslog.

Then, you can send that data to a central log server, like this:

source  s_file_abclog { file(“/var/log/abc.log” follow_freq(1));};

destination d_remote{udp(“192.168.0.3″ port(514)); };
log { source(s_file_abclog); destination(d_remote);}; Read more…

Using Syslog Logs For Validation of Security Policy Compliance

In a previous post, I wrote about the general use of syslog logs as a method of ensuring compliance with policy.  This is a specific example of how one might use syslog to do that.

As IT operations mature, particularly in regulated environments, it is not uncommon for an organization’s security policy to require controls on the use of privileged ID’s.  Specifically, the use of the “root” or “administrator” account does not provide individual accountability.  A policy requirement might be to require the use of sudo or equivalent technology to allow administrators to perform their jobs without losing individual accountability.

But how can you show that the policy is effective?  Analyzing syslog logs are a good way.  If it is a policy exception that an administrator would log into a server with “root”, then you simply have look for cases of that occurring.  An ssh root login on FreeBSD looks like this:

Aug  7 17:22:00 www2 sshd[54036]: Accepted keyboard-interactive/pam for root from 127.0.0.1 port 50496 ssh2

Building a query can be as simple as “grep root /var/log/messages”, or can be performed by any number of syslog analyzers.  Of course, root would have the ability to remove the log evidence of his presence, which is a very good reason to relay logs to a central syslog server.  It then becomes imperative that administrators with access to root accounts on systems do not also have administrative access to the central syslog server.

Pulling evidence of root logins out of syslog logs from a central server which administrators do not have access is a good control to determine whether administrators are following policy requirements.  This solution allows the administrators to retain the root password, but only use it in the event of an emergency.  The policy should require reconciliation of any root logins found with known, documented issues.  Use of the password without a valid reason will show evidence of policy violation.