Logged

I have discussed in previous posts the importance of administrators using SUDO to provide individual accountability.  SUDO provides command-by-command accounting of actions performed by administrators, with logs sent as standard syslog events looking like this:

Feb  4 19:23:23 bsd sudo:    jerry : TTY=pts/0 ; PWD=/usr/home/jerry ; USER=root ; COMMAND=/bin/ps -x
Feb  4 19:23:34 bsd sudo:    jerry : TTY=pts/0 ; PWD=/usr/home/jerry ; USER=root ; COMMAND=/usr/bin/vi /etc/passwd
Feb  4 19:23:59 bsd sudo:    jerry : TTY=pts/0 ; PWD=/usr/home/jerry ; USER=root ; COMMAND=/usr/bin/tail -100 /var/log/messages

We can see pretty clearly all the actions I took above: the user “jerry” performed a number of actions, including one that is potentially concerning: vi /etc/passwd.  The action on /etc/passwd requires some investigation.

First, we need to be sure that an administrator can’t cover his tracks by deleting logs.  This is best accomplished by streaming the logs to a hardened syslog server, where the administrator doesn’t have the ability to delete logs.  Read more

Monitoring the activities of privileged users or server administrators is becoming a common requirement in many organizations for a few reasons:

• Compliance with legal or regulatory requirements, such as PCI, HIPAA, etc
• Performing outsourcing services to clients who require controls to prevent the service provider’s employees from causing harm to the client.
• A recent experience where a trusted employee performed some malicious action

In this realm of managing administrators, there are two primary objectives:

1. Individual accountability
2. Proactive monitoring of actions taken

Many administrators have the opinion that once you allow a person to act as root, all bets are off.  That is true to a large extent, and will require a fundamental change in thinking for some.  Controls need to be implemented to manage the actions of these privileged users in a manner that is commensurate with the risk of the system(s), applications and data being managed.  Read more

In this post, I will demonstrate a way to capture logs from a series of log files, and relay those logs to a central log server, where the logs will be separated into log files, as they existed on the original host.

Reading from files

Syslog-ng has the ability to pull log data from files, then treat those logs as any other source, like this:

source  s_file_abclog { file(“/var/log/abc.log” follow_freq(1));};

In this way, you can handle log data from applications that do not have native support for syslog.

Then, you can send that data to a central log server, like this:

source  s_file_abclog { file(“/var/log/abc.log” follow_freq(1));};

destination d_remote{udp(“192.168.0.3″ port(514)); };
log { source(s_file_abclog); destination(d_remote);}; Read more

In a previous post, I wrote about the general use of syslog logs as a method of ensuring compliance with policy.  This is a specific example of how one might use syslog to do that.

As IT operations mature, particularly in regulated environments, it is not uncommon for an organization’s security policy to require controls on the use of privileged ID’s.  Specifically, the use of the “root” or “administrator” account does not provide individual accountability.  A policy requirement might be to require the use of sudo or equivalent technology to allow administrators to perform their jobs without losing individual accountability.

But how can you show that the policy is effective?  Analyzing syslog logs are a good way.  If it is a policy exception that an administrator would log into a server with “root”, then you simply have look for cases of that occurring.  An ssh root login on FreeBSD looks like this:

Aug  7 17:22:00 www2 sshd[54036]: Accepted keyboard-interactive/pam for root from 127.0.0.1 port 50496 ssh2

Building a query can be as simple as “grep root /var/log/messages”, or can be performed by any number of syslog analyzers.  Of course, root would have the ability to remove the log evidence of his presence, which is a very good reason to relay logs to a central syslog server.  It then becomes imperative that administrators with access to root accounts on systems do not also have administrative access to the central syslog server.

Pulling evidence of root logins out of syslog logs from a central server which administrators do not have access is a good control to determine whether administrators are following policy requirements.  This solution allows the administrators to retain the root password, but only use it in the event of an emergency.  The policy should require reconciliation of any root logins found with known, documented issues.  Use of the password without a valid reason will show evidence of policy violation.

An obvious weakness of the syslog network protocol is the ease of spoofing messages into a central syslog server.   The default use of UDP as a transport and lack of any sort of authentication, in fact, make it trivial to spoof any part of a syslog message.

The most concerning issue with spoofing is faking the sending host.  An attacker can create a lot of chaos by stuffing log files with bogus errors, creating a denial of service potential or an opportunity for the attacker to distract administrators with false alarms while an attack takes place.

The most basic way to improve the situation is to transport syslog over TCP rather than UDP.  Both the syslog-ng and rsyslog servers support this natively.

Alternatively, syslog can be tunneled over stunnel, as described here.

In a previous post, we looked at installing Snare to log Windows events to a syslog server.  Here, we will configure syslog-ng to accept messages from Snare and implement a few simple customizations, including storing the logs in individual files.  We will assume that Snare is operational for the purposes of this guide.  Please see the post referenced above for help with installing Snare.

For this test, I am running syslog-ng 3.0.1 on FreeBSD 7.1 and Snare 3.14 on Windows XP.

First, we will start with a very basic configuration that logs to /var/log/messages:

source src {
internal();
udp(port(514));
};
destination messages { file(“/var/log/messages”); };

log {source(src); destination(messages);};

Read more

Maintaining a reliable and secure repository of logs is important for many reasons: establishing a foresnic trail of evidence in the case of fraud or attack, and enabling event correlation across many devices, among others.  Particularly in regulated industries, management should enact controls that prevent security, application and system logs from being tampered with.

Many organizations choose to consolidate their logs on to a centralized syslog server.  Many devices and just about all UNIX-like operating systems (Linux, free/net/open BSD, Solaris, AIX) support syslog natively.  Windows-based systems require a tool to convert event logs to syslog.

Syslog is a simple protocol and is easy to wrap some very effective security around.  The goal is remove as many opportunities for the central syslog server to be compromised as practical.  There are 3 aspects to hardening a syslog server that we’ll cover:

1. The operating system
2. The network
3. The application
4. The users and administrators Read more

I was just catching up on my reading on Technorati and came across this article that details the ways attackers can cover their tracks upon compromising a Windows server.  This article should server as a warning: if your logs are not moved off to a central server, you will lose visibility and key evidence on attacks.  This applies to any type of system, whether Windows, Linux, BSD or any other UNIX OS.

I strongly encourage the practice of centralizing logs to a hardened log server.  For Windows, there are a bunch of good applications that will export Windows Event Logs out to syslog.  I recently took a at logging Windows Events to a syslog server using Snare.

It is important to note that in the event of a successful compromise, the attacker will likely still disable logging and auditing, which will probably cause the stream of logs to the syslog server to cease.  The difference, though, is that the events which were captured during the attack remain on the log server, despite the attacker having deleted the local logs.  In a better case, the attacker does not disable logging and auditing first, opting to clear the event logs later in the attack, providng more evidence in the centralized logs of what was accessed or modified by the attacker.

There are now a bunch of commercial and open source agents that can run on a Windows system to take in Windows Event Logs and send them off to a syslog server.  We’ll be looking at the Snare agent in this post.

As of this writing, Snare is compatible with Windows NT, 2000, XP, 2003 and Vista.  There is also an agent available for 64 bit Windows versions.

For my test, I am installing on a Windows XP system.  Installation is quite straight forward.  There are MSI and scripted installers available on the Snare web site for large scale deployments.

The recommended installation has Snare taking control over the Event Log configuration, to synchronize the configurable logging “Objectives” in Snare with the Event Log settings.  Read more

Reports from system logs for compliance generally have the same basic requirements regardless of the standard being measured – whether PCI, SOX or FFIEC.  There are some foundational requirements for compliance reporting of logs to be considered effective:

• The data/time are synchronized throughout the environment.  This is vital to be able to correlate events between systems and to real-world events, such as security cameras, badge systems, etc.
• System, security and audit logs are sent to and stored on a system where users/administrators of monitored systems do not have access.   Logs will not be an effective identifier of fraud, theft or other nefarious acts if the perpetrator of those acts has the ability to remove log evidence of his activities.   Subscribing to a Log Management Service is a good way to address this concern.
• Individuals who access controlled systems should not have access to update or modify the scripts and/or software the produces the security reports.

The key elements for compliance log reports are: Read more