Log Management

Defining Log Management and Log Monitoring Objectives

System logs are good for more purposes that many people realize.  In this post, I’ll describe the four broad categories of log usage.

Forensic Record Keeping

Many organizations choose to archive log data for a period of time for future reference.  Generally, the usefulness of keeping archived logs comes from the situation where a system problem is discovered some time in the future, and archived logs can help to piece together the cause of what happened, and can often be used as evidence in court should such a thing become necessary.

Proactive Identification of System Health Problems

Alerting tools can be used to monitor logs streamed from devices and notify operators of a problem, such as a failing disk, low memory, crashing processes, etc.  A big advantage of this is that many serious problems can be remediated before they impact the availability or performance of the system, by removing stale files to add drive space, contacting a vendor for a preemptively hardware replacement and so on.

Proactive Activity Monitoring of Users and Administrators

This is one area of log management that many organizations unintentionally overlook, because the perceived risk is not very high, the administrators are unaware of the options, and more commonly, the view that monitoring the activity of administrators is futile because such monitoring could be bypassed by a skilled administrator.  In separate post, I describe a process to monitor the activities of administrators.

Intrusion Detection

This is arguably the most common use of system logs – looking for evidence of system compromise.  Like system health monitoring, there are many tools on the market that can take in log streams and notify operators or security personnel when an event is detected.

A Note on Protecting Logs

In all cases, it is important to send logs to another system for storage, either real time or by some frequent batch process.  Protecting logs from a failing hard drive, crafty hacker or disgruntled administrator can be achieved by using a hardened syslog server.

Segregating Logs From Different Log Files On A Centralized Log Server Using Syslog-NG

In this post, I will demonstrate a way to capture logs from a series of log files, and relay those logs to a central log server, where the logs will be separated into log files, as they existed on the original host.

Reading from files

Syslog-ng has the ability to pull log data from files, then treat those logs as any other source, like this:

source  s_file_abclog { file(“/var/log/abc.log” follow_freq(1));};

In this way, you can handle log data from applications that do not have native support for syslog.

Then, you can send that data to a central log server, like this:

source  s_file_abclog { file(“/var/log/abc.log” follow_freq(1));};

destination d_remote{udp(“192.168.0.3″ port(514)); };
log { source(s_file_abclog); destination(d_remote);}; Read more…

Establishing a Hardened Syslog Log Server

Maintaining a reliable and secure repository of logs is important for many reasons: establishing a foresnic trail of evidence in the case of fraud or attack, and enabling event correlation across many devices, among others.  Particularly in regulated industries, management should enact controls that prevent security, application and system logs from being tampered with.

Many organizations choose to consolidate their logs on to a centralized syslog server.  Many devices and just about all UNIX-like operating systems (Linux, free/net/open BSD, Solaris, AIX) support syslog natively.  Windows-based systems require a tool to convert event logs to syslog.

Syslog is a simple protocol and is easy to wrap some very effective security around.  The goal is remove as many opportunities for the central syslog server to be compromised as practical.  There are 3 aspects to hardening a syslog server that we’ll cover:

1. The operating system
2. The network
3. The application
4. The users and administrators Read more…

Logging Windows Events To Syslog Using Snare

There are now a bunch of commercial and open source agents that can run on a Windows system to take in Windows Event Logs and send them off to a syslog server.  We’ll be looking at the Snare agent in this post.

As of this writing, Snare is compatible with Windows NT, 2000, XP, 2003 and Vista.  There is also an agent available for 64 bit Windows versions.

For my test, I am installing on a Windows XP system.  Installation is quite straight forward.  There are MSI and scripted installers available on the Snare web site for large scale deployments.

The recommended installation has Snare taking control over the Event Log configuration, to synchronize the configurable logging “Objectives” in Snare with the Event Log settings.  Read more…

What To Look For In A Compliance Report From Logs

Reports from system logs for compliance generally have the same basic requirements regardless of the standard being measured – whether PCI, SOX or FFIEC.  There are some foundational requirements for compliance reporting of logs to be considered effective:

• The data/time are synchronized throughout the environment.  This is vital to be able to correlate events between systems and to real-world events, such as security cameras, badge systems, etc.
• System, security and audit logs are sent to and stored on a system where users/administrators of monitored systems do not have access.   Logs will not be an effective identifier of fraud, theft or other nefarious acts if the perpetrator of those acts has the ability to remove log evidence of his activities.   Subscribing to a Log Management Service is a good way to address this concern.
• Individuals who access controlled systems should not have access to update or modify the scripts and/or software the produces the security reports.

The key elements for compliance log reports are: Read more…

Why Using A Log Management Service Might Be Right For You

There are a growing number of Managed Security Service Providers (MSSP’s), such as IBM and Symantec, and Verisign, and other companies, such as Savvis, offering an outsourced service to collect and retain system logs, generally called a log management service (LMS).  The initial instinct for many would be to reject such a crazy thought as outsourcing log management, but there are some big advantages, and some things to consider. Read more…

Creative Use of System Logs to Ensure Policy Compliance

Organizations that need to minimize the risks associated with managing technology infrastructure implement robust policies on access management, change management and the like.

Having robust and well understood policies is important and expected of most organizations.  However, organizations such as the FFIEC expects that financial institutions apply detective controls to affirmatively identify policy violations where ever possible.

Read more…