In this post, I will cover a basic set of best practices for managing logs. Depending on your specific objectives, regulatory requirements, and business constraints, there are likely to be a number of additional best practices.
- Forward syslog messages from clients to a secure syslog server.
- Enable NTP clock synchronization on all clients and on the syslog server. It is very important for all systems
reporting logs to be using the same time server, so that logs are all synchronized. Without doing this, it can be difficult or impossible to accurately determine the sequence of events across systems or applications.
- Group “like sources” into the same log file. (i.e. mail server, MTA, spamassassin and A/V scanner all report to one
- Use an automated tool to establish a baseline of your logs and escalate exceptions as appropriate.
- Review your records retention policy, if applicable, and determine if anything kept in logs falls under that policy. If so, establish retention periods based on the records policy. Legal requirements for keeping logs vary by jurisdiction and application.
- The “sweet spot” for log retention appears to be one year. Shorter than 1 year, and it is likely that key data would be unavailable in the wake of a long running attack, and longer than one year is most likely wasting disk space.
- Include logs and log archives in a standard backup process for disaster recovery.
- Change read/write permissions on logs files so they are not accessible to unprivileged user accounts.
Have more suggestions for logging best practices? Post them in a comment below.
Ultimately, as with all IT security programs, log monitoring programs are designed to address risks to data confidentiality, integrity and availability. Risks come in many types:
- Hardware failure
- System compromise
- User error
- Rogue administrator
An organization’s program around log & event monitoring needs to be based on the specific risks that exist in that organization. Consider two these two scenarios:
Scenario 1: The IT department of a large public corporation manages a large number of critical business systems, including systems that contain the company’s financial information, payroll information and client data. This company cares deeply about the confidentiality, integrity and availability of it’s applications.
Scenario 2: A small Internet-based retailer of custom soaps is owned by a family. Dave, a member of the family that owns the business, is technically savvy and has colocated a server to run the retailer’s web site. Dave manages the server and the web site. Dave also cares deeply about the confidentiality, integrity and availability of his company’s web site. Read more…
I have discussed in previous posts the importance of administrators using SUDO to provide individual accountability. SUDO provides command-by-command accounting of actions performed by administrators, with logs sent as standard syslog events looking like this:
Feb 4 19:23:23 bsd sudo: jerry : TTY=pts/0 ; PWD=/usr/home/jerry ; USER=root ; COMMAND=/bin/ps -x
Feb 4 19:23:34 bsd sudo: jerry : TTY=pts/0 ; PWD=/usr/home/jerry ; USER=root ; COMMAND=/usr/bin/vi /etc/passwd
Feb 4 19:23:59 bsd sudo: jerry : TTY=pts/0 ; PWD=/usr/home/jerry ; USER=root ; COMMAND=/usr/bin/tail -100 /var/log/messages
We can see pretty clearly all the actions I took above: the user “jerry” performed a number of actions, including one that is potentially concerning: vi /etc/passwd. The action on /etc/passwd requires some investigation.
First, we need to be sure that an administrator can’t cover his tracks by deleting logs. This is best accomplished by streaming the logs to a hardened syslog server, where the administrator doesn’t have the ability to delete logs. Read more…
Monitoring the activities of privileged users or server administrators is becoming a common requirement in many organizations for a few reasons:
- Compliance with legal or regulatory requirements, such as PCI, HIPAA, etc
- Performing outsourcing services to clients who require controls to prevent the service provider’s employees from causing harm to the client.
- A recent experience where a trusted employee performed some malicious action
In this realm of managing administrators, there are two primary objectives:
- Individual accountability
- Proactive monitoring of actions taken
Many administrators have the opinion that once you allow a person to act as root, all bets are off. That is true to a large extent, and will require a fundamental change in thinking for some. Controls need to be implemented to manage the actions of these privileged users in a manner that is commensurate with the risk of the system(s), applications and data being managed. Read more…
In a previous post, I wrote about the general use of syslog logs as a method of ensuring compliance with policy. This is a specific example of how one might use syslog to do that.
As IT operations mature, particularly in regulated environments, it is not uncommon for an organization’s security policy to require controls on the use of privileged ID’s. Specifically, the use of the “root” or “administrator” account does not provide individual accountability. A policy requirement might be to require the use of sudo or equivalent technology to allow administrators to perform their jobs without losing individual accountability.
But how can you show that the policy is effective? Analyzing syslog logs are a good way. If it is a policy exception that an administrator would log into a server with “root”, then you simply have look for cases of that occurring. An ssh root login on FreeBSD looks like this:
Aug 7 17:22:00 www2 sshd: Accepted keyboard-interactive/pam for root from 127.0.0.1 port 50496 ssh2
Building a query can be as simple as “grep root /var/log/messages”, or can be performed by any number of syslog analyzers. Of course, root would have the ability to remove the log evidence of his presence, which is a very good reason to relay logs to a central syslog server. It then becomes imperative that administrators with access to root accounts on systems do not also have administrative access to the central syslog server.
Pulling evidence of root logins out of syslog logs from a central server which administrators do not have access is a good control to determine whether administrators are following policy requirements. This solution allows the administrators to retain the root password, but only use it in the event of an emergency. The policy should require reconciliation of any root logins found with known, documented issues. Use of the password without a valid reason will show evidence of policy violation.
Reports from system logs for compliance generally have the same basic requirements regardless of the standard being measured – whether PCI, SOX or FFIEC. There are some foundational requirements for compliance reporting of logs to be considered effective:
- The data/time are synchronized throughout the environment. This is vital to be able to correlate events between systems and to real-world events, such as security cameras, badge systems, etc.
- System, security and audit logs are sent to and stored on a system where users/administrators of monitored systems do not have access. Logs will not be an effective identifier of fraud, theft or other nefarious acts if the perpetrator of those acts has the ability to remove log evidence of his activities. Subscribing to a Log Management Service is a good way to address this concern.
- Individuals who access controlled systems should not have access to update or modify the scripts and/or software the produces the security reports.
The key elements for compliance log reports are: Read more…
There are a growing number of Managed Security Service Providers (MSSP’s), such as IBM and Symantec, and Verisign, and other companies, such as Savvis, offering an outsourced service to collect and retain system logs, generally called a log management service (LMS). The initial instinct for many would be to reject such a crazy thought as outsourcing log management, but there are some big advantages, and some things to consider. Read more…
Organizations that need to minimize the risks associated with managing technology infrastructure implement robust policies on access management, change management and the like.
Having robust and well understood policies is important and expected of most organizations. However, organizations such as the FFIEC expects that financial institutions apply detective controls to affirmatively identify policy violations where ever possible.
RSS Error: A feed could not be found at http://www.syslog.org/forum/.xml/?type=rss. A feed with an invalid mime type may fall victim to this error, or SimplePie was unable to auto-discover it.. Use force_feed() if you are certain this URL is a real feed.