Logging and Syslog Best Practices
In this post, I will cover a basic set of best practices for managing logs. Depending on your specific objectives, regulatory requirements, and business constraints, there are likely to be a number of additional best practices.
- Forward syslog messages from clients to a secure syslog server.
- Enable NTP clock synchronization on all clients and on the syslog server. It is very important for all systems
reporting logs to be using the same time server, so that logs are all synchronized. Without doing this, it can be difficult or impossible to accurately determine the sequence of events across systems or applications. - Group “like sources” into the same log file. (i.e. mail server, MTA, spamassassin and A/V scanner all report to one
file) - Use an automated tool to establish a baseline of your logs and escalate exceptions as appropriate.
- Review your records retention policy, if applicable, and determine if anything kept in logs falls under that policy. If so, establish retention periods based on the records policy. Legal requirements for keeping logs vary by jurisdiction and application.
- The “sweet spot” for log retention appears to be one year. Shorter than 1 year, and it is likely that key data would be unavailable in the wake of a long running attack, and longer than one year is most likely wasting disk space.
- Include logs and log archives in a standard backup process for disaster recovery.
- Change read/write permissions on logs files so they are not accessible to unprivileged user accounts.
Have more suggestions for logging best practices? Post them in a comment below.
re “sweet spot” I would suggest 13 months this allows you to see if the same problems occurred last year!
A log entry is created for each event or transaction that takes place on any machine or piece of hardware—think of it as acting as your “journal of record”. Microsoft-based systems generate Windows Event Log files, and UNIX-based servers and networking devices use the System Log or Syslog standard. Event Log Management is a key component of your compliance initiatives, since you can monitor, audit, and report on file access, unauthorized activity by users, policy changes, or even major changes in organizational roles via group membership.
Learn more tips and tricks from the experts at WhatsUp Gold to maximize the effectiveness of your log retention efforts at http://bit.ly/hu79AG.
Whilst I agree NTP is good practice in general, it is not actually necessary in this case as the syslog daemon should record the system time when a record was received.
As for how long to keep records: in my experience even a medium-small organisation can generate over 50,000 entries a day. Keeping that many records for 13 months is pointless, especially as most of the messages have no relevance after a few minutes.
What is much more useful is to have good filters and analysis tools, and to process and act upon messages in good time. You can use filters to throw away messages that are no longer relevant and keep important info such as security related messages.
Analysis tools help the admin to focus where action is necessary.
Hi,
I’m the author of both LogZilla (http://www.logzilla.info) and the Cisco whitepaper that (it appears) some of your advice above came from (http://www.cisco.com/en/US/technologies/collateral/tk869/tk769/white_paper_c11-557812.html).
I think it’s pretty cool that people are referencing my work, and that the public is starting to finally understand the importance of managing syslog, thanks for building this website!
@dave, you are correct about NTP, since the more advanced daemons like syslog-ng will TS when the message is received – however, on very large systems, the OS may actually get the wrong TS due to slow performance, so it may still be useful in those situations.
Re: Sweet spot, this is also depends on the company’s compliance regulations. For example, financial institutions are required by PCI (payment card industry) to keep at least 1 year of log data.
Clayton,
Thank you for posting the link to your paper – I had not seen that before, but I really wish I had read it years ago. I wrote the original post based on my experiences of running security & compliance for a company and slugging through numerous FFIEC examinations and SAS70 audits. They must be relatively good rules of thumb if we came up with a similar list independently.
Thanks again for posting and I hope you are able to continue contributing to discussions here!
Regards,
Jerry
Thanks Jerry,
I would be happy to help out if you can add me to some type of email notification list
P.S. I must be getting old, I can’t see the captcha’s very well anymore
I have disabled the captcha for registered users, so you shouldn’t be asked for captcha’s any more. I will work on the notification.
Has anyone talked about Stealth Syslog server ?
Same principle has transparent firewall, the syslog server has no IP address but get the syslog message sent by the machines on multicast/UDP.
Anybody have some insight on best practice for syslog to two separate targets. Say, both Orion and something simple like the free Kiwi syslog server that just generates a simple text file? Thanks!
Thanks — these are some excellent best practices for log management. You can also find a wealth of Syslog references — from case studies of how Syslog has worked for users to a list of requirements for optimally running it — at WhatsUp Gold’s community site:
http://community.whatsupgold.com/blogs/bestpracticestipstricks
I note your one year “sweet spot” but some require up to three years. I would think at least 2 years just so that there is enough time for comparison and a way to spot anomalous data.
Thank you for the feedback This certainly is something that each organization has to make their own assessment of, because there are many differing regulatory requirements.
Having said that, I wholeheartedly agree that keeping logs across years will give a much better capability to understand an attack that may have only recently become known.
From an operational perspective I have seen that keeping logs in large organizations is really problematic.
We have over 10 million log file messages per day across our infrastructure, and the vast majority of those messages have no meaning after 5 minutes.
A log entry that says a user port on a switch went down is usually not important and can be thrown away quite soon. A message saying an unauthorised access occured on a server is rather more important and should be kept. For how long depends on the industry – and if there are legal or regulatory requirements
People are notoriously reluctant to throw away information, but even if you only have a few 1000 messages a day, you still need to design a data reduction policy. For example, you can throw away the actual messages, but perhaps keep statistics on the types and frequency of events.
Compression for log messages is very high and space is very cheap these days, so I don’t really see the need for deleting logs.
The end-user port flap you considered “not interesting” today might be good info when you investigate a physical security breach in your offices on a Sunday morning at 3 AM.
With regard to NTP, keep in mind that without NTP looking at logs on different systems becomes more complicated and less useful.
If, for example, you are comparing what you’re seeing in syslog to what you see in detailed debug information on one or more systems or devices that are sending the information the logs will not easily match up if time has not been synchronized.