The messages that are contained in a system's log files are typically the only record and evidence of what has happened on a system. These logs are often the only way for you to determine the source of a problem after it has happened.
But, what happens when some event happens to your system and those logs are strangely missing? In this case, there is very little that can be done, at least quickly. Log files are prone to destruction from hard drive failures as well as system compromises. Attackers will often shut down syslogd and delete log files once they have control of a system.
To guard against these potential problems, as well as provide the functionality for centrally managing and analyzing your logs, set up a central syslog repository. From a very high level, you want to create a server that listens for syslog messages coming at it from the network, then tell all of your servers/workstations to send a copy of their syslog messages to that server.
On the server side, you can get as fancy or as basic as you want. Out of the box, most versions of Unix/Linux/BSD will not accept syslog messages from remote machines. The reason is actually pretty straight forward – syslog simply takes messages and dumps them into a file. That behavior essentially provides a mechanism for a denial of service attack on your syslog server. All he has to do is send swarms of syslogs long enough to fill up your hard drive. Such an attack can not only fill up a drive, but also consume considerable cpu time will it is being done, potentially causing some syslog messages from other machines to be dropped (syslog is, after all, a UDP based protocol.
To enable logging from remote machines on linux, you need to add the ‘-r’ flag to etc/sysconfig/syslog. On FreeBSD, you will have to add each host to the syslog start up flags in /etc/rc.conf, like this:
syslogd_flags="-s –a 192.168.0.1/32
Where the last part is the IP address or networks that you want to be able to log to your server. The client side is equally simple. In /etc/syslog.conf, copy any logging service line, and replace the name of the log file with “@hostname” where hostname is the name or IP of your syslog server. That will provide a working central syslog repository. There are many refinements that can be made to this, though. If security was a motivation in the creation of the syslog repository, dedicate a machine to
solely serving as a syslog repository. Turn off all other services, save maybe ssh. This will prevent someone who has compromised one of your machines from then figuring out that you have syslogs going to another server, and being able to compromise that server, and in turn delete it’s logs. Next, using a different syslogd implementation like metalog or syslog-ng, you can do things like separate syslog messages into different files based on which host they came from, or write them into a mysql database.
Certainly a large benefit to having all of your logs in one place is being able to correlate events between systems, and applying a consistent level of monitoring across all of your systems. More on that in future articles, though.