Collecting/Analyzing evenlogs from an enterprise AD domain

[itguy]

We have about 200 Windows servers, with about 8 domain controllers supporting about 1200 users.  There is a lot of activity on the domain, and with the logs on a given domain controller set to rotate at 75 meg, we're rolling them about every 15 to 30 mintes on each DC.  What tools are there to collect and perform some reasonable analysis of those logs, plus give me some archiving capability?  That's a lot of logs - between 5 and 10 gigs per day.

[Henke]

Are you an windows only shop?
We also have alot of windows machines, not as many as you do though. About 60 Windows servers (10 DCs (we have a couple of offices), and another 25 FreeBSD.

First of all, do you have to log everything? (If you do?)
I usually set to log everything failed, and only a few successes on our windowsmachines. Depends on its task. For example; a busy fileserver, i usually dont set it to log every file access success. Only failed.

Then I would look into something like syslog-ng, store and rotate AND compress the logs every day/week/month depending on diskspace. Since its only text, it compresses rather well.

We save it like this "/$host/$year/$month/$day/$facility.log"
and rotate/compress it with the FreeBSD builtin newsyslog tool every month.

Also in syslog-ng, i would send the logs to a SQL server of choice. MySQL could be a good cheap choice since its only logs. And there is a nice interface called php-syslog-ng that is now on version 2.8 that works really well. And is so easy to setup. It comes with a rotatescript, so you could rotate the databaselogs aswell.

Oh, the client I have only run NT-Syslog. It is easy to specify Application/Security/System Facility/Priority with a litte .reg file you push to the servers you want. And you could .msi the nt-syslog installation.

Happy logging!

/Henrik

[Anonymous]

We do have a few unix systems,  but most everything is windows.  We want to log most things from the servers, so that we can log who creates and deletes AD users, directory permissions, etc., in addition to the things like failed logon attempts, and other system information.  
With 1200 users, we're constantly adding or deleting accounts and that responsibility is spread across 3 different helpdesks, one in each region of the world.  SOX has required us to implement a lot of policy on how we manage accounts and permissions, so we need to be able to audit who made what changes and when, then be able to tie them back to a change management or helpdesk ticket.
I am somewhat liking the idea of using syslog to collect all the logs - since then we can pull in logs from our routers and switches and unix machines, and apply a consistant monitoring/alerting/auditing process to all of those systems.

[mutex]

you're going to need to put some effort into creating a log analysis engine using a tool like devialog or one of the many others.  I would also recommend logging to a file and using that to archive the logs, and using the logs that are written to the database to write your analysis engine & to view the logs.