syslog data mining

[freefaller]

I have been building a data warehouse of sorts.  I collect all syslog traffic from two Cisco PIX firewalls on a linux server and run various perl programs I created to filter for porn, intrusion attempts, etc, and load a database for further analysis.  Anybody else doing something similar and looking to do some collaboration?

Also, I'm looking for opinions on a syslog replacement.  I have looked at but not tried syslog-ng.  Comments?

Thanks.
--
Freefaller

[Anonymous]

With sarbanes oxley and the anti fraud/money laundering provisions of the usa patriot act data mining syslog has all kinds of opportunities.  I had to roll my own scripts because I found that scripts like swatch were not sufficient.  Although its been a while since I did my testing, IIRC many programs like swatch did not provide a method to correlate events from separate sources over delta time.  For instance a PBX and a firewall.  Combining multi source correlation with firewalls that support labels also allows the dynamic addition and removal of rules.  For instance a feedback loop can be created with an IDS to block a particular address when a set of conditions is met.  Another thing that I could not find was a tool to reduce noise.  For instance a threshold counter is set to 24 upon seeing a particular log entry.  Each hour of normal activity the counter is decremented by one.  Of course there are lots of other algorithms that could be applied.  Bayesian filtering might be able to be applied to look for patterns of events.  I'm also looking for a method to audit whom has viewed the syslogs without sacrificing the flexible parsing.  Core Security has a free one.

[Anonymous]

Hi,

I can recommend www.syslogserver.com. A version 2.0 is due soon, with a new pricing policy :-)

It supports filters based on sql querys, which can be modified for arbitrary complexity. Alarms are supported too.

[Guest]

Try Kiwi Syslog Daemon.  It's free and works very well (it's for a win 9x/NT/NT5.x box) and has some nice settings, such as it views/shows syslog alerts and can email you/play a sound/alarm if the syslog alert is <= what you set.

[syslogguru]

Try Kiwi Syslog Daemon.  It's free and works very well (it's for a win 9x/NT/NT5.x box) and has some nice settings, such as it views/shows syslog alerts and can email you/play a sound/alarm if the syslog alert is <= what you set.

Folks
1. The problem with home grown tools like this is that they cannot scale in rea time. ie , say your PIX fires and event and then the IDS fires another event for a packet which traversed thro the firewall and was promiscuously detected by the IDS as a positive/ negative.

2. The tools / scripts you are mentioning cannot co-relate these two events and then pin point whats going on .

3. Take a look at this solution offered by http://protegonetworks.com
This solution can collect, corelate and normalise upto 20k events per second and store all the data in its terabyte storage. Should there be an anomaly , the tool can display charts/ attack vetco graphs.

[rgerhards]

1. The problem with home grown tools like this is that they cannot scale in rea time. ie , say your PIX fires and event and then the IDS fires another event for a packet which traversed thro the firewall and was promiscuously detected by the IDS as a positive/ negative.

You should be a little careful with your wording. Kiwi is a competitor of ours. But I don't think you should speak in a disregardful way.

After all, not everyone needs realtime, multi-terabyte analysis.

And if you would like to see real performance on a simple PC-class machine, you can also go and have a look at our software at http://www.winsyslog.com.

BTW: Unix syslogd is also "home-grown" []
Rainer

[Clemens]

ok, but which tool is offering:

- full syslog support, meaning of course collecting, archiving, but also automatic/manual reporting

- it should be able to follow rules, up to reacting in real-time on specified syslog messages

- should be able to filter widely and summarize specified syslogs

- should support all devices, of course

- and most important should be accessible and administrable remotly i.e. by a webaccess for different users

Could not find a single tool like that, specified on syslog!!!

Regards,
Clemens

[mutex]

I do not believe you are going to find a single tool that does all of that.  To get meaningful use out of such a system, I'm thinking that it will have to be quite tailored to each individual environment.  That kind of setup works if you're buying peoplesoft or SAP, but not a syslog watcher.  
The way I've done this in the past is to either have a program tailing the logfiles and executing on certain conditions (I've done a lot of this with php cli - the seemless database interface really opens up possibilities for looking at trends in addition to reacting to a single syslog),
or
use metalog (http://metalog.sourceforge.net/) in place of syslogd which can fire off an event when a certain syslog is received.

I know this isn't what you're looking for.  Hopefully it'll give you some ideas, though.

[mutex]

Just a follow up, it looks like most, if not all the requirements Clemens lists above is achieveable with Monitorware's application (see http://www.monitorware.com/en/)
I have not used it personally, but reading through their literature, I'd say it's worth looking into.

[Anonymous]

I've seen sysklogd-sql in action - it logs syslog data directly to a MySQL or PostgreSQL database which can then be searched.  It's open source and "home grown" as the threads below speak about - but it does the job and allows you to write whatever you wish to parse the database.
If your interested, you can find it at http://www.monkeymental.com

[Anonymous]

well, I tried Monitorware, but it is not remote accessable (I am not talking about pcAnywhere, Telnet, etc.) as I need it to be and it does not do reports. Alerting it does, but we need more!

At the moment I am trying some "homegrown" tool called SMT by Mr. "homegrower" Guthrie but it has deficits in reporting capabilities and needs some further adjustments. www.dangermen.com/smt

I might have a look at the monkeys later on.

Thanks guys,
Clemens

[mutex]

Another tool I recently found is http://www.eventlogmanager.com.  Like the others, this one isn't remotely accessible.  I think the best you'll be able to do is term-serve into the "console" to view it.   That tool really seems to target data mining and event correlationof syslog messages.  Please note, I haven't actually tried the product, but it does look like you can demo it before you buy (and the purchase price seems reasonable to me, anyhow).

[Anonymous]

Logalot, http://www.somix.com, is worth a look.  Rule creation is easy, web interface with many security levels, great web based, emailed reports, and report overlay options for event correlation.  Windows or Linux.  There is a demo that allow for monitoring 3 devices.

[Admin]

logalot looks very nice.  I'm sure the originator of this thread has long since solved their problems, but that does seem to be exactly what they were asking for on all fronts.

[Anonymous]

No one pay attention to SEC(Simple Event Correlation)?