We’re back online, this time running on a Netra T1 105 in the Loglines datacenter in Almere

I’ve rebuilt and redesigned my home network. I’ve now got a separate management vlan, and even more routers & subnets There should be a nice new network map online within a couple of days.

I’ve been playing with prelude for a month of 3 now. It’s a hybrid IDS in the sense that it takes input from a variety of sensors (snort/syslog/nepenthes/pflog/etc), sends this to a manager (messages are in IDMEF format) after which you have a central repository of all security events in your network.

One of the less good things of prelude was the way it handled it’s database queries. For instance, if you needed to delete records from your database, you got a loop construction like this:

list_of_idents = SELECT DISTINCT
for ident in list_of_idents; do
for table in all_db_tables; do
SOME SQL QUERY WHERE ident = ident
done
done

This isn’t very efficient and after some playing with the –query-logging option from preludedb-admin I found that if you’d rebuild the loop construct as follows things start flying:

list_of_idents = SELECT DISTINCT
for table in all_db_tables; do
SOME SQL QUERY WHERE ident IN (list,of,idents)
done

Last weekend at megabit I was doubting werther to start learning C++ to create a patch or to create a tool that queries the db directly. I’ve chosen the last route. However, last tuesday I was discussing this optimalisation in the prelude irc channel and Yoann was pretty interested. Some days later and this is the result