Handling Spam Surges

From: Paul Griffith <paulg(at)cse.yorku.ca>
Date: Mon Sep 10 2007 - 12:48:20 EDT

Greetings,

How do you handle Spam surges/DoS attacks? We just had a Spam surge/DoS and are looking at ways to better withstand (as best as we can) another surge

Here is how we start SA:

-c -d -r $PIDFILE -s /var/log/spamd --socketpath=$SOCKET --max-children=150 --min-children=10

Our (1) mail server is configured like this:

CentOS 4.5
Exim 4.67
SpamAssassin version 3.2.3 running on Perl version 5.8.8 ClamAV 0.91.2 (saneSecurity updates)
- handles incoming/outgoing mail
- handles imap/pop/webmail request

Intel D Cpu 3.00Ghz with 2GB of Mem
80GB SATA root disk
200GB SATA mail disk (softraid mirror)
2xIntel e1000

Our mail server was taking a pounding on Friday,

Fri Sep  7 16:17:09 2007 [26914] info: prefork: child states: BBBBB
Fri Sep  7 16:17:09 2007 [26914] info: prefork: child states: BBBBBB
Fri Sep  7 16:17:09 2007 [26914] info: prefork: child states: BBBBBBB

..snip...

..snip...
..snip...

Fri Sep 7 16:17:17 2007 [26914] info: prefork: child states:

BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB

Fri Sep 7 16:17:17 2007 [26914] info: prefork: child states:

BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBIBBB

Fri Sep 7 16:17:19 2007 [26914] info: prefork: child states:

BBBBBBBBBBBBBBBBBBBBBBBBBIBBBBBBBBBBBBBBBBBBBBBBIBBB

..snip..
..snip..

Fri Sep 7 16:19:22 2007 [26914] info: prefork: child states:

BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBSBBSBB

Fri Sep 7 16:19:23 2007 [26914] info: prefork: child states:

BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBIBBISBB

At the mist of the surge we had 95 child processess running, all busy!

Here are the sar memory stats...

               kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad

16:10:02        16804   2056424     99.19      2900   1310880    
2040036       208      0.01         0
16:20:10        37676   2035552     98.18      1872    237376   1736152     
304092     14.90     78992
16:30:51        13924   2059304     99.33      1292    308944   1044160     
996084     48.82    357444
16:40:02        76652   1996576     96.30      8208   1280796   1756236     
284008     13.92    178696
Average:        26403   2046825     98.73      5880   1364057    
2024199     16045      0.79      6152

Here are the warnings we saw in the spamd log...

Fri Sep 7 16:20:39 2007 [26914] info: prefork: child states:

IBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB

Fri Sep 7 16:20:40 2007 [25431] warn: spf: lookup failed: Can't locate object method "new" via package "Net::DNS::RR::TXT" at /xsys/lib/perl5/site_perl/5.8.8/i686-linux/Net/ DNS/RR.pm line 312.
Fri Sep 7 16:20:41 2007 [25428] warn: spf: lookup failed: Can't locate object method "new" via package "Net::DNS::RR::TXT" at /xsys/lib/perl5/site_perl/5.8.8/i686-linux/Net/ DNS/RR.pm line 312.

Fri Sep 7 16:22:18 2007 [24684] warn: plugin: eval failed: child processing timeout at /xsys/sbin//spamd line 1246, <GEN683> line 3398. Fri Sep 7 16:22:20 2007 [24711] warn: Use of uninitialized value in pattern match (m//) at /xsys/lib/perl5/5.8.8/utf8_heavy.pl line 211, <GEN749> line 3398.
Fri Sep 7 16:22:20 2007 [24711] warn: Use of uninitialized value in scalar assignment at /xsys/lib/perl5/5.8.8/utf8_heavy.pl line 227, <GEN749> line 3398.

Fri Sep 7 16:26:15 2007 [25227] info: spamd: clean message (1.5/5.0) for cs242027:9190 in 406.1 seconds, 243776 bytes. Fri Sep 7 16:26:19 2007 [24688] warn: spamd: copy_config timeout, respawning child process after 1 messages at /xsys/sbin//spamd line 1103. Fri Sep 7 16:26:24 2007 [25046] warn: spamd: copy_config timeout, respawning child process after 1 messages at /xsys/sbin//spamd line 1103. Fri Sep 7 16:26:28 2007 [24692] warn: spamd: copy_config timeout, respawning child process after 1 messages at /xsys/sbin//spamd line 1103.

Fri Sep 7 16:30:35 2007 [26914] info: spamd: server successfully spawned child process, pid 26312
Fri Sep 7 16:30:37 2007 [24685] warn: spamd: copy_config timeout, respawning child process after 1 messages at /xsys/sbin//spamd line 1103. Fri Sep 7 16:30:39 2007 [26914] warn: prefork: cannot ping 24702, file handle not defined, child likely to still be processing SIGCHLD handler after killing itself
Fri Sep 7 16:30:39 2007 [26914] warn: prefork: killing failed child 24702 fd=undefined at
/xsys/lib/perl5/site_perl/5.8.8/Mail/SpamAssassin/SpamdForkScaling.pm line 171.
Fri Sep 7 16:30:39 2007 [26914] warn: prefork: killed child 24702 Fri Sep 7 16:30:41 2007 [26914] info: spamd: handled cleanup of child pid 24702 due to SIGCHLD
Fri Sep 7 16:30:41 2007 [26914] warn: prefork: cannot ping 24687, file handle not defined, child likely to still be processing SIGCHLD handler after killing itself
Fri Sep 7 16:30:41 2007 [26914] warn: prefork: killing failed child 24687 fd=undefined at
/xsys/lib/perl5/site_perl/5.8.8/Mail/SpamAssassin/SpamdForkScaling.pm line 171.
Fri Sep 7 16:30:41 2007 [26914] warn: prefork: killed child 24687

Looking at the swap usage, I was thinking I would be better if I reduced the number of children processes and let thing queue up. I know I will also have to look at exim and it's ratelimit command. Any other idea's on handling spam surges/DoS?

Thanks
Paul Received on Mon Sep 10 12:49:03 2007