Logged

I have been the subject of a pretty persistent brute force attack, where the attacker is attempting to ssh in with thousands of different host names and presumably weak passwords.  Anyone who has run a server for a while has been the subject of such attacks.  Typically, you can see the attack starting with names that start with A and work down to Z.  I do not recall, though, a time where the attack was coming from several (well, hundreds actually) hosts.  The ones I have seen in the past were all from one IP address.  Some times, I would see several attacks running simultaneously from different hosts, but it was clear they were not related.

Then I noticed this:

Apr 17 00:00:51 www2 sshd[1649]: error: PAM: authentication error for illegal user bradley from server.powered-by-core64.net
Apr 17 00:00:51 www2 sshd[1650]: error: PAM: authentication error for illegal user bradley from server.powered-by-core64.net
Apr 17 00:02:29 www2 sshd[2013]: error: PAM: authentication error for illegal user bradley from 208.53.131.90
Apr 17 00:02:30 www2 sshd[2012]: error: PAM: authentication error for illegal user bradley from 208.53.131.90
Apr 17 00:04:14 www2 sshd[2028]: error: PAM: authentication error for illegal user bradley from 66.63.178.247
Apr 17 00:04:14 www2 sshd[2029]: error: PAM: authentication error for illegal user bradley from 66.63.178.247
Apr 17 00:05:59 www2 sshd[2081]: error: PAM: authentication error for illegal user bradley from mx1.dpstudio.com.br
Apr 17 00:05:59 www2 sshd[2082]: error: PAM: authentication error for illegal user bradley from mx1.dpstudio.com.br
Apr 17 00:08:07 www2 sshd[2106]: error: PAM: authentication error for illegal user bradshaw from correo.texsa.cl
Apr 17 00:08:07 www2 sshd[2107]: error: PAM: authentication error for illegal user bradshaw from correo.texsa.cl
Apr 17 00:09:23 www2 sshd[2219]: error: PAM: authentication error for illegal user bradshaw from 77.243.236.32
Apr 17 00:09:23 www2 sshd[2220]: error: PAM: authentication error for illegal user bradshaw from 77.243.236.32
Apr 17 00:10:55 www2 sshd[2288]: error: PAM: authentication error for illegal user bradshaw from 85.17.184.37
Apr 17 00:10:55 www2 sshd[2287]: error: PAM: authentication error for illegal user bradshaw from 85.17.184.37
Apr 17 00:12:51 www2 sshd[2447]: error: PAM: authentication error for illegal user bradshaw from 200.69.217.177
Apr 17 00:12:52 www2 sshd[2448]: error: PAM: authentication error for illegal user bradshaw from 200.69.217.177
Apr 17 00:15:58 www2 sshd[2484]: error: PAM: authentication error for illegal user brady from 208.53.131.90
Apr 17 00:15:58 www2 sshd[2485]: error: PAM: authentication error for illegal user brady from 208.53.131.90
Apr 17 00:18:03 www2 sshd[2572]: error: PAM: authentication error for illegal user brady from host54-201-static.183-80-b.business.telecomitalia.it
Apr 17 00:18:03 www2 sshd[2573]: error: PAM: authentication error for illegal user brady from host54-201-static.183-80-b.business.telecomitalia.it
Apr 17 00:21:08 www2 sshd[2651]: error: PAM: authentication error for illegal user brady from 211.51.7.220
Apr 17 00:21:08 www2 sshd[2652]: error: PAM: authentication error for illegal user brady from 211.51.7.220
Apr 17 00:22:43 www2 sshd[2700]: error: PAM: authentication error for illegal user brady from auto-schunn.ro
Apr 17 00:22:43 www2 sshd[2699]: error: PAM: authentication error for illegal user brady from auto-schunn.ro
Apr 17 00:24:31 www2 sshd[2726]: error: PAM: authentication error for illegal user braeden from 190.24.226.18
Apr 17 00:24:31 www2 sshd[2727]: error: PAM: authentication error for illegal user braeden from 190.24.226.18
Apr 17 00:26:19 www2 sshd[2829]: error: PAM: authentication error for illegal user braeden from lvps92-51-129-83.dedicated.hosteurope.de
Apr 17 00:26:20 www2 sshd[2828]: error: PAM: authentication error for illegal user braeden from lvps92-51-129-83.dedicated.hosteurope.de
Apr 17 00:27:49 www2 sshd[2841]: error: PAM: authentication error for illegal user braeden from 91.189.181.242
Apr 17 00:27:49 www2 sshd[2840]: error: PAM: authentication error for illegal user braeden from 91.189.181.242
Apr 17 00:31:12 www2 sshd[3008]: error: PAM: authentication error for illegal user braeden from 218.241.164.34
Apr 17 00:31:12 www2 sshd[3007]: error: PAM: authentication error for illegal user braeden from 218.241.164.34
Apr 17 00:33:00 www2 sshd[3061]: error: PAM: authentication error for illegal user braima from ip-67-205-112-200.static.privatedns.com
Apr 17 00:33:01 www2 sshd[3062]: error: PAM: authentication error for illegal user braima from ip-67-205-112-200.static.privatedns.com
Apr 17 00:36:19 www2 sshd[3235]: error: PAM: authentication error for illegal user braima from 218.241.164.34
Apr 17 00:36:20 www2 sshd[3236]: error: PAM: authentication error for illegal user braima from 218.241.164.34
Apr 17 00:39:40 www2 sshd[3406]: error: PAM: authentication error for illegal user braima from 85.17.184.37
Apr 17 00:39:40 www2 sshd[3407]: error: PAM: authentication error for illegal user braima from 85.17.184.37
Apr 17 00:41:32 www2 sshd[3479]: error: PAM: authentication error for illegal user braith from 62.212.74.149
Apr 17 00:41:32 www2 sshd[3480]: error: PAM: authentication error for illegal user braith from 62.212.74.149
Apr 17 00:45:00 www2 sshd[3764]: error: PAM: authentication error for illegal user braith from 85.17.36.42
Apr 17 00:45:00 www2 sshd[3765]: error: PAM: authentication error for illegal user braith from 85.17.36.42
Apr 17 00:46:50 www2 sshd[3812]: error: PAM: authentication error for illegal user braith from london.routed-networks.net
Apr 17 00:46:50 www2 sshd[3813]: error: PAM: authentication error for illegal user braith from london.routed-networks.net
Apr 17 00:48:24 www2 sshd[3836]: error: PAM: authentication error for illegal user braith from server.powered-by-core64.net
Apr 17 00:48:24 www2 sshd[3837]: error: PAM: authentication error for illegal user braith from server.powered-by-core64.net
Apr 17 00:50:44 www2 sshd[3892]: error: PAM: authentication error for illegal user bran from venus2.ccac.mg
Apr 17 00:50:48 www2 sshd[3891]: error: PAM: authentication error for illegal user bran from venus2.ccac.mg
Apr 17 00:51:46 www2 sshd[3924]: error: PAM: authentication error for illegal user bran from server.powered-by-core64.net
Apr 17 00:51:46 www2 sshd[3923]: error: PAM: authentication error for illegal user bran from server.powered-by-core64.net
Apr 17 00:53:31 www2 sshd[3962]: error: PAM: authentication error for illegal user bran from 85.17.138.147
Apr 17 00:53:31 www2 sshd[3961]: error: PAM: authentication error for illegal user bran from 85.17.138.147
Apr 17 00:55:12 www2 sshd[4015]: error: PAM: authentication error for illegal user bran from 211.51.7.220
Apr 17 00:55:12 www2 sshd[4016]: error: PAM: authentication error for illegal user bran from 211.51.7.220
Apr 17 00:56:51 www2 sshd[4041]: error: PAM: authentication error for illegal user bran from 91.186.21.193
Apr 17 00:56:51 www2 sshd[4042]: error: PAM: authentication error for illegal user bran from 91.186.21.193
Apr 17 01:00:15 www2 sshd[4262]: error: PAM: authentication error for illegal user brand from 189.52.31.178
Apr 17 01:00:16 www2 sshd[4263]: error: PAM: authentication error for illegal user brand from 189.52.31.178
Apr 17 01:03:41 www2 sshd[5086]: error: PAM: authentication error for illegal user brand from 202.103.191.32
Apr 17 01:03:41 www2 sshd[5085]: error: PAM: authentication error for illegal user brand from 202.103.191.32
Apr 17 01:05:26 www2 sshd[5125]: error: PAM: authentication error for illegal user brand from 211.51.7.220
Apr 17 01:05:26 www2 sshd[5126]: error: PAM: authentication error for illegal user brand from 211.51.7.220
Apr 17 01:07:11 www2 sshd[5152]: error: PAM: authentication error for illegal user brandee from 222.73.205.70

That was roughly an hour’s worth of logs from this morning.  So far today, I count 125 uniq hosts participating in the attack using this magic train of piped unix commands

grep “illegal user” /var/log/messages | cut -f 15 -d ” ” | sort | uniq | wc

Note the tight grouping of user names attempted, each one only attempted once or twice by each host, with up to 5 other hosts (in my casual observation) attempting the same user name before the bot herd moves on to the next name.  I am quite confident that it’s the result of a botnet because of the apparent coordinated nature of the attempts.

Interesting Observations I’ve Made

• Each attempt for a host to try a given name is only attempted twice, immediately, generally showing up in the logs as having occurred in the same second.
• There is very consistent 2 minute delay between the attempts by different hosts
• There does not appear to be a pattern for the order of hosts attempting to log in

This attack has actually been going on for several weeks.  Without looking at the logs, it is not noticable.  The method used qualifies as a “low and slow” attack, meaning that they’re not likely to set off alarms or trip firewall rules to block the hosts because of the failed attempts.

Short of disabling ssh, which is a not really an option for a server providing hosting services, there doesn’t appear to be much I can do to stop this.  The brtue force attacks don’t pose a threat so long as the passwords on this system are sufficiently complex – one could assume that the bots are trying “abc123″ or “password” or some other simple thing like that.

This is a great situation where a log analyzer/alerting tool could come in very handy – specifically to notify an administrator when someone logs in to a system. The trick would be separating out the legitimate users logging into their accounts from a bot that got lucky with a username and password guess.