hello again.. i already posted a while ago about the traffic spikes i am getting on those high port numbers.. we suspected that it was ftp traffic but i stopped the ftp server for a few days now and am still getting the same traffic spikes of multiple gigabytes.. i am slightly paranoid because i really dont know what could possibly cause all that traffic (again, it's not normal email/web traffic)..
i remember you guys talking about checking for suspicious processes but i dont know much about linux so i wouldnt know whether something is legitimate or not.. could someone maybe have a look at this list and tell me whether anything looks unusual?
[code:1]29245 www-data 277228 kB java -Dlog4j.configuration=file:conf/log4j-default.properties -Xms3m -Xmx128m -D ... 2560 mysql 147832 kB /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file ... 11590 www-data 138800 kB /usr/sbin/apache2 -k start 26546 www-data 138800 kB /usr/sbin/apache2 -k start 21878 www-data 138756 kB /usr/sbin/apache2 -k start 21879 www-data 138756 kB /usr/sbin/apache2 -k start 12399 www-data 138756 kB /usr/sbin/apache2 -k start 13617 www-data 138744 kB /usr/sbin/apache2 -k start 6549 www-data 138640 kB /usr/sbin/apache2 -k start 5213 www-data 137200 kB /usr/sbin/apache2 -k start 6683 www-data 136944 kB /usr/sbin/apache2 -k start 6685 www-data 136944 kB /usr/sbin/apache2 -k start 799 root 136152 kB /usr/sbin/apache2 -k start 2689 clamav 130776 kB /usr/sbin/clamd 11587 www-data 96036 kB /usr/sbin/apache2 -k start 11575 root 66192 kB /usr/share/webmin/virtual-server/lookup-domain-daemon.pl 2469 bind 64864 kB /usr/sbin/named -u bind 7937 root 46608 kB /usr/share/webmin/proc/index_size.cgi 912 root 40728 kB python /usr/sbin/denyhosts --daemon --config=/etc/denyhosts.conf --config=/etc/d ... 2975 root 40072 kB dovecot-auth 3017 root 38064 kB /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf 3013 root 34472 kB /usr/bin/perl /usr/share/usermin/miniserv.pl /etc/usermin/miniserv.conf 2884 root 32544 kB /usr/sbin/saslauthd -a pam -m /var/spool/postfix/var/run/saslauthd -r -n 5 2885 root 32544 kB /usr/sbin/saslauthd -a pam -m /var/spool/postfix/var/run/saslauthd -r -n 5 2886 root 32544 kB /usr/sbin/saslauthd -a pam -m /var/spool/postfix/var/run/saslauthd -r -n 5 2888 root 32544 kB /usr/sbin/saslauthd -a pam -m /var/spool/postfix/var/run/saslauthd -r -n 5 2889 root 32544 kB /usr/sbin/saslauthd -a pam -m /var/spool/postfix/var/run/saslauthd -r -n 5 2773 clamav 28456 kB /usr/bin/freshclam -d --quiet 2894 root 25844 kB /usr/sbin/sshd 3012 postfix 21832 kB tlsmgr -l -t unix -u -c 2872 postfix 20692 kB qmgr -l -t fifo -u 7045 postfix 20652 kB pickup -l -t fifo -u -c 2865 root 19616 kB /usr/lib/postfix/master 2963 root 11496 kB /usr/sbin/cron 1439 root 10592 kB udevd --daemon 2523 root 10112 kB /bin/sh /usr/bin/mysqld_safe 602 dovecot 8920 kB imap-login 28524 dovecot 8916 kB imap-login 9433 dovecot 8916 kB imap-login 31854 dovecot 8772 kB imap-login 6431 dovecot 8772 kB imap-login 7219 dovecot 8768 kB imap-login 1459 dovecot 8764 kB pop3-login 2838 dovecot 8764 kB pop3-login 3072 dovecot 8760 kB pop3-login 2935 root 7216 kB /usr/sbin/dovecot 1 root 6124 kB init [2] 7070 root 3728 kB /sbin/syslogd 3034 root 2656 kB /sbin/getty 38400 tty1 3040 root 2656 kB /sbin/getty 38400 tty5 3041 root 2656 kB /sbin/getty 38400 tty6 7075 root 2656 kB /sbin/klogd -x 2675 root 2652 kB /usr/sbin/acpid -c /etc/acpi/events -s /var/run/acpid.socket 3036 root 2652 kB /sbin/getty 38400 tty2 3037 root 2652 kB /sbin/getty 38400 tty3 3038 root 2652 kB /sbin/getty 38400 tty4 2561 root 2636 kB logger -p daemon.err -t mysqld_safe -i -t mysqld 1801 www-data 1636 kB /usr/sbin/apache2 -k start 2 root 0 kB [migration/0] 3 root 0 kB [ksoftirqd/0] 4 root 0 kB [watchdog/0] 5 root 0 kB [migration/1] 6 root 0 kB [ksoftirqd/1] 7 root 0 kB [watchdog/1] 8 root 0 kB [events/0] 9 root 0 kB [events/1] 10 root 0 kB [khelper] 11 root 0 kB [kthread] 16 root 0 kB [kblockd/0] 17 root 0 kB [kblockd/1] 18 root 0 kB [kacpid] 122 root 0 kB [khubd] 124 root 0 kB [kseriod] 176 root 0 kB [pdflush] 177 root 0 kB [pdflush] 178 root 0 kB [kswapd0] 179 root 0 kB [aio/0] 180 root 0 kB [aio/1] 423 root 0 kB [xfslogd/0] 424 root 0 kB [xfslogd/1] 425 root 0 kB [xfsdatad/0] 426 root 0 kB [xfsdatad/1] 464 root 0 kB [ata/0] 465 root 0 kB [ata/1] 466 root 0 kB [ata_aux] 496 root 0 kB [scsi_eh_0] 497 root 0 kB [scsi_eh_1] 498 root 0 kB [scsi_eh_2] 499 root 0 kB [scsi_eh_3] 1198 root 0 kB [kmirrord] 1253 root 0 kB [kjournald] 1761 root 0 kB [kpsmoused] 2786 root 0 kB [kondemand/0] 2787 root 0 kB [kondemand/1][/code:1]
here's a screenshot of the traffic so you can visualize what i mean.. thanks for any advice/help.. i really appreciate it! i'd hate having to reinstall the system from scratch (and then possibly run into the same problems again)
<a href='http://img40.imageshack.us/my.php?image=trafficn.jpg' target='_blank'><img src='http://img40.imageshack.us/img40/7937/trafficn.th.jpg' style='border:0px'></a>
did you check for any large files on your server?
Per haps someone made big files available for downloads.
ah brilliant.. so simple.. there are loads of illegal dvds/movies on there.. all of them in the wordpress directory.. i just can't work out whether that's just a cover up and they gained proper access or whether they are using some security flaw in wordpress.. ill have a deeper look into it..
files are written by www-data so that implies a problem with wordpress.. relieved to see..
<div class='quote'>ah brilliant.. so simple.. there are loads of illegal dvds/movies on there</div>
I thought something like this had happened, it shows in the image you posted that big files were uploaded.
Was this done by a user or an outsider? (shows again chmod 777 is a bad idea)
<div class='quote'>Was this done by a user or an outsider?</div>
Definitely an outsider, I am the only person with shell access for the machine and ftp access for that virtual server.. they actually tried to sneak in again as i can tell from the apache access log files this morning..
None of those processes looks suspicious to me.
The transfers are big enough to rule out things like system updates (which could be in the hundreds of megabytes, but never in the gigabytes). Users could start processes periodically and open up high ports, but I can't think of any reason for someone with malicious intent to not run it all the time.
How many users do you have on your system? You can use lastlog and w to see who's logged in recently or logged in currently, and then check their crontabs and home directories for anything suspicious.
I dunno. You've got me stumped. It's usually pretty obvious when something nasty is happening. This looks more like innocent weirdness. ;-)
Oh, you might check to be sure ps is reporting honestly. If your system has been rooted, you can no longer trust ps (or top or ls or anything else, for that matter). First step:
rpm -V procps
And then try chkrootkit. Even that can't be trusted entirely, unless you boot from a live CD and run it from there. (root kits modify system binaries in order to hide malicious activity and files, making it harder for you to find what's actually going on.)
I don't want to alarm you, as this really does look like something innocent rather than malicious. But, it's better to be sure about that. Anytime things act mysterious, assuming the worst isn't a bad idea.
--
Check out the forum guidelines!
hi joe, thanks for the hints - im still trying to rule out anything malicious just to be safe.. those commands dont work for me.. do i need to install any special packages or is it because im on debian (etch)?
alright.. i found the problem.. it was this process:
[code:1]29245 www-data 277228 kB java -Dlog4j.configuration=file:conf/log4j-default.properties -Xms3m -Xmx128m -D[/code:1]
in case anyone has similar problems this is what i think happened:
i installed a theme in wordpress and that directory had 777 permission (i know)
someone must have then uploaded slave.zip deep into the directory structure (public_html/wp-content/themes/sometheme/img/includes/image200x300/upload090254/.ignore/)
this file contains a full java development kit and some dodgy java code hidden in a separate library and launched by that open source logging tool Dlog4j.. this is also why no traffic shows up in the apache log files as its running it's own server.. and the processes seemed fine to me because i didnt know whether virtualmin might have used that logging tool and i guess it seemed fine to joe because he probably thought im using it..
funny thing is it kept full log of its doings in a log file which is now 22MB after about two weeks of 'infection'..
anyway.. glad i figured it out now.. if anyone's interested im happy to post the slave.zip file if it helps protectng against the exploit..
thanks for all your help..
<div class='quote'>and the processes seemed fine to me because i didnt know whether virtualmin might have used that logging tool and i guess it seemed fine to joe because he probably thought im using it..</div>
Yep. Just goes to show that one should <i>always</i> be suspicious of Java. ;-)
(Jamie actually likes Java OK, and there are some Java applets in Webmin, but those don't run on the server, and you'll likely never see any new Java software coming from us. The applets will be replaced with JavaScript equivalents in the coming months.)
--
Check out the forum guidelines!