Web/Mail Server Compromised

Submitted by methownet on Fri, 05/11/2012 - 01:29 Pro Licensee

We have had two of our production servers compromised. The first one was a well patched 8.08 Ubuntu box on which apache started locking up and eventually the server was blacklisted for sending out too much spam. This server has run well for years. We have not been able to determine from the log files or the firewall just where this is coming from. All of our efforts to block the email failed and so we moved the 120 or so domains to another working server today running Ubuntu Linux 10.04.3 all patched. It started to send out spam after we moved the domains over. We've been watching the processes through top and noticed occasionally 30 or more lines of just smtp with some other lines labeled bounce. We are wondering if you can give us some guidance as to where to look next to clean up the situation. We did turn off six small database sites and that seemed to help the load. We are now turning them back on individually to see how they affect the processes listed in top. Let us know what we can do to get this fixed. Thanks, Jeff

Status: 
Active

Comments

Joe's picture
Submitted by Joe on Fri, 05/11/2012 - 01:52 Pro Licensee

First up, check your maillog (or mail.log) to see which users are sending lots of mail. That'll help narrow things down some.

At first glance, I'd guess one of your sites is running an application that is compromised.

Confirm all of the apps running on those sites are up to date, and that the up to date versions are actually secure. It could also be a configuration mistake in on or more applications, or perhaps some custom code; there are lots of old form mail type apps that are notoriously bad for security, for instance...I'd check for those.

It's also possible that one or more user accounts have been compromised. If you have non-technical users they may have chosen very weak passwords, which are easy to attack with brute force tools.

Submitted by methownet on Fri, 05/11/2012 - 03:58 Pro Licensee

Joe, The log files have not been showing where the mail is coming from. Is there a way we can block this from the server level? (No mail server will talk to us any more).

May 11 01:52:45 cobra postfix/smtp[20549]: connect to a34-mta03.direcpc.com[66.82.4.104]:25: Connection timed out May 11 01:52:45 cobra postfix/smtp[20552]: connect to a34-mta04.direcpc.com[66.82.4.105]:25: Connection timed out May 11 01:52:46 cobra postfix/smtp[20602]: connect to mercury.armandcorp.com[74.92.71.121]:25: Connection timed out May 11 01:52:46 cobra postfix/smtp[20590]: connect to mbusa.net[74.117.114.119]:25: Connection timed out May 11 01:52:46 cobra postfix/smtp[20590]: 6158F385EE6: to=attrailsend@mbusa.net, relay=none, delay=46674, delays=46652/0.3/21/0, dsn=4.4.1, status=deferred (connect to mbusa.net[74.117.114.119]:25: Connection timed out) May 11 01:52:46 cobra postfix/smtp[20602]: 5E46D38626A: to=barmand@armandcorp.com, relay=none, delay=46626, delays=46604/0.29/21/0, dsn=4.4.1, status=deferred (connect to mercury.armandcorp.com[74.92.71.121]:25: Connection timed out) May 11 01:52:46 cobra postfix/smtp[20572]: connect to glafosse.com[205.178.189.131]:25: Connection timed out May 11 01:52:46 cobra postfix/smtp[20603]: connect to alltel.net[198.133.103.44]:25: Connection timed out May 11 01:52:46 cobra postfix/smtp[20603]: 5ACA23863E2: to=berhousing@alltel.net, relay=none, delay=46569, delays=46547/0.29/21/0, dsn=4.4.1, status=deferred (connect to alltel.net[198.133.103.44]:25: Connection timed out) May 11 01:52:46 cobra postfix/smtp[20572]: 4A6C138A2E3: to=gerard@glafosse.com, relay=none, delay=36323, delays=36301/0.25/21/0, dsn=4.4.1, status=deferred (connect to glafosse.com[205.178.189.131]:25: Connection timed out) May 11 01:52:46 cobra postfix/smtp[20594]: connect to mail1.sind.com[68.153.47.165]:25: Connection timed out May 11 01:52:46 cobra postfix/smtp[20594]: 3101638A283: to=genny_natter@sind.com, relay=none, delay=36325, delays=36303/0.29/21/0, dsn=4.4.1, status=deferred (connect to mail1.sind.com[68.153.47.165]:25: Connection timed out) May 11 01:52:46 cobra postfix/smtp[20583]: connect to sbi.com[199.67.196.105]:25: Connection timed out May 11 01:52:46 cobra postfix/smtp[20596]: connect to thephillipsgroup.com[205.178.189.131]:25: Connection timed out May 11 01:52:46 cobra postfix/smtp[20546]: connect to alltel.net[198.133.103.44]:25: Connection timed out May 11 01:52:46 cobra postfix/smtp[20546]: DC0593861B5: to=awcinc4@alltel.net, relay=none, delay=46668, delays=46647/0.33/21/0, dsn=4.4.1, status=deferred (connect to alltel.net[198.133.103.44]:25: Connection timed out) May 11 01:52:46 cobra postfix/smtp[20596]: 3C7E338A346: to=ghom@thephillipsgroup.com, relay=none, delay=36316, delays=36294/0.3/21/0, dsn=4.4.1, status=deferred (connect to thephillipsgroup.com[205.178.189.131]:25: Connection timed out) May 11 01:52:46 cobra postfix/smtp[20583]: E96BE38604D: to=aparks@sbi.com, relay=none, delay=46719, delays=46698/0.31/21/0, dsn=4.4.1, status=deferred (connect to sbi.com[199.67.196.105]:25: Connection timed out) May 11 01:52:46 cobra postfix/smtp[20559]: connect to mx.fakemx.net[46.4.27.149]:25: Connection timed out May 11 01:52:46 cobra postfix/smtp[20559]: 994F338606B: to=aroot@kutamba.com, relay=none, delay=46692, delays=46671/0.22/21/0, dsn=4.4.1, status=deferred (connect to mx.fakemx.net[46.4.27.149]:25: Connection timed out) May 11 01:52:46 cobra postfix/smtp[20618]: connect to countrywide.com[171.159.100.190]:25: Connection timed out May 11 01:52:46 cobra postfix/smtp[20618]: C235638A514: to=grandy_lester@countrywide.com, relay=none, delay=36283, delays=36262/0.33/21/0, dsn=4.4.1, status=deferred (connect to countrywide.com[171.159.100.190]:25: Connection timed out) May 11 01:52:46 cobra postfix/smtp[20558]: connect to mx.fakemx.net[46.4.27.149]:25: Connection timed out May 11 01:52:46 cobra postfix/smtp[20558]: 9C5EE3860B9: to=asdfsadf@sdfgs.com, relay=none, delay=46688, delays=46667/0.23/21/0, dsn=4.4.1, status=deferred (connect to mx.fakemx.net[46.4.27.149]:25: Connection timed out) May 11 01:52:46 cobra postfix/smtp[20598]: connect to aopa.net[65.125.13.158]:25: Connection

timed out May 11 01:52:46 cobra postfix/smtp[20598]: 3DB3F38D05A: to=n1801e@aopa.net, relay=none, delay=35673, delays=35651/0.28/21/0, dsn=4.4.1, status=deferred (connect to aopa.net[65.125.13.158]:25: Connection timed out) May 11 01:52:46 cobra postfix/smtp[20544]: connect to acushnet.com[107.0.159.230]:25: Connection timed out May 11 01:52:46 cobra postfix/smtp[20601]: connect to breez.net[82.98.86.162]:25: Connection timed out May 11 01:52:46 cobra postfix/smtp[20601]: 55A2D385E79: to=arden@breez.net, relay=none, delay=46696, delays=46674/0.29/21/0, dsn=4.4.1, status=deferred (connect to breez.net[82.98.86.162]:25: Connection timed out) May 11 01:52:46 cobra postfix/smtp[20547]: 76AE33863D5: to=berryhill@kw.com, relay=mx04.kw.com[66.179.173.93]:25, delay=46568, delays=46546/0.15/22/0.06, dsn=4.1.8, status=deferred (host mx04.kw.com[66.179.173.93] said: 450 4.1.8 www-data@cobra.methowdata.net: Sender address rejected: Domain not found (in reply to RCPT TO command))

Submitted by Locutus on Fri, 05/11/2012 - 04:54

In case of compromised web software, I can recommend the tool "LMD" (Linux Malware Detect) which will use the already installed ClamAV scanning engine to check all websites on your server specifically for known malware code. This might help identify, if there is one, the compromised site.

http://www.rfxn.com/projects/linux-malware-detect/

In addition to that, you should of course disable any user email account that is used to send out the spam, if such exists.

Submitted by methownet on Fri, 05/11/2012 - 05:06 Pro Licensee

great. I will install the program and try this out. we shut off Dovecot and the log file quieted down a bit. wondering if squirrelmail might be compromised. suggestions for that?

also, I am getting this message: A problem was detected with your Virtualmin license : Your serial number is licensed for only 1 servers, but is being used on 3. A renewal can be purchased at http://www.virtualmin.com/shop

Will I be OK for the moment? As far as I know, I only have this license on this one server now.

Submitted by methownet on Fri, 05/11/2012 - 06:39 Pro Licensee

The scanner pointed us to a couple of domains and we disabled them. We still are seeing some activity though. Any other ways to get at which domain is doing this. It is like clockwork, every 5 minutes there are a long list of smtp hits in the processes list show in Top. How can I find the offending website?

Submitted by Locutus on Fri, 05/11/2012 - 07:09

"Dovecot" is the IMAP/POP3 service, not the one that sends out email. To quiet down sending of emails, you'll want to shut off "Postfix".

To find out what is sending email, you can check /var/log/mail.log. If an outgoing batch is in progress, you can use postqueue -p to view the queue. Otherwise can you please specify what kind of activity you're seeing exactly? Outgoing mail? Incoming? Local user sending mail?

The thing with the license being in use 3 times can happen if you changed your IP address or similar. The license server records all IP addresses it sees using the license, for a few days. If you switched servers/IP addresses, the license notice will go away automatically after some time.

Submitted by methownet on Fri, 05/11/2012 - 09:53 Pro Licensee

It is outgoing mail that goes out in bunches every 5 minutes. The account it seems to be sending from is www-data@cobra.methowdata.net:

BE052387707 1930 Thu May 10 13:06:28 www-data@cobra.methowdata.net (host mx-rtr01.ntelos.net[216.12.0.119] said: 450 4.1.8 www-data@cobra.methowdata.net: Sender address rejected: Domain not found (in reply to MAIL FROM command)) glarge@naxs.com

tons of these.

We have 150 domains on this server.

Submitted by andreychek on Fri, 05/11/2012 - 09:57

You may want to go into Webmin -> Servers -> Postfix, and look at the Mail Queue.

From there, you can review the email headers of any of the spam messages, which should assist in revealing what exactly is at fault.

In some cases, the mail headers may contain the URL where the spam was generated (depending on how it was created).

At the very least though, you should be able to see the userid in the mail headers, which would point you to the culprit.

You could then use that information to track down the specific cause, or at least disable the account so that it can no longer send out spam.

Joe's picture
Submitted by Joe on Fri, 05/11/2012 - 11:10 Pro Licensee

www-data means it is coming from an application running as the Apache user. If you normally use suexec, this narrows it down to whichever domains are not using suexec. But, if you've disabled suexec and all domains are running as the Apache user (not recommended in a shared hosting system, by the way), it will be harder to narrow it down.

It could be Squirrelmail; possibly a compromised user account. There are spam tools out there that can send via Squirrelmail (assuming they have the credentials to login, or you're running an old exploitable version).

Submitted by methownet on Fri, 05/11/2012 - 12:15 Pro Licensee

I will check out these issues. We cleared out the mail queue and things calmed down immediately. We had 10000 items in the queue from days ago blocked by aol. Apparently, if you have that many in the queue, webmin shows 0 in the queue, which threw us off. I think in hindsight that your recommended scanner did the trick to get the files off of there, and that we were just dealing with a plugged queue from there. I'm crossing my fingers.