SOLVED - please help, server hacked (postfix, mailq is filling up FAST)

Howdy,

Now that you've disabled the site, are you continuing to get more spam in the mail queue?

If so, that may mean you're getting spam via a source other than just the site... that may mean one of the email accounts was broken into.

However, you should be able to determine that from the headers of the spam messages what the actual source is.

Once you've reviewed the headers, you may want to clear out the Postfix queue.

-Eric

If you found out which account/site is/was being used to send the spam (authentication headers and mail log entries should provide that info), it should suffice to lock down the respective account and then clear out the mail queue.

I wouldn't really know what "locking down with the firewall" is supposed to mean or help...

If the login was compromised, just change the user's password, thus disabling them from abusing your server, while keeping website and stuff active.

Firewalls don't help against compromised email accounts or websites. Still it doesn't hurt of course to only have those ports open you really need. I do the same with an external (virtual) firewall on my systems.

Using a software like OSSEC is also advisable. An alternative (which I myself use) is CSF/LFD (ConfigServer Security & Firewall / Login Failure Daemon), which integrates nicely with Webmin.

Like TSC said, brute force dictionary attacks are very common. That's what software like OSSEC or LFD are for: Detecting these and (temp/perm) blocking the attacking IP.

These repeating log lines look interesting:

Dec 24 08:27:26 sdc postfix/pickup[10970]: 1F6EB84032: uid=1014 from=<ourshore.ca>
Dec 24 08:27:26 sdc postfix/cleanup[16573]: 1F6EB84032: message-id=<20131224122726.1F6EB84032@sdc.starlingdesign.ca>
Dec 24 08:27:26 sdc postfix/qmgr[1789]: 1F6EB84032: from=<ourshore.ca@sdc.starlingdesign.ca>, size=3463, nrcpt=1 (queue active)

They suggest that the spam is not delivered to your system via SMTP, but probably from a local website. The user ID 1014 is being used to put it in the Postfix queue. You might want to check in /etc/passwd which user that is, and check if their website has been compromised.

Changing the password for that user won't have an effect if there's malicious code in their website. It's probable that a hacker found a security hole in the web software and installed stuff there. Access to that is independent from the administrative user. You need to disable the virtual server, thus making the web files inaccessible. Also check that user's Apache logfiles for suspicious activity, like downloads or uploads of unknown files. Do the same with ProFTPD logs.

A changed passwd file is normal if a user (here: OSSEC) was added. It's to be expected, since you installed OSSEC.

You might want to run the software "Linux Malware Detect" on the potentially compromised web directory; best is if you run it on the entire /home. Also check out what kind of web software the "ourshore.ca" is using; make sure it's up to date and is not using a version with known security issues.

Oh, also check for active processes that are running under the user ID 1014, with ps aux | grep 1014. Maybe they started a process that's sending the mail or is doing other stuff.

Good, so there are no processes running as ID 1014.

You should remove fail2ban, yes, it'll probably lead to problems if you have multiple software installed that does login failure blocking.

If you have "ourshore.ca" disabled, you should be seeing no more spam mails being added to Postfix.

As I said, you might want to use "Linux Malware Detect" to scan your /home folder.

If OSSEC operates anything like LFD, it writes a logfile of its own where it records its actions. IP blocks are performed via iptables, and it also has a block management of its own. (Then again, CSF/LFD has a GUI that can be integrated in Webmin, don't know if OSSEC has the same.)

i turned my emails on my phone tonight and it came through 600 undelivered mail returned,

logged into the virtualmin tonight and its still going (9,500 in under 1 hour) i have tried disabling the server, changing the email password. The account in question is info@espressowebdesign.co.uk

email headers as follows:

Return-Path:<> X-Spam-Checker-Version:SpamAssassin 3.3.1 (2010-03-16) on ewd01.espressowebdesign.net X-Spam-Level:* X-Spam-Status:No, score=1.7 required=5.0 tests=NO_RELAYS,URIBL_BLACK autolearn=no version=3.3.1 X-Original-To:info@espressowebdesign.co.uk Delivered-To:info-espressowebdesign.co.uk@espressowebdesign.net Received:by ewd01.espressowebdesign.net (Postfix) id 46BCACC80A; Fri, 20 Jun 2014 19:28:49 +0100 (BST) Date:Fri, 20 Jun 2014 19:28:49 +0100 (BST) From:MAILER-DAEMON@espressowebdesign.net (Mail Delivery System) Subject:Undelivered Mail Returned to Sender To:info@espressowebdesign.co.uk Auto-Submitted:auto-replied MIME-Version:1.0 Content-Type:multipart/report; report-type=delivery-status; boundary="7B025CC7F5.1403288929/ewd01.espressowebdesign.net" Message-Id:20140620182849.46BCACC80A@ewd01.espressowebdesign.net Message contents

This is the mail system at host ewd01.espressowebdesign.net.

I'm sorry to have to inform you that your message could not be delivered to one or more recipients. It's attached below.

For further assistance, please send mail to postmaster.

If you do so, please include this problem report. You can delete your own text from the attached returned message.

               The mail system

hubert.pattermann@tele2.at: host mailgw.swip.net[212.247.156.1] said: 550 hubert.pattermann@tele2.at unknown user account (in reply to RCPT TO command)

Failed delivery status Final recipienthubert.pattermann@tele2.at Reason for failure550 hubert.pattermann@tele2.at unknown user account Remote mail servermailgw.swip.net Reporting mail serverewd01.espressowebdesign.net

problem is guys the person who controls, runs support and helps me with this is away on holiday, i dont have a clue how this works. can anyone help?

desperate