These forums are locked and archived, but all topics have been migrated to the new forum. You can search for this topic on the new forum: Search for Securing email server on the new forum.
Web Hosting and Cloud Computing Control Panels
Virtualmin configures the mail server out-of-the-box so that authentication is required to send outgoing mail, if you used the installer script. To make sure about that, you can google for "email server open relay test websites", and have them test your server.
When your server is sending out spam anyway, it is usually due to a hacked user email account (user's password was retrieved by malware on their PC or similar) or website.
The entries in syslog should indicate what user account is being used to authenticate and send out the mails. You should disable that account/site, change their password and inform the owner.
Thanks Locutus, Can you point me to a quick reference on deciphering the syslog entries? I can see the return address of the spam email, but the user in the return address does not exist and I don't see any other user info. From my knowledge of spam, not using the correct return email address seems par for the coerce but how do I follow it back to the offending account?
I did find, after a quick look at the user accounts, that the nobody user was set to 'No password required'. I deleted the account and that seemed to end the spam. After, checking a known clean virtualmin install, I found that the nobody account did exist there. Should I put the nobody account back or is it ok to leave it deleted? Also, I am concerned that someone managed to reset the nobody account from 'No login allowed' to 'No password required'. What else should I be looking for?
Chris
Howdy,
Yeah, you would indeed want the "nobody" account to exist, but it shouldn't have a password set (ie, it should be disabled), and you may want to set the shell to something that's not allowed to login (such as /sbin/nologin).
It's tough to say why that might have happened though.
-Eric
Thanks andreychek, I put the nobody account back.
I was wondering about the libuuid, proftpd, clamav and the mysql users. I have been going through the users one by one and I found these 4 are also is set to 'No password required'. When I check my known clean system I find the same thing. However, I would still like to set this to 'No login allowed', just for my peace of mind. Can I do this without breaking something?
Also, as I mentioned before, is there some way to determine what account is being used to send email. In syslog, the only thing I see that has anything to do with the sender is the from email address. These messages have from users that do not exist on my system. So obviously they are spoofed. Which is leaving me with the problem of finding the compromised account.
Thanks in advance, Chris
The "libuuid" user should not be able to log in (through SSH), since it has an invalid password in /etc/shadow. The checkbox "Login temporarily disabled" should be checked in Webmin's user manager when look at that account.
When you take a look at /etc/shadow, all those system users like nobody and postfix and dovecot and so on should have a "!" or a "*" as their encrypted password (the bit after the first colon). That means they cannot log on from the outside, but their account can only be used for spawning processes from already logged in or active users (like root when services are started up).
When a user sends authenticated email via Postfix, you'll find a line like this in the mail log in /var/log:
Feb 1 21:37:34 HOSTNAME postfix/smtpd[19922]: 7F88E4091B: client=CLIENT-FQDN[CLIENT-IP], sasl_method=PLAIN, sasl_username=CLIENT-USERNAMEThanks Locutus, I missed the little checkbox for "Login temporarily disabled". My mind is a bit more at ease now.
As for the log entry, I was looking in the wrong log. I was in syslog, not the mail log. Once I got to the right log, I did not find any entry exactly like the one that you show but I was able to find the account that was responsible. It turns out that the account was a root virtual server account that was given to a web developer to upload their work to. Also, even after I have changed the password, every time I activate the SMTP server, it starts spewing spam. Could this mean that some malware was installed in the offending account? Is there any way for me to find it?
Chris
You can use the command "ps aux" to check for any processes that should not be there. Otherwise, can you post a log excerpt of "spam getting spewed out as soon as you start up the SMTP server"? It's a bit hard to tell what might be happening exactly just by guessing...
Locutus. Sorry for the slow response, but I have been rebuilding the server. That takes a while when you have to coordinate with the host to transfer vhd files.
I have another couple weeks until launch and I figure if I rebuild the server, then at least i can trust that portion of the system. The only untrustable part left then will be the website code.
But to answer your questions, I have prepared 3 files. One is the output from the ps aux, the other two are mail log excerpts. Note that I have changed identifying info such as domain name and user name to generic terms. I just don't want to advertise that this is a potentially vulnerable server. Also, note that the 'domain-user' is the one that is 'spewing the spam' and that the UID of that user is 1003. That UID in the pickup cleanup file is how I determined the offending account.
Thanks, Chris
In the process list, I noticed that there's a "ps" command and also an "sftp-server", running as user 1003. Are you aware of those? Did you do the "ps" as that user, and did you (or your customer) have an open SFTP connection?
As for why Postfix continues sending spam: it's possible that those mails are still sitting in the mail queue, which is persistent across restarts, and Postfix tries to deliver them as soon as it gets started back up. You can verify the mail queue contents in Webmin's Postfix module, or with the
postqueue -pcommand.Yes, I did run the ls command as 1003 through a secure shell terminal and it has a SFTP window open.
I ran the postqueue -p command and found almost 400k messages. Then I Googled how to kill these messages and found this command and this is what happened:
root@VM:~# postsuper -d ALL postsuper: Deleted: 399553 messages root@VM:~#
I noticed that there was still entries being made to the log after this command was run. There was a lot of errors, pickups, cleanup, smtps, and so forth. Is it now safe to let the smtp server run? Is there any thing else I should do?
Chris
Okay, the processes from user 1003 are explained then.
When the Postfix queue is empty and the hacked account has a different password now, it should be safe to start Postfix back up. Watch the log when you do so.
You'd need to post the mentioned "lots of errors, pickups, cleanup, smtps and so forth" log lines here though if you'd like me to take a look at them. Guessing if they constitute a problem is a bit hard. ;)
Here is a sample of the mentioned log entries. They seem to come in distinct waves. Sometimes a bunch of errors, other times smtps, and like below a mix.
Feb 10 10:28:48 VM-name postfix/error[16285]: CDFE824037F: to=thiensu1504@yahoo.com, relay=none, delay=0.02, delays=0.01/0/0/0.01, dsn=4.4.3, status=deferred (delivery temporarily suspended: Host or domain name not found. Name service error for name=yahoo.com type=MX: Host not found, try again)
Feb 10 10:28:48 VM-name postfix/pickup[14967]: D1B59240380: uid=1003 from=virgie_craft@domain.com
Feb 10 10:28:48 VM-name postfix/cleanup[16781]: D1B59240380: message-id=20140210182848.D1B59240380@VM-host.com
Feb 10 10:28:48 VM-name postfix/qmgr[14968]: D1B59240380: from=virgie_craft@domain.com, size=828, nrcpt=1 (queue active)
Feb 10 10:28:48 VM-name postfix/error[16365]: D1B59240380: to=thiensu198981@yahoo.com, relay=none, delay=0.01, delays=0.01/0/0/0, dsn=4.4.3, status=deferred (delivery temporarily suspended: Host or domain name not found. Name service error for name=yahoo.com type=MX: Host not found, try again)
Feb 10 10:28:48 VM-name postfix/pickup[14967]: D5F53240381: uid=1003 from=virgie_craft@domain.com
Feb 10 10:28:48 VM-name postfix/cleanup[16777]: D5F53240381: message-id=20140210182848.D5F53240381@VM-host.com
Feb 10 10:28:48 VM-name postfix/qmgr[14968]: D5F53240381: from=virgie_craft@domain.com, size=828, nrcpt=1 (queue active)
Feb 10 10:28:48 VM-name postfix/error[16275]: D5F53240381: to=thiensu771969@yahoo.com, relay=none, delay=0.01, delays=0.01/0/0/0, dsn=4.4.3, status=deferred (delivery temporarily suspended: Host or domain name not found. Name service error for name=yahoo.com type=MX: Host not found, try again)
Feb 10 10:28:48 VM-name postfix/pickup[14967]: D92F5240386: uid=1003 from=virgie_craft@domain.com
Feb 10 10:28:48 VM-name postfix/cleanup[16781]: D92F5240386: message-id=20140210182848.D92F5240386@VM-host.com
Feb 10 10:28:48 VM-name postfix/qmgr[14968]: D92F5240386: from=virgie_craft@domain.com, size=825, nrcpt=1 (queue active)
Feb 10 10:28:48 VM-name postfix/error[16297]: D92F5240386: to=thiensu@yahoo.com.vn, relay=none, delay=0.01, delays=0.01/0/0/0, dsn=4.4.3, status=deferred (delivery temporarily suspended: Host or domain name not found. Name service error for name=yahoo.com.vn type=MX: Host not found, try again)
Feb 10 10:28:48 VM-name postfix/pickup[14967]: DC9602403AE: uid=1003 from=virgie_craft@domain.com
Feb 10 10:28:48 VM-name postfix/cleanup[16777]: DC9602403AE: message-id=20140210182848.DC9602403AE@VM-host.com
Feb 10 10:28:48 VM-name postfix/qmgr[14968]: DC9602403AE: from=virgie_craft@domain.com, size=827, nrcpt=1 (queue active)
Feb 10 10:28:48 VM-name postfix/error[16323]: DC9602403AE: to=thiensu_1641@yahoo.com, relay=none, delay=0.01, delays=0.01/0/0/0, dsn=4.4.3, status=deferred (delivery temporarily suspended: Host or domain name not found. Name service error for name=yahoo.com type=MX: Host not found, try again)
Feb 10 10:28:50 VM-name postfix/postfix-script[16935]: stopping the Postfix mail system
Feb 10 10:28:50 VM-name postfix/master[14966]: terminating on signal 15
Feb 10 10:53:24 VM-name dovecot: imap-login: Aborted login (auth failed, 1 attempts): user=<info.domain-user>, method=PLAIN, rip=74.111.162.67, lip=198.252.68.71
Those look rather harmless to me, like somebody trying to send email to an incorrect address. You can check the queue contents in such cases if the mails are still there and if they have suspicious contents.
The "aborted login" at the end is probably a dictionary attack. Those happen all the time.
I checked the mail queue and found over 7k more messages. I deleted them, checked to ensure that the queue was empty, waited about 10 minutes and checked again, and found another 200 messages in the queue.
This is the account with the website in it. Is there a way to determine if there is a addition in the website code causing the problem. It is a Joomla site. So it is done in PHP.
My apologies, it seems your log entries were not so harmless after all if you still experience a queue growth of that kind.
You could disable the site and wait if that stops the influx of spam. Or you could run the software Linux Malware Detect against its web files. The mail log should also reflect if the site is adding spam through local delivery (for which Postfix doesn't need to be running, since it can be done by invoking executables from PHP).
Locutis, I have deduced that the problem is in the website. I have disabled the virtual server and the spam stopped and then I disabled the apache server to see if the spam was in the website. It turns out that the problem was in the site.
So I have been spending my time since removing malware scripts from the site. I have been picking them out of the aphache logs and then greping for copies of the malware files. I have managed to cut the spam but every so often a new copy of the spam script creeps in. I have followed the source of the spam files back to other files by searching the logs. I removed those files also. The problem is that I am not sure if there is any more files hiding.
I tried to install the malware detect package but the DNS on the system seems to have been tampered with because the domain to install the package will not resolve. I checked the hosts file but did not find anything. Any ideas of what is wrong with the DNS?
Furthermore, when I installed the malware detect program on a development server all I got was the output in the attached file. Any ideas what I am doing wrong with the Malware Detect package?
Chris
What domain was it that did not resolve? What exactly did you type? It's a bit unlikely that the DNS on your system has been tampered with, because that would require much more than one hacked website, since root access is required for that.
The parameter to maldet is --scan-all with two dashes.
If you can't get the website cleaned, it's better to restore it from a backup probably.
Locutus, I tried to reproduce the DNS problem to send yo the output and it worked today. So it must have been an intermittent internet problem or more likely I typed or cut and pasted it wrong.
I got the malware detect to work and I found 7 more files that were hiding. So maybe I am done now. Any last advice?
Chris