Howdy all,
I've rolled out Webmin version 1.930 and Usermin version 1.780 for all repositories. This release includes several security fixes, including one potentially serious one caused by malicious code inserted into Webmin and Usermin at some point on our build infrastructure. We're still investigating how and when, but the exploitable code has never existed in our github repositories, so we've rebuilt from git source on new infrastructure (and checked to be sure the result does not contain the malicious code).
I don't have a changelog for these releases yet, but I wanted to announce them immediately due to the severity of this issue. To exploit the malicious code, your Webmin installation must have Webmin -> Webmin Configuration -> Authentication -> Password expiry policy set to Prompt users with expired passwords to enter a new one. This option is not set by default, but if it is set, it allows remote code execution.
This release addresses CVE-2019-15107, which was disclosed earlier today. We received no advance notification of it, which is unusual and unethical on the part of the researcher who discovered it. But, in such cases there's nothing we can do but fix it ASAP.
It also addresses a handful of XSS issues that we were notified about, and a bounty was awarded to the researcher (a different one) who found them. (So, if you're a security researcher and responsibly disclose your findings, feel free to hit us up with anything you find. We pay.)
Anyway, we're all a little grumpy, as we needed to drop everything and sort out what happened, figure out how to fix it, and get new releases rolled out immediately. But, the packages in our repos fix it, and also include theme updates and some other new stuff (changelog coming soon). But, you definitely need to update if you use that feature. As far as we know the malicious code is not exploitable if you aren't using that option, but there are some unrelated XSS issues in Authentic theme that are also fixed, so no reason not to upgrade right away, even if the more serious one doesn't effect you.
Cheers,
Joe
Thank you for all your dedication webmin team
Thank you for all your efforts!
Thanks !
Thanks. Updated two servers.
--Richard
OK good luck
Take care while for example and not the only one.
Canonical's GitHub account hackedonce UBUNTU so :( be carefull with GITHUB ( not this one but ) and other such things / places you all.IF you read back forum there where few times Installations couldn't find connect to source to install so errors not online or other problems, maybe you can find those and have a look at those dates when these outages hapened , is only a 123 idea from me.
Appreciate how fast you guys triaged this. I'm updated.
Hi,
Is it only Webmin that is targeted? On Usermin configuration that authentication setting was set to "Prompt users with expired passwords to enter a new one". And when I changed it to "Always deny users with expired passwords" i got an error: "Failed to save authentication : Failed to open PID file", but the change seems to be saved. Is this because Usermin wasn't running, we don't use usermin so it is never running... or should I be worried???
And of course, our servers are updated.
Regards, Leffe
Is Usermin wasn't running it definitely couldn't have been exploited. We don't now of any published exploits of Usermin, but I think it would be subject to the same problem, as the build environment for Usermin links to shared files in Webmin, including this
password_change.cgifile. I haven't tested whether modifying the existing exploit in metasploit or via curl or whatever would work in Usermin, but I suspect it would.But, if you don't run it, you don't need to care.
--
Check out the forum guidelines!
i upgraded but virtualmin destroyed all my php5 config file :(
Regards, Ghislain
Check if your doing this can get everything working quickly:
https://www.virtualmin.com/comment/814992#comment-814992
after upgrade, the file manager stopped working....
on click of file manager it loads only the first level files and folders. if you try to navigate any folder from there then it load nothing.... it takes forever...
update: suddenly it started to work now... earlier the folder navigation window was also missing.... now that is also visible....
Would love to see a Solaris installer like webmin-1.930.pkg.gz, too.
Jamie revived his old Solaris box to build a package just for you, our one remaining Solaris user. (I'm kidding, there may be two or three others running Solaris.) ;-)
Our of curiosity, are you running Solaris from Oracle or one of the OpenSolaris off-shoots, like IllumOS or SmartOS?
--
Check out the forum guidelines!
Is is possible to comb thru logs to see if this exploit was abused (in my case during the time I ran 1.890)?
What should I be looking for? Access to password_change.cgi?
Yes, it shows up in the access log as an access of
password_change.cgi; though that won't necessarily indicate a successful exploit. Any version other than 1.890 requires that password expiry/change option to be enabled, so accesses would fail.When I was testing a couple of different methods of exploit (for 1.890 and for 1.920) it shows up in
miniserv.error, as well, though it depends on what is in the input (and which version of Webmin is running) as to what shows up. But, there maybe a perl error looking something likePerl execution failed - Your password has expired [...] at /usr/share/webmin/password_change.cgiSeeing that error probably indicates the system was exploited in some way, as reaching that code indicates the option is enabled or it was running under version 1.890.
--
Check out the forum guidelines!
thank you joe. does anyone know when was 1.890 released? this is not going to be fun, but i'd have to scrap my system if I find something.
It looks like 1.890 was released in the Virtualmin repos on 7/16/18, and upgraded to 1.894 about three months later. I guess it wasn't as short-lived as I'd assumed, but from what we can tell it was not being exploited widely until the tweetstorm and media coverage starting on the 17th. It wasn't being tracked by researchers who track that sort of thing until it was "dropped" at DefCon, and attacks started ramping up almost immediately, but weren't super widespread until the 17th.
Reproducing the attack based only on what was in the "0day drop" required tweaking, and so script kiddies couldn't make it work without people in forums figuring out how to make it work (it even took me a few minutes to figure out why it was failing for me). I suspect the researcher just didn't understand the bug (either discovering it through fuzzing or finding it mentioned on some black hat site, or similar), and so his PoC was kinda wonky and required a lot of luck to make work (but only a little tweaking was needed to make it super dangerous on 1.890, but it needed the right Webmin configuration on all later versions).
What I'm trying to say is that it was a slow ramp up, at least partly, because the published exploit wasn't workable in a widespread way without tweaking, and a lot of attackers are low-skilled; just running existing exploits and adding the results to their botnets for spamming or whatever. I followed the discussion all day on Twitter and reddit, and was surprised by how much wrong information was flying around (wrong in the sense of it didn't accurately describe the bug or the way to exploit it) and for how long.
All that said, if you find something, you probably should scrap the system. You can never really trust a rooted system again, until it's been reinstalled with a fresh OS. We hate that this happened, and we're sorry to everyone who does end up having to reinstall...know that we're in the same boat (all of our servers run Webmin or Virtualmin, too, and we have quite a few), and have been auditing everything since Saturday. We're just hoping the slow ramp up and getting a new release out minimized the damage. We wish the researcher had responsibly disclosed the bug, but not everyone does responsible disclosure.
--
Check out the forum guidelines!
Sooo...
Would the mere fact that someone attempted to access that function, even if it failed because password-changing was not enabled, serve as a natural honeypot?
--Richard
I think so to while function disabled we see these kind of calls in that miniserv error log to
Okay, thanks. Later on I think I'll take a look at that file and figure out how to capture those attempts, report them, and add them to a blocklist I maintain. This server would be ideal for that because I'm the only human user, so it can't be a genuine lost password reset request from a client.
--Richard
First day starting 14-08-2019Here a short list of trying ip's at 1 of our boxes. last 2 days
csf is not blocking probable i don't know or manual or automatic trys from kiddies and....?
[21/Aug/2019:16:19:25 +0200] [176.67.80.61] /password_change.cgi : Perl execution failed : Password changing is not enabled! at /usr/libexec/webmin/password_change.cgi line 12.
[21/Aug/2019:18:50:27 +0200] [144.217.207.30] /password_change.cgi : Perl execution failed : Password changing is not enabled! at /usr/libexec/webmin/password_change.cgi line 12.
[22/Aug/2019:06:50:20 +0200] [88.99.146.161] /password_change.cgi : Perl execution failed : Password changing is not enabled! at /usr/libexec/webmin/password_change.cgi line 12.
[22/Aug/2019:06:50:20 +0200] [88.99.146.161] /password_change.cgi : Perl execution failed : Password changing is not enabled! at /usr/libexec/webmin/password_change.cgi line 12.
[22/Aug/2019:06:50:53 +0200] [18.224.72.187] /password_change.cgi : Perl execution failed : Password changing is not enabled! at /usr/libexec/webmin/password_change.cgi line 12.
[22/Aug/2019:06:50:54 +0200] [18.224.72.187] /password_change.cgi : Perl execution failed : Password changing is not enabled! at /usr/libexec/webmin/password_change.cgi line 12.
[22/Aug/2019:08:31:12 +0200] [191.101.63.23] /password_change.cgi : Perl execution failed : Password changing is not enabled! at /usr/libexec/webmin/password_change.cgi line 12.
First day starting 14-08-2019
[14/Aug/2019:08:34:23 +0200] [89.248.171.57] Document follows : This web server is running in SSL mode. Try the URL <a href='https://ip-199-:10000/'>https://ip-199-.:10000/</a> instead.<br>
[14/Aug/2019:08:34:23 +0200] [89.248.171.57] /password_change.cgi : Perl execution failed : Password changing is not enabled! at /usr/libexec/webmin/password_change.cgi line 12.
[14/Aug/2019:10:11:27 +0200] [89.248.171.57] Document follows : This web server is running in SSL mode. Try the URL <a href=':10000/'>:10000/</a> instead.<br>
[14/Aug/2019:10:11:27 +0200] [89.248.171.57] /password_change.cgi : Perl execution failed : Password changing is not enabled! at /usr/libexec/webmin/password_change.cgi line 12.
[14/Aug/2019:13:20:53 +0200] [85.25.118.188] /password_change.cgi : Perl execution failed : Password changing is not enabled! at /usr/libexec/webmin/password_change.cgi line 12.
[14/Aug/2019:13:22:22 +0200] [85.25.118.188] /password_change.cgi : Perl execution failed : Password changing is not enabled! at /usr/libexec/webmin/password_change.cgi line 12.
some more bad ips trying: 103.231.92.23 51.38.184.92 37.252.122.212 193.42.221.14 89.248.171.57 34.200.237.162 124.47.191.14 95.222.60.31
And mine:
[17/Aug/2019:16:53:05 -0400] [62.210.110.80] /password_change.cgi : Perl execution failed : Password changing is not enabled! at /usr/libexec/webmin/password_change.cgi line 12.[18/Aug/2019:19:13:04 -0400] [140.143.210.34] /password_change.cgi : Perl execution failed : Password changing is not enabled! at /usr/libexec/webmin/password_change.cgi line 12.
[20/Aug/2019:17:03:28 -0400] [192.3.70.143] /password_change.cgi : Perl execution failed : Password changing is not enabled! at /usr/libexec/webmin/password_change.cgi line 12.
[20/Aug/2019:17:03:29 -0400] [192.3.70.143] /password_change.cgi : Perl execution failed : Password changing is not enabled! at /usr/libexec/webmin/password_change.cgi line 12.
[20/Aug/2019:17:03:30 -0400] [192.3.70.143] /password_change.cgi : Perl execution failed : Password changing is not enabled! at /usr/libexec/webmin/password_change.cgi line 12.
[20/Aug/2019:20:46:44 -0400] [207.38.89.142] /password_change.cgi : Perl execution failed : Password changing is not enabled! at /usr/libexec/webmin/password_change.cgi line 12.
[20/Aug/2019:20:46:44 -0400] [207.38.89.142] /password_change.cgi : Perl execution failed : Password changing is not enabled! at /usr/libexec/webmin/password_change.cgi line 12.
[22/Aug/2019:01:02:30 -0400] [50.116.18.236] /password_change.cgi : Perl execution failed : Password changing is not enabled! at /usr/libexec/webmin/password_change.cgi line 12.
[22/Aug/2019:01:02:31 -0400] [50.116.18.236] /password_change.cgi : Perl execution failed : Password changing is not enabled! at /usr/libexec/webmin/password_change.cgi line 12.
Actually, I have another box running Virtualmin that's not in production use yet. I think I'll use that one as a sandbox. My Perl skills are less-than-wonderful. But I do love setting honeypots.
--Richard
On test box running shorter time 4 months now only few domains doesn't have this logs so they didn't find yet!
Or as JOE says be carefull, i am not sure??????
You will pull out a Solaris box but won't support raspberry pi. That's just rude.
You can run Webmin on Raspbian just fine. It's even in the main OS repos, isn't it?
Virtualmin requires rebuilding (and maintaining) architecture specific binary builds.
Besides, Jamie already has a Solaris box. I don't have a Raspberry Pi.
--
Check out the forum guidelines!
so 1.890 was super dangerous how? Was the change password mitigation not required for this particular version?
ON some forum they write such kind of replys so...
here the webmin one http://www.webmin.com/exploit.html
The default configuration of 1.890 was exploitable; i.e. root-level access to anyone who knew how to trigger it. The default configuration of later versions was not exploitable. Most people with later versions will not be impacted, as most people will not have modified that option.
--
Check out the forum guidelines!
@JOE
no matter running which version in the past running >= 1900 now , is this enough for beeing safe for the scriptkiddies? ( password change option was alway disabled in GUI)
Alternately, if running versions 1.900 to 1.920, edit /etc/webmin/miniserv.conf, remove the passwd_mode= line, then run /etc/webmin/restart.You write here: http://www.webmin.com/security.html
I see this log entries at all of my servers except one: a virtualmin installation not using standard port 10000.
glad to hear this was fixed
While updating
warning: file /usr/libexec/webmin/authentic-theme/unauthenticated/js/codemirror/mode/meta.js: remove failed: No such file or directoryUnrelated. Indicates updates of the theme outside of package updates. Harmless.
--
Check out the forum guidelines!
Can someone tell me how to know if a server was compromised...
What do I look for?
//Leffe
Webmin -> System -> System Logs
For both File /var/usermin/miniserv.error and File /var/webmin/miniserv.error:
My understanding is finding that line doesn't mean it was definitely compromised, but it means you need to investigate further. I could be wrong.
--Richard
Thanks Richard,
I have a ton of lines like this on our servers
/password_change.cgi : Perl execution failed : Password changing is not enabled!but when reading what Joe wrote is it only on the later Webmin versions that setting had to be enabled for the exploit, and it has been around for a year. How and what else should I look for!? It is a catastrophe for us if we have to reinstall - We can't do that! We have to buy a new dedicated server first and then start to move customers there, because many of our customers run services as sending text messages, and they send thousands on daily basis!Regards, Leffe
You're welcome, Leffe. And I don't know. I'm new at Virtualmin and hadn't used Webmin in years until a couple of months ago. So if I answer a question at 10:00, it's quite possible that I may have just learned the answer myself at 9:59.
My guess would be that any such lines after you applied the update can be safely ignored, and any before you applied the update would need investigating. But don't take that as gospel. I'm just basing it on many years of experience as a server admin, not a lot of experience with Webmin.
To be safe, I'd wait until an expert answers. My answer is just an educated guess. I'm no expert.
--Richard
Thanks again Richard,
We have been using Virtualmin Pro for almost 15 years, we was running Pro version already in early adapter period. Virtualmin has been running fine almost all the time, but in the last years all sort of bugs keep coming up! Hopefully they get them fixed! And also tell us what to look for to know if a server have been compromised.
Our customers is mostly Swedish communs (municipality) and they don't allow downtime for their services! We have backup servers but they have the same software as the main servers.
Regards, Leffe
My pleasure, Leffe.
Would reinstalling the OS and WMin / VMin on the backup servers, and migrating just the accounts (assuming no problems in UMin), be an option if it came down to it?
Let's hope none of that is necessary, however.
--Richard
in 1890 version the exploit was completly possible without the enabled setting because of "bug" or so.
So i guess there you don't see these text in log files at all.
So not knowing where to look for in the time space when having version 1890 installed in the log files if yes or no hmm.
Problem is how many did know about that the time 1890 version, and did they abuse with .......... if you read several people find out, even kind off discussing who was first and when trying to selling 0day exploit to some . The buying site didn't want to buy because it was already known.
As JOE and so are writing a lot of true not true txts on the web and different infos to.
Even at one place a lot of ...
https://www.reddit.com/r/netsec/comments/crk77z/0day_remote_code_executi...
I gues i hope Joe can tell where to look for with systems that had version 1890 installed onceThe exploit does trigger log events, which I've posted in a few messages in this thread and on github, whether successful or not (the code of the exploit is very sloppy...a good programmer could have made it not trigger any logs or visible errors, but this one does). But, if a server is rooted, it may not be possible to trust the logs (because root can modify or delete logs).
So, if you find the errors that come from the exploitable code path (
expired passworderror), you probably need to treat the system as if it has been compromised. If you find the other errors, you probably still want to look for breadcrumbs, because a competent attacker would clean up the logs. But, not all, or even most, attackers are competent. Most are just using automated tools to spin up as many nodes on their botnet as possible, and don't care if some get found out and removed.--
Check out the forum guidelines!
That specific line is a failed attack.
The other error I mentioned (
password is expired) is indicative of a likely successful attack (or a probe by a researcher that is counting exploitable hosts). The code path that produces thePassword changing is not enabled!error is not exploitable, and is behaving as intended.If you only see "Password changing is not enabled!" messages, you're probably fine. Though it's hard to be 100% sure. I recommend doing all the other stuff one would do in an "am I compromised?" situation.
rkhunterand similar tools, verify your packages (rpm -Vaon CentOS) looking for modified sensitive binaries (ps,passwd, pam and login-related libraries, in particular), if you have backups, compare/etcbetween them, check/root/.bash_historyfor suspicious stuff, etc. You're looking for crumbs, as most attackers are sloppy and leave them lying around. This is what we've been doing on our servers that can't be re-installed easily (we're in the midst of migrating a bunch of stuff into cloud-based services, so some stuff was already scheduled for retirement).You also want to watch for outward signs of abuse: high network usage (meaning it's sending spam or otherwise participating in a malicious botnet), mysteriously high CPU usage (mining Bitcoin, or whatever), in particular.
--
Check out the forum guidelines!
Thanks we now know where to search for. At our box is only the scam scanners trying to, not
password is expiredI share some IPS. Are some or all in abusedip com don't know or legitimate scanners or scammers, don't have time for both! For Richards picking them up is a good idea even if legitimate they are spoiling resources. ;)@JOE @JAMIE
Why one can still download old version are those cleaned up?
https://sourceforge.net/projects/webadmin/files/webmin/1.920/webmin-1.920.tar.gz/download?use_mirror=vorboss
https://sourceforge.net/projects/webadmin/files/webmin/1.920/webmin_1.920_all.deb/download?use_mirror=netcologne
We leave old versions available for archival purposes. But, you should never use an old version. They are not modified, and are exploitable.
If you use apt or yum, you'll always get the latest available version. You'd have to go out of your way to install an old version...it won't happen accidentally.
--
Check out the forum guidelines!
Ok somewhere on the web sorry can't find it now complaining about having such online and saying.... about webmin because of that
Have bad experience with companys using not accidentally very to old insecure versions of software, because they can still download such from platforms as github and so on.
To have a more secure CYBERworld everything known unsafe should be deleted where possible, OK is my Opinion. ;)
damn jfro stop spamming the thread.
UH only asking and helping out with info's why is that spamming? ( sorry 'm Dutch and dislexie for my Gramar)
Need for that while fixes are later then exploit known dates, so you have to check your boxes for ....
quote Joe: It looks like 1.890 was released in the Virtualmin repos on 7/16/18, and upgraded to 1.894 about three months later. I guess it wasn't as short-lived as I'd assumed,And the other version was public 10 -08-2019 or known that date so fixes are later! See here the date https://www.pentest.com.tr/exploits/DEFCON-Webmin-1920-Unauthenticated-R...
EDB-ID: 47230
CVE-2019-15107
10 Aug, 2019 • EXPLOIT
So needed how to find out if abuse of exploits on boxes between known dates and fixed dates yes or no?
At our boxes the tryouts to abuse this exploit 0Day started also before fix date of Webmin self namely 14-08-2019I posted that part of log file starting hacks 14-08-2019 here https://www.virtualmin.com/comment/816085#comment-816085 And no the password option isn't enabled!Also if for those who have this had enabled read here what todo, is only an advice from someone, while hacking to eploit this one started before the patch as in our log files 14-08-2019 https://github.com/webmin/webmin/issues/1093#issuecomment-523876740
IN this EXPLOIT DATABASE as of date 12-08-2019 So explains the start of scams after that date https://www.exploit-db.com/exploits/47230
JOE JAMIE you can ask here why the are scanning for this exploit starting at on of our box 14-09-2019 maybe the know more and why?! https://www.abuseipdb.com/check/89.248.171.57
ISP Incrediserve LtdUsage Type Data Center/Web Hosting/Transit
Hostname(s) scanner20.openportstats.com
Domain Name incrediserve.net
Country Netherlands
City Den Haag, Zuid-Holland
For the rest i don't know if it helps here but this audit / scanner can give more info is also free version available https://cisofy.com/lynis/
We only know what we've discussed. The researcher who released the exploit did not disclose it to us before releasing it. We didn't hear about it until the 17th, which is when we released an update.
We can't go back in time and change anything. This is the situation we're all in. It's the most serious security issue in Webmin in more than a decade, and all we can do about it is try to make sure nothing like it ever happens again. I don't know why the researcher didn't responsibly disclose the bug to us in advance of announcing it. And, I don't know why news of it didn't trickle down to us sooner (I have Google alerts setup for
webmin,virtualmin, etc., but it wasn't until it became bigger news on Saturday that Ilia pointed it out to us and we started researching it).I don't have any way to say, "Your servers are fine. Forget about it." We all have to poke around and look for signs of trouble. We're sorry it happened. It's not a great feeling. But, we're doing everything we can to prevent similar situations in the future.
--
Check out the forum guidelines!
Pages