Hello and a happy new year!
I use Virtualmin for some years and offering hosting services to some clients, mostly friends and recommendations. Every now and then I get complaints from my datacenter about spam abuse and I always track it to a hacked Joomla or Wordpress installation or a hacked weak-passworded email account. The stupid part is that I then have to speak with website owners and convince them to figure this out and in most of the cases those are guys not knowing a thing about websites and security, the website was made by some amateur not maintained and not reachable so I have to start digging into other's dirt but with limited power as I don't know much about that website.
Although I read many topics on the forums about hacked servers and got some interesting recommendations, I would like to open a discussion as there are many experienced administrators here, to figure out an up-to-date checklist of things that should be done in 2016 to optimally secure a shared Virtualmin hosting platform against today's threats.
Mostly, how to prevent server/website hacks and what strategies are good to limit impact in case this happens. Hopefully this thread could become an important shared resource for any Virtualmin user.
So I guess this would be the requirements for such an environment:
- Account isolation: I think Virtualmin does this by default via mod_fcgid?
- Brute force / intrusion prevention: I know there are tools like CFD/OSSEC, what would you recommend?
- Request protection: Are there any request filters preventing POST's with malicious content to the servers?
- public_html protection: I guess this is really website-dependant but are there any recipes for major CMS platforms to make most of the things readonly/inaccessible to website user?
- Intrusion detection: I have some experience with Linux Malware Detect, also read about RKHunter. But unfortunately they don't find by far all the issues. Do you use them? Are there any other tools? Is there any Virtualmin integration support here?
- Limiting internet access: Account isolation is good but still servers are usually hacked also for their internet powers. What can be done to generally limit internet access to website users to prevent network scans / attacks originating from a hacked server?
- Limiting email access: By default, you can just use php's mail() and send whatever you want. Is using SMTP only a good strategy? How can we configure this?
- Scanning/limiting outgoing email: I know we have SpamAssasin/ClamAV and they're doing their thing for incoming email. What about outgoing email, how could we scan/limit/detect something is wrong here? Could we integrate SpamAssasin/ClamAV also in the outgoing chain and throttle the sending, maybe also send an email about this? The pattern is that after being hacked, a website that was maybe sending a few notification emails per day, now queues hundreds per hour
- Disinfection: I think that this is really hard and unless you have huge time/experience, you cannot do it completely. Wordfence is great though at this as does most of the work but only for Wordpress. LMD scans, finds but you have to do it manually. Are there any tools that may help?
By contributing to this topic with your experience or at least some tools or guidelines pertaining to some categories it would mean a great community help in efforts to combat attacks. I am still wondering how large hosting companies do it? What's build in cPanel or other tools?
Thank you all.
Until Virtualmin support Cloudlinux i would not use in any production environment. To host your website ok but to host others definitely no. Today there is almost no shared hosting what is not based on Cloudlinux for the simple fact that if something go bad with one account others are safe and not affected.
You can add Fail2Ban on the list
Keep updated everything, from control panels to scripts (e.g. CMS), do not use hacked/nulled addons or themes...
Yes but they are asking a lot of time to keep up with initial setup and later with all changes. Usually this is sold as separate service.
Scripts will just help but they are not 100% bulletproof and everything comes down to your knowledge how to deal with each problem.
All what i said from 1-5
Just turn off php mail() function but keep in mind not all scripts/addons can work with SMTP and some of them probably will need custom coding/adjustment
Spamexperts no question are the best but its not free and you must pay per domain. Their software can scan incoming and outgoing emails and prevent spam. Actually their filters are really good and personally i never had any problems with them.
Yes but like i said in 5. aside of few scripts you will need a lot more knowledge how to deal with each situation.
If you didnt make any special deal you are not responsible for the website of your clients. If they get hacked you need to shut down their account and ask them to repair by themself or in case you know what are you doing to pay you for the service. In case this will repeat again you must permanently shut down their account and based on your ToS refund the money or not.
It is clear that you are not ready to go with production hosting and you have two options, keep hosting your family and friends and continue to learn or be prepared to pay someone else to secure your server and keep up with all updates e.g. pro-active management.
I know 3 companies who are doing all this for cPanel/Plesk for 30-50$/month but i'm not aware of any reputable company who will do the same for Virtualmin. To give someone so sensitive and important data of your server they really must be trustworthy or you could end in big disaster. What is best thing to do i dont know and you are the only one who can answer to this question. There is a chance for someone to jump in this topic with "contact me i know what to do" but 99.9% this is only one guy who cant keep up 24/7 one server let alone many servers and clients in the same time. When it comes to hosting this is really bad solution because the problem can appear at 8 in the morning, 5 afternoon or midnight and one person cant handle all this alone. Thats why i said you need reputable company where you have 10-15+ sysadmins who can cover 24/7 and answer to your request/problem in less than 15-30 min otherwise you will have hours or even days of downtime, a lot of angry customers and pretty decent amounts of chargebacks.
To pay pennies and expect something good in return never worked out so keep this in mind.
- I often come to the conclusion that my brain has too many tabs open. -
Failing at desktop publishing & graphic design since 1994.
Thank you for your insight. There are some interesting names you mentioned.
However, to me Virtualmin doesn't seem just a personal project control panel. Professional has Reseller support and the options available seem far beyond what I've seen in cPanel for example. It isn't so user friendly but I think it's more of an experienced user tool. I'm surprised to read your positioning. And even more surprised that no one actually replied to this.
Anyway. the idea was to do best we can with Virtualmin and available tools, maybe raise some ideas about where it can be improved. You got it wrong. I don't want to go with 'production hosting'. It's just that I want to improve and I thought that together (the community) can share our thoughts and find a good 'recipe' for securing our installations and hosted virtual servers.
I agree with most of Diabolico comments above. Virtualmin is worth learning but in it's current form it's not very 'user' friendly and it seems that in some ways the 'hosting' side of the tool was a little after thought (IMHO) cPanel / Plesk have essentially cornered the market (in different ways) The issue with VM is that as a 'user' panel it's messy, far from intuitive and has a pretty steep learning curve. I have spent the last 6 months firing up VPS machines and installing (mainly centos 6/7) Virtualmin / ConfigServer / FailtoBan and learning the 'basics' of hardening an install... changing ports / ssh etc etc
I can now do it in about 30/40 mins and get a pretty locked down server but I would still not put it into a 'production' enviroment
It's interesting to see the direction Virtualmin are now taking it seems to have taken on a new lease of life but I cannot really believe it will become a consumer lever panel, for administrators etc but this will probably mean it doesn't become a 'defacto' option with hosting providers as most 'clients' like you haver stated want a 'point and click' enviroment, what goes on underneath the GUI is not their concern.
I think VMin will possibly become the reason cPanel etc start to lose custom to systemadmins as from the backend cPanel is too simple and Plesk the same. With a confident sysadmin able to hire a few VPS / Dedis they are much more likely to consider VMin.
Howdy,
We of course disagree with diabolico's view on using Virtualmin in production. Installations of Virtualmin are measured in the hundreds of thousands, so there are plenty of folks using it as such.
But yes when someone breaks into a web app and uses it to send spam, that can be troublesome for all involved.
There's a few thoughts on that --
What you're describing is actually one of the reasons Jamie added the email rate limiting feature. The email rate limiting can catch a spam attack like that and prevent it from spewing large amounts of spam.
Also, you can always put repeat offenders on their own IP address, and use the Postfix Sender Dependent IP address feature.
That way, the IP address owner is sending out email on their IP, not other IP's... so if they end up getting it blacklisted by sending spam, that doesn't affect the other users on the box.
-Eric
Thank you for your help. I get (again) valuable insight from you.
I also thought about rate limiting and I will really give it a look. Is it possible to get a notification when sending queue gets large? That's about detecting the issue. And regarding acting, what can one do to quickly flip a switch and turn off email sending capabilities for a given Virtual Server?
Regarding per-server IP, well, IPv4 is expensive. But we have tons of IPv6. Is it possible to give Virtualmin a pool of IPv6 and have one easily attached to each server during creation? Then configure the same address to be used in Postfix email sending for that Virtual Server and have priority over IPv4. That would be a cheap way of sending email via a dedicated IP in most cases, shared IPv4 as fallback.