Hi, this is off-topic a little bit, but is a phenomenon that is happening on my Virtualmin box.
I have about 30 domains hosted on this box, and on one of them something is hogging up all the memory and causing the machine to crash. I suspect it's an exploit of some bad PHP somewhere on the box or some sort of DOS. It could just be a bad code that's hogging all the memory. I've scoured all the apache logs, dmesg, var/log/messages, etc and I can't find the smoking gun. Unfortunately at the time this happens I usually don't get alerted until some hours later and then I can't tell what caused it or at what time precisely.
My question to you all : Is there a good way to turn on some sort of profiling in apache or otherwise track down the offending site / program?
Thanks, -m
Howdy,
Yeah, tracking down those sorts of issues can be tough!
So you have a suspicion of what domain is causing the trouble? One thing you might start with is to limit how many processes the domain is allowed to run... that should help prevent your server from falling over.
You can do that in Administration Options -> Edit Resource Limits. The "Max Number of Processes" would allow you to specify how many processes owned by that user can be running at any given time.
To diagnose it, you may also want to take a peek at MySQL's slow query log... if there's a query that's taking a while to finish, that could be causing the hangup. Running the "mytop" utility can give you an idea in real-time as to what MySQL statements are being executed -- that may help you get a bit of a sanity check as to what's running.
You may also want to review what processes are running now, and verify that they're all legitimate. Same with reviewing the files in that domain's directory... you may want to take a close look at the various files and directories in there, to make sure nothing unusual found it's way in there.
Checking out the Apache logs in $HOME/logs/access_log around the time you're seeing the problems might help you figure out what's being accessed that's causing trouble.
-Eric
you should see that in the logs, just before the server crashes.
also, does the server reboot again?
You can turn on Bandwidth monitor to check the traffic in case of ddos.
if only apache crashes, you may want to set System and Server Status to restart Apache after it crashed
Thanks for the tips guys. I did grep all the logs for the hour before the crash and didn't see any strange requests or any traffic that really stood out. The database server is totally idle right now so no help there. I still don't have any clue on which domain was causing the issue so I don't want to limit any single one yet. I guess I'll just keep top running and hope I catch it in the act.
-m
I posted my solution for monitoring this here : https://www.virtualmin.com/node/16835
-m