Hi all,
I'm having some problems with Webmin on my system - it hasn't been the most stable system ever, but it's less so now. Every time i go to the panel, as soon as I login I get either 'The server refused the connection' or 'Empty response' error from Chrome. I've had this problem quite a lot with Webmin, and restarting it with '/etc/init.d/webmin restart' used to fix it for a while, but now I can't get past the login screen. If I do, as soon as I click on something from the menu on the left, I get the error again.
I'm not sure where the Webmin logs are, if there are any, which is why I'm posting this. I can't think of any more information to give, but as soon as you mention something I'll be able to tell you. There's no errors in the syslog either.
I'm running a VPS with Ubuntu 12.04.2, which had Virtualmin (along with Webmin and Usermin) installed by default. I have 10GB of disk (with over 6GB free, currently), 1GB of RAM (often with over 512MB free), and 1 core CPU @ 1Ghz. It was running fine for ages, with the same problem cropping up a few times, but now it's impossible to access.
Thanks in advance for the help, Tosh.
Howdy,
Hmm, what kind of VPS are you using -- is that by chance OpenVZ?
If so, can you post the contents of your /proc/user_beancounters file?
-Eric
I'm not sure if it is OpenVZ, however a quick Google told me that if user_beancounters exists then it very well could be, so it is!
Here's the output:
Version: 2.5
uid resource held maxheld barrier limit failcnt
1435: kmemsize 36219356 46432256 88080384 100663296 0
lockedpages 0 11 4096 4096 0
privvmpages 274444 327680 262144 327680 10642
shmpages 9292 19500 262144 262144 0
dummy 0 0 0 0 0
numproc 126 174 1024 1024 0
physpages 196165 262144 0 262144 0
vmguarpages 0 0 262144 262144 0
oomguarpages 132399 167927 262144 262144 0
numtcpsock 44 170 4096 4096 0
numflock 125 169 4096 4096 0
numpty 3 7 16 16 0
numsiginfo 0 69 4096 4096 0
tcpsndbuf 900736 5984496 14680064 16777216 0
tcprcvbuf 720896 2064688 14680064 16777216 0
othersockbuf 446216 1122144 14680064 16777216 0
dgramrcvbuf 0 201144 14680064 16777216 0
numothersock 290 614 4096 4096 0
dcachesize 24233306 33554432 29360128 33554432 0
numfile 2205 4370 16384 16384 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 24 24 4096 4096 0
The VPS is a cheap one from an American company, if you must know, but very fast and reliable :)
Howdy,
Yup, that looks like OpenVZ!
And unfortunately, I think that's the problem -- I think you're running into OpenVZ-based resource limits.
Looking at the "failcnt" field on the far right, it's showing a significant amount of memory failures.
Each one of those failures is an attempt by a process to allocate memory that was denied... and each one of those can lead to a service crashing.
We see that a lot on OpenVZ-based systems... if you want to continue using that particular host, you'd either need to significantly reduce your memory usage, or request more RAM -- or both!
But the issue is that OpenVZ presents RAM as being available to the Linux kernel that's actually not guaranteed, so you'd want to make sure that the RAM in use on your server isn't using any of what OpenVZ calls "burst" RAM.
-Eric
Hi Eric,
I appreciate your expertise on this matter, however the only thing I'm confused about now is why Webmin was running fine for a long time, but is now misbehaving. The server's uptime is 50 days, so perhaps I should just try and reboot it?
Over half of my RAM is available, according to Webmin (when it works) and htop; most of it is cached, but shouldn't it be able to use it when it needs it? I thought that cached memory was only there so long as no other process needed it, but removed itself when it was needed?
I do have a small amount of swap space too, or some kind of virtual RAM, but I don't think it's using that?
EDIT: I just restarted the server, and all looks good so far. I'll see if it starts to break down again over the next few weeks. It may have just been something with some old files, as it was up 50 days?
Howdy,
Well, the problem is that the RAM shown to you in "free" isn't all guaranteed, that also includes burst memory.
If at any point, there's a spike in RAM usage that causes it to begin using burst memory,
As to why this wasn't happening earlier? Well, the Linux kernel assumes that the RAM being made available to it is all guaranteed, it doesn't have any concept of "burst" memory.
Over time, the kernel begins utilizing RAM as cache, if it's not being used for something else. And the longer that goes on, the more RAM becomes used by the kernel.
On a typical system, this is an awesome thing -- it's a way for the Linux kernel to become more and more efficient, and cause things to run efficiently and smoothly.
On an OpenVZ system, this eventually leads to problems like you're seeing.
Rebooting your server would indeed help with the problem by resetting the cache -- but usage will increase over time until it begins happening again. If you ever start seeing that "failcnt" field being at a non-zero number again, that means processes may be dieing.
Unfortunately, this has become so common, that we plan to write a FAQ entry that begins something like "If you're seeing processes dieing and other strangeness, and you're using OpenVZ, this is why".
If you like, perform a search of the Virtualmin Forums for the term "user_beancounters" -- which only exists on OpenVZ systems. You'll see a crazy amount of people who have processes just randomly dieing.
That issue doesn't occur on other VPS types or dedicated servers :-/
-Eric
Eric,
Now I understand then. Perhaps there's some way to change Webmin's RAM allocation so it has a higher priority? Just trying to think of ideas :)
Anyway, thanks for the help. If there are any problems, I'll just have to restart it. I could get more RAM, but I don't think the host would simply give it to me, even though I have been with them for ages - I would've thought they'd reward loyal customers :P
Cheers, Tosh.
There's no mechanism to prioritize memory allocation for one process over another really. And even if there was - then you'd have Webmin survive and other services crash. So no real solution. :)
Getting a non-OpenVZ server with more RAM is the only real feasible way to go here. If you'd like one in Germany, I have VMware based virtual machines to rent! :)
Hi,
Yea, really this server is just going to be a problem.
Renting a server sounds good, but I don't know if you'll beat the prices I currently pay. Plus, this server at the moment, for me, is simply a 'mess-about' server to learn stuff.
Thanks anyway :)
Well, I had posted a hosting offer here on the forum a few weeks back, with info and prices, but for some reason I can't find it right now. Seems to have gotten lost somehow.
So here's a shorter version. :)
I have two powerful VMware hosts located at the German hoster Hetzner AG, on which I'm offering virtual machines. They are not resource-overcommitted, memory you see is actually available, and Webmin/Virtualmin runs on it without problem (I'm using it myself on a daily basis). You get an IPv4 address and an IPv6 /112 subnet. Management (start, stop, reboot, snapshots etc.) is possible through a web interface and a Windows application. VMware is a full hardware virtualizer, i.e. you can install ANY operating system that runs on Intel x86/64 architecture. Weekly full external backups are available on request.
Standard monthly fees are:
Individual offers are possible too, if you need specifications not mentioned here! Current conversion rate is 1,3$ for 1 EUR.
Hi,
Sorry for the late reply, however those prices still don't match what I pay without getting anything extra :)
Anyway, as I stated the server is simply a platform for me to learn on, and I suppose having not enough RAM and issues with that is a good way to learn (as I'm having to fix the problems myself).
Thank you for the information anyway.
Regards, Josh.
Howdy,
Well, what you'd be getting is 100% guaranteed RAM... your services would stop crashing, and you'd also be able to use swap space, which provides additional RAM.
If you can get by on your current system, and use that as practice to reduce memory usage, that's great... but if you find services continue to crash and that causes problems for you, you may want to consider a different provider.
-Eric
What log says (/var/webmin... ) ?
I know there is a big issue with OpenVZ and old kernel, basically, running vmstat can crash, and webmin runs this as a cron task every X hours ^^