Perl Segmentation fault for server and user manipulation commands

Howdy -- it sounds like your Linux system is restricting what the processes running on it are allowed to do.

It looks like grsecurity may be responsible for at least some of that.

You may want to try disabling grsecurity, or configure it to ignore the Webmin processes.

There likely isn't a simple way to get things working with grsecurity itself. So coming up with a way to disable grsecurity is the direction we'd suggest.

If you're open to using paid support, that does open up some options!

There's the option of switching to a Virtualmin Pro license, or purchasing a support incident. Or, if things are dire, you could actually hire us to migrate things to a new server for you.

We can definitely help out in that case; I'm not sure how yet, but we can figure something out.

I'm not sure which of those payment options to choose at the moment though, as this isn't an issue we've run into very often. Maybe you could hold off on buying one of those options until we have a better idea as to the problem?

I'd like to think that there is a way to switch kernels and have that work.

Can you describe what happens when switching to a standard kernel? You said you couldn't use the eth driver -- what was happening there exactly?

Just to verify, as I'm not fully certain I understand -- are the kernels you're trying now provided by OVH? Or are they Debian kernels, provided by the distro?

No problem on the delay!

If we thought this was some sort of "normal" configuration issue, or a coding bug, we'd be more than happy to work with you on that, and to look deeper into that problem.

The kinds of problems you're seeing are really unusual though, and appear to be specific to the environment at your provider there. While there are folks using OVH who aren't experiencing that particular issue, I am wondering if the easiest solution isn't to just change providers (which I think you were mentioning as an option above, assuming you were able to run the appropriate commands to do so).

You mentioned that list-users isn't working. However, is the backup function working? Or does that error out too?

It really looks like there is some kind of security or resource restriction that is preventing things from working properly.

All that said, I do have a few questions for you just to rule out a handful of simpler things --

What kind of VPS do you have? For example, Xen, KVM, OpenVZ?

And what is the output of these two commands:

Oh, and this one might generate quite a bit of output, but I'm curious what it shows:

Segfaults mean that "something under the hood" is breaking. Something at the kernel level, or maybe a library.

Perl code can't cause a segfault. Instead, it indicates that something is very awry.

It shouldn't happen due to MySQL columns and such -- that would generate friendlier errors, as that's a more normal/expected kind of problem.

That MySQL activity in the strace output is most likely just normal activity that Virtualmin was performing when something else caused a problem.

That gave me an idea though.

What is the output of this command:

cat /etcsecurity/limits.conf

Try commenting out everything in your limtis.conf file, and see if there is a change after that.

Try increasing the MySQL section perhaps to 200000, and then comment out everything else.

You can always wait until after business hours to test that.

Okay... I'm unfortunately not quite sure what else to try there.

Have you tried talking to your provider about this issue?

It's possible they've heard about it before and would know what to do.

Other than that, the only other thought i have would be to attempt migrating elsewhere, either to another provider, or to another type of host at this provider.

The normal way to do that is using Virtualmin's backup functionality, though if that's not working properly that would be a problem.

The other way would be to stop all the services, and to rsync the various important files and directories (but not rsync'ing more than necessary, as you wouldn't want to accidentally copy over the cause of the problem). That wouldn't be easy, especially with 150+ sites, but if the backup functionality doesn't work that may be the only other way to get that up and running somewhere else.

I have to agree with Eric that there's a kernel issue happening here (it could also be underlying hardware; memory or disk corruption can lead to weird stuff like this that kinda spans a bunch of different services and tasks...though rebooting would likely change the symptoms). Nothing Webmin/Virtualmin does should even be able to cause a segfault (being written in high level languages without directly managing memory, Webmin doesn't have any control over this stuff), much less consistently trigger them for some tasks.

Am I understanding you correctly that you've tried booting into different kernels? Perhaps rolling back to one that worked in the past would be an option? If not, any solution will probably have to involve the OVH folks; kernel level segfaults are out of our reach...we can't fix them.

You could also try reducing memory usage on the system, to see if it leads to different behavior. If OVH are overselling memory, it might manifest this way (I don't know, as I haven't seen this particular behavior before, but some weird errors in the past have been due to oversold memory). So, you could turn off library caching in Webmin, reduce MySQL buffer sizes, get rid of unneeded Apache modules, etc. If the symptoms change or go away, you'd be able to guess that maybe that's the problem. Some VM hosting platforms are kinda notorious for memory problems (OpenVZ and Virtuozzo, in particular, as it is used so heavily by cut-rate hosts who oversell their systems heavily). OVH doesn't have a bad reputation, as far as I know, but it could still be a memory related issue, if they've misconfigured something.

Oh, wow! I remember that issue now, but it's been years since anyone ran into it, so I didn't even think of it. I would think that would be a MySQL DBD bug. But, why it's hitting a new system is weird. I thought it was an old bug that had since been fixed. Does your system have a non-standard Perl or libraries from a source other than the OS repos?