Hello,
I'm having an issue trying to restore a single domain backup. The system is a freshly installed Debian 10 (Webmin 1.942, Virtualmin 6.09), on a 16 thread Ryzen CPU with 64GB of RAM and plenty of disk space. The backup, composed of the .dom, .info and the .tar.bz2 itself, is 1.3GB in size, and was dumped on a Debian 9 system (same *min versions). The old system relied on apache2, the new one on nginx (installed by passing "-b LEMP" to the Virtualmin install script).
Attempting the restore works up until an unknown step (unknown as in no step is being echoed) completely fills up the RAM over 15-30 seconds on the webmin process, while keeping it at 100% on 1 core, at which point the kernel kills the process to survive.
This happens both via the web interface and virtualmin-cli; I'm attaching the output log for the latter. Thank you.
Comments
Hi,
I will assign this to Jamie for a review.
Meanwhile, it looks to me like an incompatibility bug. What if you restore it without website feature enabled? What if you back it up, without website feature?
What are the outcomes in both scenarios?
Submitted by markkuit on Wed, 06/17/2020 - 09:14 Comment #2
Hello,
the culprit actually seems to be in MySQL (MariaDB). Restoring with all features enabled (including Nginx website) except for database contents is successful. Note that I had to enable the "Only create servers with selected features" option, else the issue was still present. Weirdly, after that, I tried restoring enabling only the database content feature and it went smooth, resulting in an entirely correctly restored virtualserver. However trying to restore it in one go brings up the mentioned RAM issue.
The previous server runs MariaDB 10.1.25, the new one 10.3.22.
Submitted by andreychek on Wed, 06/17/2020 - 10:36 Comment #3
Thanks for the info, that helps!
To generate a database backup, Virtualmin is just using mysqldump though. That may mean you're seeing some sort of MySQL issue rather than a Virtualmin one... that's probably not a Virtualmin bug.
Some my.cnf settings can end up using a lot of RAM, you may want to review any custom settings in there to see if any could be causing it to use a lot of RAM during the restore process.
If you aren't seeing any obvious culprits, one option would be to run through the Post-Install Wizard (available in System Settings), and in there, copy out a MySQL config designed to use less RAM.
Submitted by markkuit on Wed, 06/17/2020 - 14:45 Comment #4
Restoring the backup via mysql cli rather than Virtualmin poses no issue, as does restoring it on the already existing virtual server. Restoring it on virtual server creation seems to trigger the RAM bug.
Also, as mentioned earlier, the webmin process itself starts allocating more and more RAM (up to almost 100% before being killed by the kernel), not any other. This still definitely looks like a Virtualmin bug, not a DBMS bug or a config issue.
Is there any Virtualmin log location that could be useful, or any way to debug what restore step/line of code is being executed right then? I tried stracing it, but only got a loop of brk() (heap rw).
Submitted by JamieCameron on Wed, 06/17/2020 - 14:57 Comment #5
If you run the
topcommand during a restore and sort by RAM usage, which process is shown as using up the most RAM?Submitted by markkuit on Thu, 06/18/2020 - 03:48 Comment #6
It's "/usr/share/webmin/virtual-server/restore-domain.pl".
Submitted by JamieCameron on Fri, 06/19/2020 - 21:32 Comment #7
That seems like a Virtualmin bug for sure then.
Would it be possible to get a copy of a backup file that triggers this bug when restoring?