Virtual Server Backup fail after upgrade to Virtualmin

Same exact problem here. My OS is Ubuntu 10.04 LTS

Are these scheduled backups, or are they done via the UI or the command-line API?

There was a change in Virtualmin 4.13 that makes the mysqldump command used for backups run as the domain's user, rather than root , for security reasons.

You should try SSHing in as the domain user instead, and see if you can still connect to MySQL.

I sshed with the domain user and I do connect to MySql as the domain user so mysql connectivity is not the issue ... As rhostager states the error occurs for both manual or scheduled backups, configured from the web ui (virtualmin's backup and restore utility).

Hello,

I have the same problem on 2 of 3 VPS servers with scheduled backups (set through UI and running daily):

2xDebian 6 failed to backup with:

Dumping MySQL database xxxxxxxx ..
.. dump failed! sh: can't create \/home\/lembpro\/\.backup\/yyyyy\-yyyyy\-yyyyy\.tld_mysql_xxxxxxxx: No such file or directory

The failure is for the first virtualhost, backup lasted 3 sec.

1xDebian 7 succeeded ... ???? don't know why this one, shouldn't be different as far as setup is concerned.

Pierre.

This error occurs both on manual (via UI) and scheduled backups. Everything was working before upgrade.
mysqldump: Got error: 1045: Access denied for user 'user'@'localhost' (using password: YES) when trying to connect

Same problem here, runing Virtualmin on several Debian 7 (and some Debian 6) servers :

Creating backup for virtual server domain.com .. Dumping MySQL database domain .. .. dump failed! mysqldump: Got error: 1045: Access denied for user 'domain'@'localhost' (using password: YES) when trying to connect

Edit : happened on all scheduled backups

Hi, I have the same problem after the upgrade to version 4.13 gpl.

mysqldump: Got error: 1045: Access denied for user 'xxx'@'localhost' (using password: YES) when trying to connect

The backups are scheduled, but I tried also to run the backup manually, with same result.

My VPS is a Ubuntu Linux 12.04.5 box.

Hi! Same problem here.

OS Debian GNU/Linux 7.7

I can connect via ssh as the domain user and make a mysqldump on the command line, but scheduled or manually backups using Virtualmin don't work.

Hello!

We can verify the issue with UI scheduled backups after the latest virtualmin update yesterday.

Couple of notes:

-This is verified by us on multiple Wheezy Debian Virtualmin setups

-Squeeze (Debian 6) scheduled backups run fine.

-backup API works as expected

eg: virtualmin backup-domain --dest /backup/incident08012015/ --all-domains --all-features --separate

UPDATE:

The new feature running backup as user seems to introduce a couple of quota limit errors as well

1) output from scheduled backup Failed to write to /home/USERNAME/.backup/example.com_mysql when closing : Disk quota exceeded

2) Even for API backups maybe? md2: write failed, group block limit reached. Failed to write to /tmp/.webmin/387460_31096_1_backup-domain.pl/example.com_web_alog : Broken pipe at ../web-lib-funcs.pl line 1397, line 885718.

Just adding a "me too" (sorry!). Lots of error reports emailed to us this morning! Backups are scheduled.

Hey guys,

We're also seeing this across our Pro and our GPL servers, running Ubuntu 12.04 and Ubuntu 14.04.

Scheduled and manual for me as well as of last night.

Debian 7 Ubuntu 12.04+ & 14.04+

same here.. after update

" Dumping MySQL database databasename .. .. dump failed!

mysqldump: Got error: 1045: Access denied for user 'domain'@'localhost' (using password: YES) when trying to connect "

Ubuntu 14.04.1 with Virtualmin 4.13.gpl GPL

Same issue. I cant seem to subscribe without adding a comment here. Thanks.

me too, subscribe

Also having this issue with 3 out of 4 servers. Running Debian 7 with MySQL version 5.5.40

Same problem here, subscribe. Ubuntu 14.04 LTS.

Same here... CentOS 6.6

Scheduled backup: mysqldump: Got error: 1045: Access denied for user...

I should mention though that I have two nearly identical servers that are both up to date and the backups succeeded on one of them.

Hi, here are results of my investigation :

Only one of my servers backuped normally and it happened that it had the same password as the mysql admin password.

When I changed mysql admin password in webmin to match password of my other servers, my servers backuped but the one previously working (matching with old admin password) stopped working

So it seems that backup routine is broken and now uses the mysql admin password whatever is the server it tries to backup ..

Please help

Same here... CentOS 6

I've the same issue with CentOs 6.6 and Schudeled Backup from UI.

Same here on Debian Linux 6.0.10

Same here, subscribing. Scheduled backups now failing for any virtual server with a MySQL database since upgrade to 4.13.gpl running on Ubuntu 10.04.

Same heer, sorry!

For those who need a quick fix for this (which reverts Virtualmin to the old insecure behavior of writing backups with root permissions can do the following :

1. Edit the file /usr/{share,libexec}/webmin/virtual-server/feature-mysql.pl

2. Change lines 947-948 to :

3. Run /etc/webmin/restart

Thanks Jamie but when can we expect a full solution?

Did you not test things before release?

Can we expect a secure answer and update soon?

Also better testing in the future.

Failed to lock file /home

Think I did all right?

Backup failed.

Edit....

Fix is working for me now, I must have goofed earlier.

Thanks Jamie.

hello,

I have the same problem when I run backup module in this server :

SO Debian 7 Webmin version 1.730
Virtualmin version 4.13.gpl GPL Theme version 8.7 Kernel and CPU Linux 3.2.0-4-686-pae on i686

Error:

Dumping MySQL database default_page .. .. dump failed!

mysqldump: Got error: 1045: Access denied for user 'default-page'@'localhost' (using password: YES) when trying to connect

now we can't do backups! any solution?? thanks

same probleme here on my debian systems, both gpl and pro

Thank you for this fix JamieCameron (http://virtualmin.com/node/35764#comment-142284). Keep up the good work!

Hi so it worked for you?

Hello! Yes it did! At least i was able to execute them manualy from the web ui... I assume this resolves my scheduled nightly backups too. I 'll let you know if it doesn' t...

Thank you very much for the fix Jamie :)
For those that still can't use backup, please do restart webmin or the server itself.
It should work after...

Annoyingly, I cannot re-produce this issue on any of our test systems :-(

If anyone has a system that is showing this problem and would like to allow me remote root SSH access, please email me at jcameron@virtualmin.com

Using Debian version 7.7 (amd64). with 4.13.gpl we stumble into the exact same problem. All backups that worked before (either automaticly or manually) now break:

Dumping MySQL database mike_forum ..
.. dump failed! mysqldump: Got error: 1045: Access denied for user 'heersers'@'localhost' (using password: YES) when trying to connect

All MySQL database backups seem to break. Unfortunately we don't host PostgreSQL so no clue if that's broken as well.

I'm currently trying to track this bug.

I'm not a perl guru, but the problem seems to be that authstr is not recomputed when dumping as a specific user.

I've dumped it and it looks like "--password=ROOT_PASSWORD" while it should be "-u USER_LOGIN --password=USER_PASSWORD"

It occurs in function backup_database in file /usr/share/webmin/mysql/mysql-lib.pl

We should have this before command execution, but we dont know the user password at this point and if hashed passwords are in use, this is a no go. local $authstr = make_authstr($user, $pass);

So, i don't think it's even possible to backup as site user if we're using hashed password.

To reproduce the bug, you simply have to create a Virtual server with the user's mysql password different from root's one.

Hope it helps !

Temp fix worked. Many thanks.

Awaiting for permanent fix.

I also see that if we have long domain name eg. network-interfaces.com in virtual host details database login is shorter e.g. network-interfa than in backup log script is using wrong login to database network-interfaces instead of network-interfa

Same problem here, just to confirm this, if not already.

Temp fix is working. Thank you for that Jamie

Also of concern: as a side effect of this, the cron emails relating to these backups are sending our backup target's password in clear text to me. This definitely shouldn't be happening.

Hi

JamieCameron fix is good and now my backups work very well!

http://virtualmin.com/node/35764#comment-142284

Good job Jamie! Thanks.

Hi Jamie,

do any of the account names on your test systems have long usernames ? mine failed as one of the usernames is cowhide-footstool, but for db access it has to be cowhide-footstoo

Access denied for user 'cowhide-footstool'@'localhost'

using the quick fix I can now backup ok again

Issue is happening on my systems as well. Centos 6.6, latest virtualmin release.

Work around code change worked for me. But running /etc/webmin/restart from within virtualmin brought virtualmin down and had to SSL in to start it with /etc/webmin/start.

issue here as well

Operating system CentOS Linux 6.6 Webmin version 1.730
Virtualmin version 4.13.gpl GPL

jamie's fixed solved it for me, thanks

First of all, Happy New Year Jamie and Joe and team, and many thanks for your great work and this big step in better security, which is an important step.

Regarding this issue: Same here, all automated backups of sites with mysql database(s) on all my PRO and all my GPL servers did fail for all mysql backups last night (and sites with 95% disk quotas used failed too, which is not right too, and generated an error mail containing password): So we have 3 bugs:

1. --user and --password of database missing in mysqldump command
2. backups scheduled by root are stored as owned by site instead of root, generating disk quota overflow
3. when disk quota overflows, an email containing user and password of user or database is sent to root.
4. another dump is still done by root

PROBLEM 1: (REGRESSION)

Same issue here on all (all are Ubuntu 12.04) servers for all scheduled backups of sites with MySQL databases (the ones without mysql database, and that had enough disk quota worked fine):

Note that it's the SITE user and not DATABASE user that is used to try to access the database.

If this can help: My setting is to use different password for mysql databases than for the user (as usually the database password is stored in a config file of the site, it avoids priviledge escalations). But even for the sites created earlier which still have same database password as user (I checked the config file setting, the database password and the user password which all correspond), they fail too.

I did a bit of digging into the Virtualmin GPL code: The mysqldump command is completely missing the --user=USERNAMEHERE --password=PASSWORDHERE parameters, which should come from the "Edit Databases" Username and Password settings, and not from the user himself. And this is probably the main bug.

PROBLEM 2: (REGRESSION)

Regarding file owner of backups: IMHO: Scheduled backups should be owned by the user who scheduled the backup. If root did configure the scheduled backup, the backup file should be owned by the root user imho .

PROBLEM 3: (SURE ALREADY EXISTING)

PROBLEM 4: (SURE ALREADY EXISTING)

Btw: In same file at line 693, there seems to be another backup_database() still called with user root. And doign a grep -r 'backup_database' . finds a few more.

PROBLEM 5: (SURE ALREADY EXISTING)

If PROBLEM 2 arrises, disk quota exceeded, for 1 scheduled domain backup, the whole scheduled backup stops and no other sites are backed up.

PROBLEM 6: (sure already existing too)

If PROBLEM 2 arrises, the already written files stay in /home/coastcoa/domains/toliministry.org/.backup and this makes the site fail, as disk quota which wasn't exhausted becomes exhausted, and as MySQL can't write to files anymore, the site stops working!!!!

GENERALLY:

Are you scheduling a new fix release today (European time) for urgent regressions 1 and 2 (e.g. with the --user= and --password= mysqldump parameters set properly, and file ownership set correctly) into the repositories ?

If yes I'll wait, otherwise I will have to go through all servers to do your temporary mysql fix so that next night's backup run again on most domains. But i don't have a quick fix for the disk quota issue.

Hope the above helps a bit.

Thanks again for your prompt replies and fixes.

EDIT: ADDED PROBLEM 5 and 6.

Jamie's fix worked for me. Thanks

Thanks Jamie, subscribing

Awesome, fix is working!

Subscribing.

Temporal fix is working good for me.

Thank you Jamie ;)

Happening on my servers too after upgrading to Virtualmin Pro 4.13 for some scheduled backups. But if I perform an backup for the ones which failed using the MySQL Database Server module the backup succeeds.

check your email :P

Ok, thanks to some helpful users I found the cause of this bug - it can happen when Virtualmin is not confgured to login to MySQL as a specific user (normally root), and instead connects as whatever use the command is running as.

The quick fix (which will be included in future releases) is to edit the file /etc/webmin/mysql/config and add the line :

Future releases will do this automatically.

I have a bit more to offer.

On GPL'd Virtualmin I get a complete failure of the automatic or manual Mysql part of the virtualmin backup, on domains that don't have the mySQL and Admin password sync'd. I can access the DB via the SQL module, but the backup appears to be grabbing the wrong pass? Accessing via the DB module is fine,

On Paid Virtualmin I get an email that says Subj: Cron <root@webhost> /etc/webmin/virtual-server/backup.pl --id 1248796215786 Message: user= pass=

That ID is my daily backup.

Hope this helps.

if you have mysql backups enabled those run normally - just info, they seem to be done by root

I agree with #50, backups should not be stored under user owner due to quota limits, backups should not eat up user space.

This works: "The quick fix (which will be included in future releases) is to edit the file /etc/webmin/mysql/config and add the line : login=root"

any progress dude?,

The reason why backups are now run as the domain owner user is that previously a user could create a malicious symlink from the .backup directory to a critical file in /etc/shadow , which would cause that file to be over-written when the backup is run!

However, we will be releasing an update shortly that addresses the quota issue by temporarily disabling them when the MySQL phase of the backup is done.

Hello

I don't understand that fix because I already have that login=root in all my /etc/webmin/mysql/config files (3 servers up to date ... the debian 6 backup fails while the Debian 7 succeeds ...). Is there a specific place where it should be ?

I compared the config files they all look the same but the "pass=" declaration. The server where I have a similar pwd for Virtualmin and phpmyadmin works, not the others. However I tried to sync those pwd on one of the non-working servers, still does not work, so I guess there is something else ...

The backup always fails on :

sh: can't create \/home\/userid\/.backup\/domain.tld_mysql_dbname: No such file or directory

(userid being the home dir of user ...)

I hope we get a fix soon, I'm getting nervous with no backups ...

@Pierrot - is that "No such file or directory" message the only error you are getting?

This sounds different from the initial problem of mysqldump not being able to login to MySQL at all.

I see there is an update available. wbm-virtual-server Webmin module for 'Virtualmin Virtual Servers (GPL)' 4.13.gpl-2

Do we need to revert the temporary fixes applied? Or we just go and update to 4.13-2 ?

Thanks.

I did the fix ( first one ) and was ok, but also updated the new version release this evening. All seems ok.

Yes this is the only error I'm getting and it's the one I reported on comment #6 :-)

I get the errot for the first VH included in my scheduled backup and everything stops. And I'm getting that error on 2/3 servers (the 2 debian 6). I didn't apply the fix because as I said I already had that "login=root" in config ..

The error started with 4.13 upgrade, and I just applied the 4.13-2 and I am still getting the same error on the same servers, the debian 7 one is working as usual !

My backup are incremental, all VH's, SSh to a remote server.

"#11" asserts "Squeeze (Debian 6) scheduled backups run fine." but scheduled backups fail on my debian 6 (squeeze) system too.

Jamie,

Having same issues as everyone else here on two installations. You can use mine for investigation if you like. I have not modified with the temp unsecure fix as I really don't need to at this point if you can get it fixed before my backups run next week.

Thanks. Let me know.

If anyone is still seeing this error even after updating to version 4.13-2 , please email me at jcameron@virtualmin.com

If we changed "feature-mysql.pl" file (according to the fix "#28") shall we restore the original file before updating ?

If you upgrade to 4.13-2 , the change to feature-mysql.pl will be automatically reverted.

Hi Jeamie, Thanks, 4.13-2 has solved the main problem number 1 of my list at https://virtualmin.com/node/35764#comment-142350

But not the disk quota problem number 2 (and consequentially triggered problems number 5 and 6)

Thus the backup of the whole server is yet not done, but old backups are still being deleted.

Also could you identify problems 3 and 4 of my post (others in here have posted them too) and fix them ?

This issue has been resolved in Virtualmin virtual-server module version 4.13-2.

Oops. sorry, I see there are several other things being discussed in this ticket. Any chance we could break the into new tickets, since the original issue has been resolved? It can to keep up when the thread gets so long.

Hi Joe, Done! Copied over the 5 remaining bugs here: https://virtualmin.com/node/35816

Don't want to spam bug-system for 5 tightly related issues, so made 1 new ticket for all 5 remaining ones.

I don't know if this is resolved, at least not for me. I upgraded to the latest 4.13-2 and overnight my backups failed for any MySQL databases.

Dumping MySQL database db_prod1 ..
.. dump failed! mysqldump: Got error: 1045: Access denied for user 'root'@'localhost' (using password: YES) when trying to connect

working here with the 4.13-2 thanks all

tmiller91, in your case, it does appear to be correctly attempting to login as root, though the authentication is failing.

You may want to verify that in Webmin -> Servers -> MySQL -> Module Config, and there, verify that "Administration password" is set to your correct MySQL root password.

Since this thread is getting fairly long now -- if there's any additional followup questions, it'd be great if you could open a new request, and then we can go over the issues being seen in that new request.

Thanks!

Does the new upgrade fix restart webmin or do we need to do it maually?

I have the same issue

Same issue as @tmiller91, Virtualmin upgraded but still got an error on mysql backup. No changes has been made on the mysql root password (note that I cannot refresh system information due to https://www.virtualmin.com/node/35824 ).

Ok, I found another problem that can cause the backup to fail with a message like Access denied for user 'root'@'localhost'

Specifically, if a domain has a .my.cnf file in its home directory and this specifies a default MySQL password, it will be used instead of the correct root password. This isn't a standard file that Virtualmin creates, but I guess some admins have set it up.

The work-arounds are :

1. Get rid of the .my.cnf files.

2. Apply the following code patch : https://github.com/webmin/webmin/commit/07e9a019199ed33c1cefbd7da8993960...

Greeaat ! Indeed not easy to find. Is there a way that this patch will be apply to the next release ?

Also, about the comment of @beat https://www.virtualmin.com/node/35764#comment-142350 : Problem 2 (including 5 & 6) : The ".backup" folder which is saved in the user space, and so can cause disk quota exceeded error, I think it's easy to create tar file of only ".backup" folder, and after append the user files to this tar file (tar command has an option to append files).

Maybe we should open a new ticket for this ?

Thanks !

Yes, this definitely will be in the next release.

Please open a separate ticket for any quota issues..

Thanks to release quickly please...

Could this ticket be closed? Are the separate questions filed into separate tickets yet?