Disk Quota issue in scheduled backups (introduced in 4.13, not solved in 4.13-2)

Submitted by beat on Mon, 01/12/2015 - 04:34

These are remaining problems caused by 4.13, and not resolved by 4.13-2 from http://virtualmin.com/comment/reply/35764/142510#comment-142350

As requested put into this new bug report, by Joe here http://virtualmin.com/node/35764#comment-142510 :

  1. (problem solved by 4.13-2)
  2. URGENT: backups scheduled by root are stored as owned by site instead of root, generating disk quota overflow
  3. when disk quota overflows, an email containing user and password of user or database is sent to root.
  4. another dump is still done by root
  5. in case of quota error, whole scheduled backup stops and no other sites are backed up
  6. temp files are not cleaned in case of quota error, resulting in site failing due to database quota fail.

PROBLEM 2: (REGRESSION)

Failed to write to /home/niceuser/domains/nicedomain.org/.backup/nicedomain.org_mysql when closing : Disk quota exceeded at ../web-lib-funcs.pl line 1397.

Regarding file owner of backups: IMHO: Scheduled backups should be owned by the user who scheduled the backup. If root did configure the scheduled backup, the backup file should be owned by the root user imho .

PROBLEM 3: (SURE ALREADY EXISTING)

From: root@mydomain.com (Cron Daemon)
To: root@mydomain.com
Subject: Cron <root@mydomain> /etc/webmin/virtual-server/backup.pl --id 122312522671517
Content-Type: text/plain; charset=ANSI_X3.4-1968
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
Message-Id: <20150109070327.E6818121851@mydomain.com>
Date: Fri,  9 Jan 2015 01:06:31 -0600 (CST)

user= pass=
user=mydomain.backups pass=CLEARTEXTPASSWORDEMAIEDHERE!!!!!!!!!!!!!!!!!!!!!!!!

PROBLEM 4: (SURE ALREADY EXISTING)

Btw: In same file at line 693, there seems to be another backup_database() still called with user root. And doign a grep -r 'backup_database' . finds a few more.

PROBLEM 5: (SURE ALREADY EXISTING)

If PROBLEM 2 arrises, disk quota exceeded, for 1 scheduled domain backup, the whole scheduled backup stops and no other sites are backed up.

PROBLEM 6: (sure already existing too)

If PROBLEM 2 arrises, the already written files stay in /home/coastcoa/domains/toliministry.org/.backup and this makes the site fail, as disk quota which wasn't exhausted becomes exhausted, and as MySQL can't write to files anymore, the site stops working!!!!

GENERALLY:

Are you scheduling a new fix release today (European time) for urgent regressions 1 and 2 (e.g. with the --user= and --password= mysqldump parameters set properly, and file ownership set correctly) into the repositories ?

If yes I'll wait, otherwise I will have to go through all servers to do your temporary mysql fix so that next night's backup run again on most domains. But i don't have a quick fix for the disk quota issue.

Hope the above helps a bit.

Thanks again for your prompt replies and fixes.

Status: 
Active

Comments

Submitted by JamieCameron on Mon, 01/12/2015 - 18:09

So, having the MySQL dump written to the .backup directory with domain owner permissions is expected and required for security reasons - if this wasn't done, a malicious link from that directory could be used to overwrite other files on the system.

However, the other problems (in particular the leftover files in .backup) look like bugs. Can you post the output from a failed backup in which files are left over?

Submitted by xorax on Sun, 01/18/2015 - 10:47

Is it not possible to create the .backup folder in the /tmp folder, then tar the home directory, and after append the .backup folder in the tar file ? tar command has an option to append files.

Thanks !

Submitted by JamieCameron on Sun, 01/18/2015 - 13:18

Can tar append files to a compressed file though? I don't think so ..

Submitted by xorax on Sun, 01/18/2015 - 15:32

Exact, not to a compressed file, only a tar file. There is no index of the files in the tar format : http://www.gnu.org/software/tar/manual/html_node/appending-files.html It's not possible for a compressed file without loosing too much space

I don't know the internal process of the backup, But if you need to create tgz file to compress multiple folders of different locations to one tgz file in one command line, it's possible :

cd /home/
tar czf domain.tld.tgz domain.tld/ -C /tmp/virtualmin/backup-123/ domain.tld/.backup

This is equal to :

cd /home/
mv /tmp/virtualmin/backup-123/domain.tld/.backup/ domain.tld/
tar czf domain.tld.tgz domain.tld/

The trick is to put the -C option between the source folders.

Thanks

Submitted by JamieCameron on Mon, 01/19/2015 - 22:35

That wouldn't solve the original issue unless /tmp was on a different filesystem to /home though.

Submitted by xorax on Tue, 01/20/2015 - 05:16

I don't understand why the mysql dump have to be written with domain owner permission ? Why not by nobody for example ?

Submitted by JamieCameron on Wed, 01/21/2015 - 00:12

Because if it was written by another user, there is a possible attack in which the domain owner could create a symlink from the temporary file that mysqldump will write to pointing to a file that would then get over-written.

Submitted by xorax on Wed, 01/21/2015 - 04:05

So I think it's ok if we exec mysqldump with a user like "nobody" which has no right but no quota too. And create a .backup folder with nobody as owner and with the group of the virtual-server owner, to allow "nobody" user write inside.

Submitted by beat on Wed, 01/21/2015 - 04:51

Although I prefer your way for user-generated (scheduled or not) backups, the error handling when disk quota is exceeded should in all cases be improved: Right now the whole server backup stops and following domains are not backed up (as documented in this bug report).

Here would imho be different ways to Mitigate to this attack possibility (second one is my prefered together with proper over-quota errors-handling deleting temporary files and continuing backing up other domains):

  1. To check as root that the link doesn't exist before writing the file.
  2. When a (scheduled or not) backup is made by root (and not the user himself), the whole backup should not be built in a folder in the /home/user directory owned by the user, but inside a new to be created directory owned by root and randomly named: That would allow to do the backup as the user having instructed it, without quota issues and without ownership attack issues.

Submitted by beat on Wed, 01/21/2015 - 04:54

Another mitigation possibility for many attack vectors: If VirtualMin could use docker.io for separating completely user-accounts at all levels, that would be the most secure and flexible solution ? ;-)

Any plans for support for Docker in Virtualmin and Cloudmin ?

Submitted by xorax on Wed, 01/21/2015 - 07:14

I think it's ok if we exec mysqldump with a user like "nobody" which has no right but no quota too.

And create a .backup folder with "nobody" user as owner and with the group of the virtual-server owner (to allow "nobody" user to write inside).

Submitted by andreychek on Wed, 01/21/2015 - 09:04

Thanks for all the ideas! Jamie is looking into how to best resolve the issue.

Regarding Docker -- we have been looking into Docker, and are exploring how to utilize it within the various Min environments. Docker is indeed very useful! However, I don't believe we'll end up using it as the sole means of providing hosting within Virtualmin. We'll likely start by offering it as an option within Cloudmin, though we're continuing to explore how best we can put it to use.

Submitted by beat on Wed, 01/21/2015 - 11:11

Docker should not become a replacement for current virtualmin-domains , but could be a nice new method to add new completely isolated "virtual servers". It looks and is very simple to use and would really fit well into Virtualmin too. Virtualmin and cloudmin could also be commanding an OpenStack installation. Many new exciting things to integrate with! :-) - But this is far outside of this bug and its reasonable fixes. ;-)

Submitted by beat on Thu, 01/22/2015 - 06:58

PROBLEM 7:

Other disk quota exceeded also block other domains from beeing backed-up, and temporary .backup folder is not deleted:

This happens despite setting "Action on error" not being set to "Halt the backup immediately" but to "Continue with other features and servers" !!!

Backup failed! See the progress output above for the reason why. Total backup time was 00 minutes, 48 seconds.


Sent by Virtualmin at: https://serverdomain.com:10000

Running pre-backup command ..
.. done

Creating backup for virtual server quotaissueddomain.org ..
    Copying virtual server configuration ..
    .. done

    Copying records in DNS domain ..
    .. done

    Saving mail aliases ..
    .. done

    Saving mail and FTP users ..
    .. done

    Backing up mail and FTP user Cron jobs ..
    .. none to backup

    Backing up Dovecot control files ..
    .. none found

    Copying Apache virtual host configuration ..
    .. done

    Copying Webalizer configuration files ..
    .. done

    Copying Logrotate configuration ..
    .. done

    Dumping MySQL database quotaissueddomain_joomla ..
    .. dump failed! mysqldump: Got errno 122 on write


    Dumping MySQL database quotaissueddomain_tolim ..
    .. dump failed! mysqldump: Got errno 122 on write


    Copying Procmail and SpamAssassin configuration files ..
Failed to write to /home/quotaissueddomain/domains/quotaissuedsubdomain.org/.backup/quotaissuedsubdomain.org_spam_auto when closing : Disk quota exceeded at ../web-lib-funcs.pl line 1397.
    Deleting backups from local file /home/mybackupsfolder/backup-%Y-%m-%d older than 3 days ..
        Deleting directory /home/mybackupsfolder/backup-2015-01-18, which is 3 days old ..
        .. deleted 20.73 GB.

    .. deleted 1 old backups, and skipped 5 that were not old enough

Scheduled backup by root starts with that domain, and no backups of other domains are written in backup-2015-01-22 folder which is EMPTY as a result of one customer having quota exceeded !!!!

This is NOT OK. I don't care which solution you are choosing, but having Virtualmin Pro not being able to backup server at random days due to a customer exceeding quota could be considered as a vulnerability and should really be fixed with high priority. Thanks.

Submitted by JamieCameron on Thu, 01/22/2015 - 20:57

Looks like the only safe option here is for Virtualmin to not enforce quotas during backups, to prevent this kind of failure. I will update this ticket with progress on that change..

Submitted by JamieCameron on Sun, 01/25/2015 - 18:43

FYI, the next release will disable quotas completely temporarily during backups to avoid this issue.

Submitted by sunsetsystems on Mon, 03/09/2015 - 05:32 Pro Licensee

I'm still having this issue in 4.15-2. Only with websites that go over quota. It leaves the backup files in the user's directory and when it fails it stops the entire scheduled backup.