Back-up errors referencing "setquota"

Since sometime last week some back-ups have been failing with this message:

Creating backup for virtual server example.org ..
<tt>setquota: Cannot open quotafile //aquota.user: Operation not permitted
setquota: Not all specified mountpoints are using quota.
</tt> at /usr/libexec/webmin/web-lib-funcs.pl line 1433.

This is happening on at least two virtual servers, and I can't think what they might have in common. Another virtual server is not having problems.

What could have started causing this out of the blue last week?

Craig

Status: 
Closed (fixed)

Comments

It's possible that disk quotas were disabled on your system. Try going to Webmin -> System -> Disk Quotas, and re-enable them on the / filesystem.

I'm just getting more and more confused, and this quota issue has come up before in https://www.virtualmin.com/node/52800 . To summarise:

  • I set this server up with quotas enabled from day one.
  • Months later a different back-up issue came up, as described in the above link. Despite the fact that I had enabled quotas from day one, they were now disabled.
  • I re-enabled them.
  • A couple of months later now I go to Webmin -> System -> Disk Quotas (as you asked) and I see that quotas are again "inactive".

How do quotas keep being disabled? There was nothing I did last Thursday to trigger this. I did enable back-ups on a virtual server that I had transferred in about a week earlier (not the domain in question, but the errors are the same); I got one failure, one successful back-up, and then all scheduled daily back-ups have failed since.

It sounds like something external to Virtualmin is disabling disk quotas.

Is this a physical machine, or a VM of some kind?

Also, was it recently rebooted - perhaps quotas weren't re-enabled at boot?

Well, apparently there was something I did on Thursday to cause this, because I did indeed reboot the server on Thursday. Good catch.

I have re-enabled quotas as you suggested.

However, under Webmin | System | Bootup and Shutdown, "quotaon.service" is set to "Always" under "Start at boot?" and "Yes" under "Running now?" (Didn't think to look before I re-enabled quotas.)

In case it's helpful, here are the system details of quotaon.service:

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Enable File System Quotas
Documentation=man:quotaon(8)
DefaultDependencies=no
After=systemd-readahead-collect.service systemd-readahead-replay.service systemd-quotacheck.service
Before=local-fs.target shutdown.target
ConditionPathExists=/usr/sbin/quotaon

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/quotaon -aug

And here is my /etc/fstab:

[09:13:27 root@cp31 ~]# more /etc/fstab

#
# /etc/fstab
# Created by anaconda on Mon Jul  7 20:25:55 2014
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/sda        /       ext3    grpquota,errors=remount-ro,usrquota,data=ordered,noatime,rw     0       1
/dev/sdb       none            swap    sw                              0 0
/proc           /proc           proc    defaults                        0 0
tmpfs           /dev/shm        tmpfs   nodev,nosuid,noexec,mode=1777             0 0
devpts          /dev/pts        devpts  gid=5,mode=620                  0 0
sysfs           /sys            sysfs   defaults                        0 0
[09:14:02 root@cp31 ~]#

To answer your other question, this is a Linode VM.

Hmm, so it sounds like you're saying quotas aren't being enabled when the server reboots?

Does it work properly when manually enabling quotas after the server is online?

Hi Eric,

That seems to be the case.

To answer your question, I just tried to run a manual back-up when logged in as one of the users whose back-ups are failing (NOT LOGGED IN AS ROOT), and I was surprised by the output:

Starting backup of 143 domains to local file /home/THISUSER/virtualmin-backup/backup/daily_%Y%m%d ..
Creating backup for virtual server EXAMPLE.org.zm ..
.. failed to create temporary backup directory /home/OTHERUSER/.backup : mkdir: cannot create directory '/home/OTHERUSER/.backup': Permission denied

.. completed in 1 seconds

Creating backup for virtual server EXAMPLE.com ..
.. failed to create temporary backup directory /home/ANOTHERUSER/.backup : mkdir: cannot create directory '/home/ANOTHERUSER/.backup': Permission denied

.. completed in 0 seconds


... AND SO ON FOR ANOTHER DOZEN OR SO DOMAINS UNTIL ...


Creating backup for virtual server EXAMPLE.ca ..
Copying virtual server configuration ..
.. done
Backing up Cron jobs ..
.. none defined.

Copying records in DNS domain ..
.. done

Saving mail aliases ..
.. done

Saving mail and FTP users ..
.. done

Backing up mail and FTP user Cron jobs ..
.. none to backup

Copying Apache virtual host configuration ..
.. done

Copying Apache log files ..
.. done

Copying Webalizer configuration files ..
.. done

Copying SSL Apache virtual host configuration and certificate ..
.. done

Copying Logrotate configuration ..
.. done

Dumping MySQL database bcpsaca ..
.. done

Backing up Webmin ACL files ..
.. done

Backing up AWstats configuration file ..
.. done

Creating TAR file of home directory ..
.. TAR failed!

sh: /home/THISUSER/virtualmin-backup/backup/daily_20170913/EXAMPLE.ca.tar.gz: Permission denied

gzip: stdout: Broken pipe
/bin/tar: -: Cannot write: Broken pipe
/bin/tar: Error is not recoverable: exiting now
Backup failed! See the progress output above for the reason why.

So at least now I am not getting "setquota" errors, but I seem to have three problems now, two new:

  1. Why does running an ordinary user's scheduled back-up cause it to try and back up every domain on the server?!
  2. Why did one domain partially succeed (even though nothing was successfully written to this user's home directory)?

And I can't tell whether or not the setquota problem is fixed. Another scheduled back-up will run in a little over eight hours, and I'll know then.

So, umm, one step forward, two steps back.

Craig

For this failed backup, excactly which page in Virtualmin did you do it on, and how did you get to that page? I'm interested to know if this was a scheduled backup that you re-executed, or a one-off.

Yes, started after a reboot. I cannot backup now.

I'm interested to know if this was a scheduled backup that you re-executed, or a one-off.<<

I schedule using cron(1) and crontab(5). I have tried manually. Both failing now.

    Destination     Backup contents     Started at      Started from   
local file /root/B/backups4     All virtual servers     10/Aug/2017 04:30   Scheduled backup

Odd. this is my Thursday backup from August 10th! Still running??

Hi Jamie,

As a regular user (the URLs in brackets [which is what I assume you mean by ".. excactly which page in Virtualmin did you do it on ..."] are the addresses of the main frame that displays what is described to the left):

  1. Log in.
  2. Navigate to Backup and Restore | Scheduled Backups.
  3. Click the green "Backup" button to the right of the saved and scheduled back-up. ( https://cp.mydomain.net:10000/virtual-server/list_sched.cgi )
  4. Leave all settings in place. "All virtual servers" is selected, and only the user's domains are shown in the greyed-out domain list. ( https://cp.mydomain.net:10000/virtual-server/backup_form.cgi?oneoff=1504... )
  5. Click the "Backup Now" button.

I just did it again, and got the same output as yesterday, an attempt to back-up all domains on the server.

(Note that in this example, "mydomain.net" is my company domain on which there is a wildcard SSL certificate. And I am using the Authentic theme.)

On a now-separate note, the scheduled back-up for this user -- the exact same scheduled back-up described above that tried to back-up everything on the server -- succeeded last night.

Hope that helps.

Craig

Thanks - that's exactly the information I was looking for. Turns out there is a Virtualmin bug that happens when you login as root and run a schedule backup created by a domain owner - instead of it just backing up his domains, it backs up all of them!

This will be fixed in the next release.

Status: Active » Fixed

OK, thanks, but I emphasise that I was not logged in as root in the above example; the process I described above was when logging in as an ordinary user from the log-in screen. It is not the same as the situation from July described at https://www.virtualmin.com/comment/779865#comment-779865, which I have not tested in a while. If they'll both be fixed, that's great.

However, I'd like to go back to the original cause of the problem that prompted me to open this ticket: quotas not being enabled on boot. Any idea why that might be happening?

I checked, and yes the same bug could apply in that situation was well :-(

Regarding the original issue - is there any way to see the boot messages from your system, like what you'd see at the console? The output from the script that enables quotas may explain why they aren't being activated.

"Thanks - that's exactly the information I was looking for. Turns out there is a Virtualmin bug that happens when you login as root and run a schedule backup created by a domain owner - instead of it just backing up his domains, it backs up all of them!

This will be fixed in the next release."

But nothing had change. All the backups are run as root. I never backup as a user. So what happened when if failed? And messing with quote seemed to fix it. I do a daily backup of all domains. The results appear in the Virtualmin console.

I'm confused now - back in comment #6, you mention "running an ordinary user's backup". The bug is triggered when a backup created by a non-root user is run as root.

Are these distinct bugs?

1) Quota issue.

2) User backups run as root.

Hi Jamie,

There seems to be some confusion here. I am not "LoopZilla". Everything I wrote in comment #6 was and still is factual.

As for boot messages, will copying and pasting the output of "dmesg" give you the information you need?

Craig

And actually, there still seems to be some confusion about the circumstances under which Virtualmin tries to back up all domains on the server. The bug I described in #6, to emphasise for the third time, is NOT when root runs a scheduled back-up created by a non-root user. It is when a regular user -- e.g., "bob" -- performs a clean log-in from the Virtualmin log-in screen and runs his own scheduled back-up that is configured to back-up all domains in his account. This is a regular user who should not have access to (or even know about) all of the other domains on the server, and when he follows the procedure in comment #11, Virtualmin tries to back up every single domain on the server, not just that user's domains. Root is not logged in anywhere else in the browser. Root is logged out. Root has left the building. I just confirmed this for a third time a few minutes ago.

But the same weirdness happens -- successfully this time -- when root does try to manually run a regular user's scheduled back-up that is configured to back up all domains in the user's account. It's weird enough that I don't really want to try it right now ... but against my better judgement I just have, which was really stupid on a production server that was two percentage points away from hitting the threshold to have more disk space added to it, as the back-up then consumed the remaining free space on the server:

cat: write error: No space left on device

gzip: stdout: Broken pipe
/bin/tar: -: Cannot write: Broken pipe
/bin/tar: Error is not recoverable: exiting now

Backup failed! See the progress output above for the reason why.

Once it had failed -- because there does not seem to be a way to halt a back-up in progress -- I deleted the whole back-up directory from the user's account, which was now way over quota.

Right, it's the same bug that causes a non-root user's backup to attempt to include all domains, OR if the root user run's a non-root user's backup. Both of these situations will be fixed in the upcoming 6.01 release.

OK, that's good. Thanks.

Do you have an answer to the "dmesg" question?

No, the dmesg output won't help as it contains only kernel messages.

However, the contents of /var/log/messages starting at boot time maybe useful.

Does this help?:

[23:52:53 root@cp31 20170927_boot_log]# grep -C4 -i quota boot_log_20170907.txt
Sep  7 02:46:12 cp31 kernel: pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
Sep  7 02:46:12 cp31 kernel: PTP clock support registered
Sep  7 02:46:12 cp31 kernel: PCI: Using ACPI for IRQ routing
Sep  7 02:46:12 cp31 kernel: clocksource: Switched to clocksource kvm-clock
Sep  7 02:46:12 cp31 kernel: VFS: Disk quotas dquot_6.6.0
Sep  7 02:46:12 cp31 kernel: VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Sep  7 02:46:12 cp31 kernel: FS-Cache: Loaded
Sep  7 02:46:12 cp31 kernel: CacheFiles: Loaded
Sep  7 02:46:12 cp31 kernel: pnp: PnP ACPI init
--
Sep  7 02:46:12 cp31 systemd[1]: Reached target Remote File Systems.
Sep  7 02:46:12 cp31 systemd[1]: Starting Remote File Systems.
Sep  7 02:46:12 cp31 systemd[1]: Reached target Encrypted Volumes.
Sep  7 02:46:12 cp31 systemd[1]: Starting Encrypted Volumes.
Sep  7 02:46:12 cp31 kernel: EXT4-fs (sda): re-mounted. Opts: grpquota,errors=remount-ro,usrquota,data=ordered
Sep  7 02:46:12 cp31 journal: Journal started
Sep  7 02:46:12 cp31 systemd-fsck: /dev/sda: clean, 673061/6258688 files, 15863058/25034752 blocks
Sep  7 02:46:12 cp31 systemd: Starting Flush Journal to Persistent Storage...
Sep  7 02:46:12 cp31 systemd: Started Load/Save Random Seed.
Sep  7 02:46:12 cp31 systemd-udevd: starting version 219
Sep  7 02:46:12 cp31 systemd: Started File System Quota Check.
Sep  7 02:46:12 cp31 systemd: Starting Enable File System Quotas...
Sep  7 02:46:12 cp31 systemd: Started udev Kernel Device Manager.
Sep  7 02:46:12 cp31 quotaon: quotaon: Cannot stat() mounted device /dev/root: No such file or directory
Sep  7 02:46:12 cp31 systemd: Started Enable File System Quotas.
Sep  7 02:46:12 cp31 journal: Runtime journal is using 8.0M (max allowed 600.6M, trying to leave 900.9M free of 5.8G available ? current limit 600.6M).
Sep  7 02:46:12 cp31 systemd: Started Flush Journal to Persistent Storage.
Sep  7 02:46:12 cp31 systemd: Started Configure read-only root support.
Sep  7 02:46:12 cp31 systemd: Reached target Local File Systems.
[23:54:02 root@cp31 20170927_boot_log]#

I see an obvious error there (/dev/root is the main drive), but is this something I'll need the data centre to look at?

The message quotaon: Cannot stat() mounted device /dev/root: No such file or directory seems like the cause. You may need to ask your data center about this, especially if you're running a custom linux distribution that they provided.

Thanks. I'll let you know how it goes.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

I had this issue, There's a command that specifically checks for damaged quota files and repairs them if necessary. I usually use that after e.g. copying /home to a new partition:

quotacheck examines each filesystem, builds a table of current disk usage, and compares this table against that recorded in the disk quota file for the filesystem (this step is omitted if option -c is specified). If any inconsistencies are detected, both the quota file and the current system copy of the incorrect quotas are updated (the latter only occurs if an active filesystem is checked which is not advised).

Try this:

quotaoff / quotacheck -m / quotaon /

"-m" causes quotacheck to not try to remount the filesystem read-only, which it normally does to prevent processes from writing to it while it counts disk usage. You can't remount / read-only though while the system is booted.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.