Hello,
With the latest versions (3.95, 3.94) Virtualmin stopped deleting old backups. It used to work before! I am attaching a screenshot of the settings, which all seem correct.
Listing backup directories gives me
sftp> ls -la
drwxr-xr-x 4 u46155 u46155 4 Nov 5 11:58 .
drwxr-xr-x 3 u46155 u46155 3 Apr 21 2012 ..
drwxr-xr-x 2 u46155 u46155 59 Nov 2 05:54 02-11-2012
drwxr-xr-x 2 u46155 u46155 54 Nov 3 08:04 03-11-2012
My screenshot does not seem to appear as an attachment, but here it is: https://www.virtualmin.com/system/files/Edit%20Scheduled%20Backup_0.png
What output do you get in Virtualmin when you try to run this backup?
Also, make sure that the user you are connecting to the SSH server as doesn't have an
scponlyshell - this prevents Virtualmin from listing the destination directory.''
Thanks for coming back. I am not sure how to check whether virtualmin uses scponly for backups. I thought virtualmin runs with root?
Here is the full output of a test run. It fails to create the directory because the drive is already full. The drive has plenty of disk space, but due to the failed deletion it becomes full (100 GB drive, one backup is 25 GB)
Starting backup of 1 domains to /backup/%d-%m-%Y on FTP server xxx.xx.xx, local file /backup/daily/%d-%m-%Y ..
Creating backup for virtual server mygreatsite.com ..
Copying virtual server configuration ..
.. done
Backing up Cron jobs ..
.. none defined.
Copying records in DNS domain ..
.. done
Saving mail aliases ..
.. done
Saving mail and FTP users ..
.. done
Backing up mail and FTP user Cron jobs ..
.. none to backup
Backing up Dovecot control files ..
.. none found
Copying Apache virtual host configuration ..
.. done
Copying Apache log files ..
.. done
Copying Logrotate configuration ..
.. done
Dumping MySQL database booksbon_bonanza ..
.. done
Backing up Webmin ACL files ..
.. done
Creating TAR file of home directory ..
.. done
.. completed in 20 seconds
Uploading archive to FTP server xxxx.xxxxxxx.de ..
.. upload failed! STOR /backup/17-11-2012/mygreatsite.com.tar.gz failed : /backup/17-11-2012/mygreatsite.com.tar.gz: No such file or directory
Backup failed! See the progress output above for the reason why.
Before the drive became full I received errors like
Uploading archive to FTP xxx.xx-xxxxxx.de .... upload failed! Upload incomplete - file size is 7676658117, but sent 7039603712
It looks like that backup didn't complete at all .. which will prevent deletion of old backups from happening, as Virtualmin won't attempt to clean up old backups unless it knows that the most recent one was successful.
''
I think the backup completed since it says ". completed in 20 seconds" The ftp upload does not complete. I can see the backup in my local backup location (I have two, one locally, and the second one on the remote ftp)
The creation of the backup tar file completed, but it didn't get uploaded to the FTP server .. so the overall status was incomplete.
''
Yep, the reason is not completed is because the server is full, because the previous backups were not deleted.
So, the issue is the old backups are not deleted, and the server runs out of space. :)
Try manually deleting some old backups from the server to free up space, running the backup again in Virtualmin, and let us know what it outputs..
''
Yep, been doing that for a few weeks now. :)
I delete old backups, and the backups run again, but not deleted until the disk space runs out.
Tried completely wiping the drive, so no old folders are present, still old ones are not removed.
I'd be interested to see the full backup output when the remote drive isn't full, to see if it attempts to purge old backups, and if so what goes wrong.
''
Thanks, I enabled reporting for successful backups as well, and this time it seems to have worked. Not sure why. Here is the output:
One weird thing I noticed that it reported some directories having the same age in days, even though they have different dates , eg Deleting directory /backup/daily/11-11-2012, which is 8 days old .. Deleting directory /backup/daily/10-11-2012, which is 8 days old ..
.. completed in 1 minutes, 15 seconds
Uploading archive to FTP server xxxx.xxxxxxxxx.de ..
.. done
19 servers backed up successfully, 0 had errors.
Deleting backups from /backup/%d-%m-%Y on FTP server xxxx.xxxxxxxxxx.de older than 2 days ..
Deleting FTP file /backup/12-11-2012, which is 6 days old ..
.. deleted 27.33 GB.
Deleting FTP file /backup/14-11-2012, which is 4 days old ..
.. deleted 27.77 GB.
.. deleted 2 old backups
Deleting backups from local file /backup/daily/%d-%m-%Y older than 5 days ..
Deleting directory /backup/daily/13-11-2012, which is 6 days old ..
.. deleted 27.60 GB.
Deleting directory /backup/daily/09-11-2012, which is 10 days old ..
.. deleted 26.66 GB.
Deleting directory /backup/daily/11-11-2012, which is 8 days old ..
.. deleted 1.80 GB.
Deleting directory /backup/daily/12-11-2012, which is 6 days old ..
.. deleted 27.33 GB.
Deleting directory /backup/daily/10-11-2012, which is 8 days old ..
.. deleted 27.78 GB.
.. deleted 5 old backups
The number of days old it displays the backup as is based on the timestamp on the directory, not the file name. So if files were added to or deleted from the directory, the timestamp could change.
''
Ok thanks, will keep an eye on this and report back with some concrete workable material if it fails.
Deletions started failing again, and it seems the script is not recognizing / finding old backups, even though they are there. Here is the output from the backup command. In 1 or 2 days the remote sftp server will be full again:
Uploading archive to FTP server xxxx.xxxx-xxxxxxxx.de ..
.. done
19 servers backed up successfully, 0 had errors.
Deleting backups from /backup/%d-%m-%Y on FTP server xxxx.xxxxx-xxxxxx.de older than 2 days ..
.. no backups to delete were found
Deleting backups from local file /backup/daily/%d-%m-%Y older than 5 days ..
Deleting directory /backup/daily/15-11-2012, which is 6 days old ..
.. deleted 27.88 GB.
.. deleted 1 old backups
Could you post the listing of the directory /backup from the FTP server? Ideally obtained by using the
ftpcommand to connect to the FTP server. The issue may be that the dates as shown by FTP aren't in the format Virtualmin expects..''
Thanks, here is the list
sftp> dir19-11-2012 20-11-2012 21-11-2012 22-11-2012
or
sftp> ls -ladrwxr-xr-x 6 hidingmyftpuserid hidingmyftpuserid 6 Nov 21 22:56 .
drwxr-xr-x 3 hidingmyftpuserid hidingmyftpuserid 3 Apr 21 2012 ..
drwxr-xr-x 2 hidingmyftpuserid hidingmyftpuserid 59 Nov 19 04:10 19-11-2012
drwxr-xr-x 2 hidingmyftpuserid hidingmyftpuserid 59 Nov 20 08:10 20-11-2012
drwxr-xr-x 2 hidingmyftpuserid hidingmyftpuserid 59 Nov 21 06:09 21-11-2012
drwxr-xr-x 2 hidingmyftpuserid hidingmyftpuserid 2 Nov 21 22:56 22-11-2012
Thanks, that looks fine.
How many days do you have the backup configured to delete from the FTP server after though?
''
Thanks, 2 days.
Here is a screenshot https://www.virtualmin.com/system/files/Edit%20Scheduled%20Backup_0.png
In your comment from 2012-11-21 23:33 , it looks like you are using sftp to list the directory.
What if you use the regular
ftpcommand? In that case, the listing format is controlled by the server .. that's what I'm interested to see.''
Ah, ok. I didn't have ftp installed on my server, so after installing this is what I have:
ftp> dir227 Entering Passive Mode (xxx,xxx,xx,xxxx).
150 Opening ASCII mode data connection for file list
drwxr-xr-x 2 hidingmyftpuserid hidingmyftpuserid 59 Nov 19 04:10 19-11-2012
drwxr-xr-x 2 hidingmyftpuserid hidingmyftpuserid 59 Nov 20 08:10 20-11-2012
drwxr-xr-x 2 hidingmyftpuserid hidingmyftpuserid 59 Nov 21 06:09 21-11-2012
drwxr-xr-x 2 hidingmyftpuserid hidingmyftpuserid 36 Nov 22 07:29 22-11-2012
226 Transfer complete
Again, that looks fine..
You might want to wait a few days till Virtualmin 3.96 comes out, as I have fixed some bugs in this area .. it's possible that the new version will solve this problem.
''
Lovely! Much appreciated, look forward to that version. :)
Success!
Reporting that the remote files were deleted with Webmin 3.96! The backup failed though because the disk was full already (I was lazy to delete it), so the upload failed, but from tomorrow it will be working fine as there is enough space now...Many thanks.
Uploading archive to FTP server xxxxxxx.xxxxxxx.de ..
.. upload failed! STOR /backup/28-11-2012/mysitename-m.com.tar.gz failed : /backup/28-11-2012/nysitename-m.com.tar.gz: No such file or directory
Deleting backups from /backup/%d-%m-%Y on FTP server xxxxxx.xxxxxx.de older than 2 days ..
Deleting FTP file /backup/19-11-2012, which is 9 days old ..
.. deleted 28.49 GB.
Deleting FTP file /backup/20-11-2012, which is 8 days old ..
.. deleted 28.73 GB.
Deleting FTP file /backup/21-11-2012, which is 7 days old ..
.. deleted 28.96 GB.
Deleting FTP file /backup/22-11-2012, which is 6 days old ..
.. deleted 13.81 GB.
.. deleted 4 old backups