Backup fails

Previously backups were set to use FTP and delete every 14 days and it worked. It deleted old backups and uploaded new. However when I changed to SSH and set delete every 3 days it does not delete the old backups and I get errors.

Status: 
Closed (fixed)

Comments

Howdy -- can you attach a screenshot of your scheduled backup screen that isn't working properly? Thanks!

Thanks for your image!

Last night, Virtualmin 3.97-2 was pushed out, which corrects some bugs that were discovered.

Can you try upgrading to that, and let us know if that helps with the problem you're having?

If not -- you mentioned that you were getting errors during the backup/delete, which errors are you seeing? We can use your screenshot and those errors to troubleshoot the problem you're having. Thanks!

The issue is that Virtualmin does not delete old backups on an SSH host. I specified 3 days but it does not delete after 3 days and then the backup system gets full (10GB disk) and then I get errors from no space available.

On my previous system Virtualmin did delete old backups on an FTP host.

Just to clarify, are you using the Virtualmin version that just went out, version 3.97-2?

You can determine that with this command:

dpkg -l 'webmin-virtual-server'

Same here, Virtualmin 3.97.gpl and 3.97.pro on Debian 6. Backup over SSH is broken, because target directory names will be escaped unnecessarily. That leads to "directory doesn't exist" while copying the file.

I do not see 3.97-2 available anywhere. Is it in the repositories? Regards,

Christian

@datenimperator - 3.97-2 should be available now, and fixes this issue (along with a couple of others).

Are you using the GPL or pro version, and on which Linux distribution?

Both GPL and Pro (on different systems), all on Debian 6. Target system for backup is Ubuntu 10 LTS.

I've just ran apt-get update && apt-get dist-upgrade but no update was available from neither download.webmin.com nor download.virtualmin.com

Hm, strange. The deb package is installed, but Virtualmin still shows its version as 3.97.gpl instead of 3.97.gpl-2. Backup still fails.

root@v2:~# dpkg -l | grep webmin-virtual-server
ii  webmin-virtual-server           3.97.gpl-2                   Webmin module for 'Virtualmin Virtual Servers (GPL)'
ii  webmin-virtual-server-mobile    2.5                          Webmin theme 'Virtualmin Mobile Theme'
ii  webmin-virtual-server-theme     8.6                          Webmin theme 'Virtualmin Framed Theme'

… but

root@v2:~# virtualmin info host
host:
    hostname: v2.software-consultant.net
    module root: /usr/share/webmin/virtual-server
    os: Debian Linux 6.0
    root: /usr/share/webmin
    theme version: 8.6
    virtualmin version: 3.97.gpl
    webmin version: 1.610

Virtualmin doesn't actually report it's version with the "-2" -- you can only determine that by looking at the package that's installed.

So, even though you have webmin-virtual-server version 3.97.gpl-2 installed, when looking at the version in Virtualmin, it will show just 3.97.

Are you continuing to see the same problem after upgrading to the new Virtualmin version?

@datenimperator - what is the exact path you are backing up to?

It backs up to a SSH server. This is the backup job that failed just some hours ago:

134253772113869
    Domains: Except domain.info
    Include sub-servers: Yes
    Destination: ssh://user:password@host.dyndns.org:backup/%Y-%m-%d
    Delete old backups after: 14 days
    Features: All
    Format: One file per server
    Incremental: Yes
    Enabled: Yes
    Cron schedule: 2 2 * * 1-6
    Send email to: user@domain.com
    Send email: Only on failure
    Notify domain owners: No

It's set to "create target directory", but that looks weird, too (notice the backslashes in the directory names):

root@backup:/home/home/homes/v2# ls -lR backup\\
backup\:
insgesamt 8
drwxr-xr-x 2 v2.home home 4096 18. Dez 02:02 2012\-12\-18
drwxr-xr-x 2 v2.home home 4096 19. Dez 02:02 2012\-12\-19

backup\/2012\-12\-18:
insgesamt 0

backup\/2012\-12\-19:
insgesamt 0

The log contains:

Backup failed! See the progress output above for the reason why. Total backup time was 00 minutes, 22 seconds.

Sent by Virtualmin at: https://virtualmin.host:10000

Creating backup for virtual server next-hr.com ..
   Copying virtual server configuration ..
   .. done

   Backing up Cron jobs ..
   .. none defined.

   Copying records in DNS domain ..
   .. done

   Saving mail aliases ..
   .. done

   Saving mail and FTP users ..
   .. done

   Backing up mail and FTP user Cron jobs ..
   .. none to backup

   Copying Apache virtual host configuration ..
   .. done

   Copying Apache log files ..
   .. done

   Copying Logrotate configuration ..
   .. done

   Creating incremental TAR file of home directory ..
   .. done

   Uploading archive to SSH server raq550.dyndns.org ..
   .. upload failed! user@host.dyndns.org's password:
scp: backup/2012-12-19/next-hr.com.tar.gz.info: Not a directory


.. completed in 2 seconds

I suspect that this might be triggered by using a destination path that doesn't start with /

If you change your backup destination to a full path, do you still see this problem?

By the way, I tested this with Virtualmin 3.97-2 and a relative path, but still couldn't re-produce the problem.

I changed it to /tmp/backup/… to try it and it would give me the same error. What puzzles me is the fact that 5 Virtualmin systems (both GPL and Pro) stopped their backup on the same day after working flawlessly for months. I did not change anything on the target system. What I did was upgrading webmin/virtualmin on the source systems when upgrade were available.

As a developer, I'd try to isolate the source problem with the escaped path. Looks as if a path string would have turned into a regex by escaping all dashes and slashes and whatnot. Did something change recently with regards to that?

Does the destination SSH user have an scp-only shell? That could impact how the destination directory is created ..

Yes, all destination users were scp-only, which worked before.

I've changed it to a full-featured shell but the error still persists. Can I increase the log detail by any means to help you understand what's going on?

All my SSH backups on all webmin-based systems stopped working , so this is getting more urgent with every day.

If possible, I'd be happy to login to your system myself to see what is going wrong here.

Source or destination? Please send your SSH key.

I'd need access to the source (Virtualmin) system, as that is where the bug most likely is.

Email me directly at jcameron@virtualmin.com and I'll reply with my SSH key.

I am having the exact same problem Virtualmin GPL 3.97. When I try to backup over SSH with SCP only user I get this error: Uploading archive to SSH server dev.hype.ro .. .. upload failed! webhosting@dev.hype.ro's password: scp: /backupservers/web.hype.ro_backup/16-01-2013/active.hype.ro.zip.info: Not a directory

I started to happen after the virtualmin upgrade no other info changed to the backup setup.

@daniel - does the destination SSH user have an scponly shell, or does he have a regular shell?

it is scponly shell on the destination. from filezilla or an other SCP client the upload of an file works without problems.

Unfortunately, the scponly shell would cause the problems you're seeing.

Although the scponly shell can work from filezilla or other clients designed for SCP, Virtualmin uses functionality that doesn't work properly with the scponly shell.

Try changing the shell to bash or similar to see if your backups begin working at that point.

I haven't tested this in awhile, but if you're interested in using a restricted shell for your backup user, I believe using lshell would work. You can find details of it here:

http://lshell.ghantoos.org/

I just found a bug in Virtualmin that is triggered when the scponly shell is in use, and you are backing up to a directory that doesn't exist yet. This will be fixed in the upcoming 3.98 release.

Ok I have changed the user shell to bash from scponly and the problem is that the error remains the same.

Make sure you delete the file that was incorrectly created instead of the destination directory.

I am on 3.97 and the backup fails when using ~/dom/theduck/%Y%m%d-%H%M%S because it won't create the directory. This started with 3.97.

If I switch back to "One file per server (old format)" then it works fine.

Still having an issue with Virtualmin not deleting old backups. I always have to manually delete old backups every 3 days. (I have a 10GB backup and full backups are around 2.5GB a day).

It used to work fine (Deleting old backups) using an FTP host. Using an SSH host it does not delete old backups even though I have specified 2 or 3 days.

@cgdhosting - not sure if I asked this already, but does the user you are backing up to via SSH have an scponly shell? That will prevent the listing of directories needed to purge old backups.

It uses the user root for backups. Root has a full shell as expected.

root:x:0:0:root:/root:/bin/bash

Could you post your backup output, or email it to me at jcameron@virtualmin.com ?

I'm mainly interested in the section near the end where it mentions purging old backups.

I have emailed you.

Regards,

Curtis

Thanks. In cases where the backup failed, no purging of old backups will be done, so that you don't end up in a situation where all your valid backups have been lost.

However, for those that succeeded, purging should have worked.

If you run ls -l /home/curtis/backups on your remote server via SSH as the same user you use for backups, what does it output?

I changed it to delete after one day and I got this output:

Deleting backups from /home/curtis/backups/%d-%m-%Y on SSH server ns4.cgdhosting.com older than 1 days .. Deleting file /home/curtis/backups/24-01-2013 via SSH, which is 1 days old .. .. deleted 4 kB.

Shouldn't it be the full size and not just 4kB?

Was the directory actually deleted?

That incorrect size could be reported if Virtualmin couldn't run the command du -sk /home/curtis/backups/24-01-2013 on the remote host to get the full directory size..

Deleting after one day failed at the same as previously (On the fourth backup, 2.5GB average size x 4 = 10GB which is the VPS disk space)

I deleted all the backups manually and the scheduled backup is in progress and the command you mentioned appears to work.

root@ns4:/home/curtis/backups# du -sk /home/curtis/backups/31-01-2013/ 236004 /home/curtis/backups/31-01-2013/ root@ns4:/home/curtis/backups# du -sk /home/curtis/backups/31-01-2013/ 247440 /home/curtis/backups/31-01-2013/ root@ns4:/home/curtis/backups#

Ok, let me know if that scheduled backup completes OK or not.

It completed as expected. However it will fail again in 3 days because it won't delete old backups (It says deleted 4kb as before)

It looks like SSH commands to the backup destination by Virtualmin aren't actually working, causing the incorrect size to be displayed and deletion to fail.

Would it be possible for me to login to your system again to see what is going wrong?

Are there no logs I could send?

You could send the log file /var/log/messages from the destination system - that should contact debugging information from the SSH server.

/var/log/messages is empty on destination system.

I cannot see anything relevant in /var/log/syslog

Does it make any difference that the main system is Ubuntu 12.04 and the backup system is 10.04?

The Ubuntu version shouldn't matter.

Is there any log file under /var/log on the destination system that contains messages from the SSH server sshd ?

There are no relevant logs. The only thing I can find are invalid logins.

Another thing you could do is enable logging of commands run by Virtualmin, at Webmin -> Webmin Configuration -> Debugging Log File. Under "Events to log" check on "Commands executed ", and set "Debug log enabled? " to "Yes".

The kick off a backup in which the purging fails, and after it completes email the log file /var/webmin/webmin.debug to me at jcameron@virtualmin.com

Automatically closed -- issue fixed for 2 weeks with no activity.

Nothing interesting in /var/webmin/webmin.debug

However it now works after enabling debug.

Does it go back to failing again if you disable debugging?

This is now happening again. Old backups do not get deleted over SSH when backed up to my VPS. I can backup from another server to this same server over SSH, this is a dedicated server.

Backing up as root does not fix the issue.

Is the debugging still enabled on your server?

I did not change it. How do I change/check it again?