Backup fails when using SSH and "strftime-style time substitutions"

Submitted by bcw on Sun, 04/17/2016 - 06:26

When using strftime-style time substitutions over SSH the backups fail. My guess from the backup log is that there is a missed substitution in the script as it would appear that the ls command does not have the substitution applied:

Deleting backups from /data/backups/%Y/%m/%d on SSH server [ADDRESS] older than 90 days ..
    .. failed to list directory via SSH : backup@[ADDRESS]'s password:
ls: cannot access /data/backups/%Y/%m: No such file or directory

I should not that the upload works fine and that the folders /data/backups/2016/[01 through 04] exist so ls should not be failing on trying to find /data/backups/2016/01 for example.

Status: 
Closed (fixed)

Comments

Submitted by andreychek on Sun, 04/17/2016 - 09:05

Howdy -- if you look in Backup and Restore -> Scheduled Backups -> SCHEDULE_NAME, is the "Do strftime-style time substitutions on file or directory name " option checked there?

Submitted by bcw on Tue, 04/19/2016 - 04:47

Yes, the current settings are:

Destination: Backup via SSH to [HOST ADDRESS] /data/backups/%Y/%m/%d with user [USER], password [PASS] and delete old backups after 90 days.

Additional destination options: Do strftime-style time substitutions; Transfer each virtual machine after it is backed up

Backup Format: One file per server; Create destination directory

Action on error: Halt the backup immediately

Backup Level: Incremental

Submitted by JamieCameron on Tue, 04/19/2016 - 22:50

Do you perhaps have multiple destinations configured for this backup, and if so is the strftime option checked for all of them?

Submitted by bcw on Wed, 06/01/2016 - 09:05

Apologies for the late reply, I'm incredibly short on time at the moment and as a temporary measure I've set up my own backup for the whole disk.

I've also modified the configuration to see if there is any bearing on when the backup is done, how often the backup is done etc.

There are now multiple backups (list below).

The backup structure is:
@00:00 Daily run incremental which ssh's to /data/backups/D/%u-%A/[incremental backup per virtual server] delete older than 7 days
@01:00 Weekly on Monday run full which ssh's to /data/backups/W/%y/%m/%d/[full backup per virtual server] delete older than 90 days
@02:00 Monthly on 1st run full which ssh's to /data/backups/M/%Y/$m/[full backup per virtual server]

The only one that fails every time is the weekly one - the daily one seems fine. I could add a 3 year delete to the yearly backup but I see little point other than to see if the error also occurs there. The error is always the same:

Deleting backups from /data/backups/W/%Y/%m/%d on SSH server [address] older than 90 days ..
    .. failed to list directory via SSH : [user]@[address]'s password:
ls: cannot access /data/backups/W/%Y/%m: No such file or directory

Submitted by JamieCameron on Wed, 06/01/2016 - 23:25

Should the %y actually be %Y ?

Submitted by bcw on Mon, 06/06/2016 - 03:37

That was a typo in my write up. The quote below it shows it as a %Y (capital)

Submitted by JamieCameron on Mon, 06/06/2016 - 21:10

I think I see the problem - can you changing the backup path to be like /data/backups/W/%Y-%m-%d ?

Submitted by bcw on Fri, 06/10/2016 - 06:18

That seems to have made the difference, at least a manual backup worked when using dashes.

Will there be a fix implemented to allow directory based strftime substitutions?

Submitted by JamieCameron on Sat, 06/11/2016 - 00:52

It's tricky, because when subdirectories are allowed we may need to delete multiple parent directories as part of the cleanup process.

Submitted by ckwsp101 on Mon, 01/02/2017 - 09:51

On Ubuntu 14.04.5 Webmin version 1.830
Virtualmin version 5.05.gpl

The suggested backup path did not work. This is a known issue?

-- 1 servers backed up successfully, 0 had errors.

Backup is complete. Final size was 239.52 kB.

Deleting backups from full/%Y-%m-%d on SSH server {sshserverhost} older than 7 days ..

.. failed to list directory via SSH : ssh: Could not resolve hostname {ssh login name}: Name or service not known

Submitted by andreychek on Mon, 01/02/2017 - 10:06

Howdy -- it looks like you may be seeing a DNS issue there... Virtualmin attempted to get a directory listing, but instead received a DNS error.

You may want to double-check that the hostname is spelled correctly. If it is, check your DNS settings in /etc/resolv.conf, and also verify that you can SSH to your backup host from your Virtualmin server.

Submitted by ckwsp101 on Mon, 01/02/2017 - 10:22

If I set delete old backups Never, the data file was successfully sent to the ssh server.

There is different dns requirement when performing the directory listing?

Submitted by ckwsp101 on Mon, 01/02/2017 - 10:37

The archive reached ssh server. Why does the message say it used my {ssh login name} as a hostname lookup. I would expect the code to use {ssh server name}

Uploading archive to SSH server {ssh server hostname} .. .. done

.. completed in 2 seconds

1 servers backed up successfully, 0 had errors.

Backup is complete. Final size was 239.54 kB.

Deleting backups from full/%Y-%m-%d on SSH server {ssh server hostname} older than 7 days ..

.. failed to list directory via SSH : ssh: Could not resolve hostname {ssh login name}: Name or service not known

Submitted by JamieCameron on Mon, 01/02/2017 - 14:19

Does your username or password for the SSH server perhaps contain the % character?

Submitted by ckwsp101 on Mon, 01/02/2017 - 16:25

There is a shared substring in the login name and password. The {magicstring} does not contain the % character.

ssh user account login name {magicstring}bbbbb all lower case. ssh password: b{magicstring}b

Your suggesting choosing different ssh account login name and password string?

Short terrm go with Delete old backups set to Never

Submitted by JamieCameron on Mon, 01/02/2017 - 17:28

Can you run the command virtualmin list-scheduled-backups and post the full backup URL here? It should be something starting with ssh://

Submitted by ckwsp101 on Mon, 01/02/2017 - 18:26

I will try tomorow. I will create a test case with no customer info and will report back.

The login and passwords fields are transposed on live site. Here is my hand edited posting removing live data.

Destination: ssh://sshloginame : password@sshservername.dyndns.info:full/%Y-%m-%d

Submitted by JamieCameron on Mon, 01/02/2017 - 19:30

You might want to try changing the destination path to an absolute directory (starting with /)

Submitted by ckwsp101 on Tue, 01/03/2017 - 11:44

I just noticed the authentic theme allows a trailing space to be part of the ssh login account username field. I removed the trailing blank and the error went away.

Thank you for taking time to look into this issue.

full and relative destination paths also worked

Submitted by JamieCameron on Tue, 01/03/2017 - 17:36

Ah ... that explains it. I will fix Virtualmin in the next release to prevent this.

Submitted by JamieCameron on Tue, 01/03/2017 - 17:36

Status: Active » Fixed