virtualmin commandline Not restoring home directories of virtual servers

Submitted by Jason at eROI on Wed, 03/15/2017 - 19:28 Pro Licensee

Hello,

I am using backup and restore to move to a new server, namely from ubuntu 12.04lts to 16.04 lts. I am running vitualmin 1.831 on both systems.

Restoring a virtual server at the command line fails to untar the public_html files. I have verified the public_html directory and its contents are present in the file.

Restoring via gui works. I need this to be scriptable.

I have created a backup file with the following command line:

     virtualmin backup-domain --dest $DESTINATIONDIR --newformat --all-features --domain $i

restoring using the following command line is failing:

     virtualmin restore-domain --source /home/testrestore  --all-features --domain taco.eroi5.eroidev.com --skip-warnings 2>&1 |tee cli_restore.txt

The apparent apache error at the end of the enclosed messages isn't the problem here: that's a separate issue. I will file a separate issue. Note the "skip-warnings" directive above.

The contents of "cli_restore.txt" are the output messages from running the above restore command line.

The gui output is in the "gui_restore.txt" file.

Of note is that the gui_restore has the following additional lines in it:

Extracting TAR file of home directory ..
.. done

Setting ownership of home directory ..
.. done
Files: 
Plain text icon cli_restore.txt
Plain text icon gui_restore.txt
Plain text icon new_curlout.txt
Plain text icon short-miniserv.error_.txt
Plain text icon short-miniserv.log_.txt
Plain text icon short-webmin.log_.txt
Status: 
Closed (works as designed)

Comments

Submitted by JamieCameron on Thu, 03/16/2017 - 00:58

That's very odd - do either of the old or new systems have the /home directory on NFS? Or use a custom location for home dirs that is outside of /home ?

Submitted by Jason at eROI on Thu, 03/16/2017 - 13:52 Pro Licensee

Hello,

No, neither. The first couple of times I was restoring from a file located on an nfs share, but I moved it local with the same effects.

I tried to see if it made a difference whether it was a subserver or not, but no docroot files are being restored.

--jason

Submitted by Jason at eROI on Thu, 03/16/2017 - 18:30 Pro Licensee

Hello,

Oh, a bit of new information, Sorry I didn't catch this...It's not the document roots that are not being restored: it's the entire home directory of the virtual server. Certain boilerplate is being created such as the fcgi-bin directories, etc which led me to believe the wrong thing.

Some other new information: I built a curl session to mimic my firefox session. I copied the referrer, cookie, authorization headers and built a form to run the same restore. This session directly called the restore.cgi program, rather than using the cli version of "virtualmin restore-domain", or calling remote.cgi. it ALSO failed to restore the home directory.

I have uploaded a few more files:

new_curlout.txt is the html-stripped output of curl: note the missing sections regarding restoration of home directory...

short-miniserv.log.txt, short-miniserv.error.txt, and short-webmin.log.txt each contain the new entries as a result of running the curl command.

curl \
-k \
-v  \
-H'Referer: https://localhost:10001/virtual-server/restore.cgi' \
-H'Cookie: <a long cookie>'     \
-H'Authorization: Basic <a long string>' \
-F 'dom=taco.eroi5.eroidev.com' \
-F 'src=/mnt/eroione/restoring' \
-F 'only=0' \
-F 'dir_homes=1' \
-F 'delete_existing=0' \
-F 'reuid=1' \
-F 'feature_all=1' \
-F 'virt=0' \
-F 'fix=0' \
-F 'confirm=RestoreNow' \
-F 'skipwarnings=1' https://localhost:10001/virtual-server/restore.cgi  \
2>curlerr |tee curlout

Submitted by JamieCameron on Thu, 03/16/2017 - 23:49

Are you sure the backups contain the home directory contents? You can manually dump their contents with a command like :

tar tvzf /home/testrestore/domain.com.tar.gz

Submitted by Jason at eROI on Fri, 03/17/2017 - 12:05 Pro Licensee

Hello Jamie,

The contents of the home directory are indeed there. In addition, it was the same backup file I restored from using all three methods: cli, gui, and my faking it with curl.

Here is a (modified) output of tar -tzvf

./.bash_logout
./.stats-htpasswd
./logs
./logs/error_log
./logs/access_log
./cgi-bin
./cgi-bin/awstats.pl
./cgi-bin/plugins
./cgi-bin/lib
./cgi-bin/lang
./awstats
./.awstats-htpasswd
./dpkgout
./.virtualmin-src
./.profile
./etc
./etc/php5
./etc/php5/php.ini
./etc/php.ini
./may12
./.bashrc
./tmp
./public_html
./public_html/--a-bunch-of-files...
./homes
./.backup
./.backup/--domainname--_web_alog
./.backup/--domainname--_mysql
./.backup/--domainname--_virtualmin_template
./.backup/--domainname--_webalizer
./.backup/--domainname--_virtualmin
./.backup/--domainname--_mysql_wp_tacodev.gz
./.backup/--domainname--_web_elog
./.backup/--domainname--_virtualmin_plan
./.backup/--domainname--_webalizer_conf
./.backup/--domainname--_virtualmin_scripts
./.backup/--domainname--_logrotate
./.backup/--domainname--_web
./.backup/--domainname--_virtualmin-awstats
./.backup.lock
./fcgi-bin
./fcgi-bin/php5.fcgi

But, now that I think on it, try this one on for size: I am using a script to generate the backups. I will do a gui dump and compare the various restore procedures. Meanwhile, the pertinent line in the script is:

virtualmin backup-domain --dest $DESTINATIONDIR --newformat --all-features --domain $i

$DESTINATIONDIR is a valid directory with correct permissions, and $i contains the correct domainname.

I chose "--newformat" because the file format matches the "one file per-server" setting, and manually extracting the contents using tar most closely matched the output from the gui using the options: "only selected domain", "any plan", "backup all features", no virtualmin features selected, "local file or directory", "one file per-server", probably "halt immediately", and "full".

This happens on all of my domains, regardless of top-level or subserver type.

Submitted by Jason at eROI on Fri, 03/17/2017 - 16:40 Pro Licensee

Title: virtualmin commandline Not restoring document_roots of virtual servers » virtualmin commandline Not restoring home directories of virtual servers

Changed subject line to be accurate.

Submitted by JamieCameron on Fri, 03/17/2017 - 18:59

If you do a command-line restore, what's the exact command that you are using?

Also, can you post the output that you get from the restore command?

Submitted by Jason at eROI on Mon, 03/20/2017 - 16:59 Pro Licensee

Hello Jamie,

it's a script, but the command line itself is:

virtualmin restore-domain --source <restore-file-directory>  --all-features --continue-on-error --skip-warnings --domain <domain>

And restore output is a follows... The apache config failure is expected, and not part of this problem...

Checking for missing features ..
.. all features in backup are supported

Checking for errors in backup ..
.. no errors found

Starting restore..
Extracting backup archive files ..
.. done

Restoring backup for virtual server the2030group.com ..
    Restoring virtual server password, quota and other details ..
    .. done

    Updating administration password and quotas ..
    .. done

    Restoring Cron jobs ..
    .. done

    Restoring Apache virtual host configuration ..
    .. done

    Checking restored PHP execution mode ..
    .. mode FCGId OK for this system

    Updating home directory in PHP configuration ..
    .. done

    Restoring Apache log files ..
    .. done

    Restoring Webalizer configuration files and Cron job ..
    .. done

    Restoring allowed MySQL hosts ..
    .. done

    Deleting old MySQL databases ..
    .. done

    Re-loading MySQL database wp_2030 ..
        Creating MySQL database wp_2030 ..
        .. done

    .. done

    Restoring Webmin ACL files ..
    .. done

    Restoring AWstats configuration file ..
    .. done

    Updating Webmin user ..
    .. done

Applying web server configuration ..
.. failed! An Apache configuration error was detected : AH00526: Syntax error on line 37 of /etc/apache2/sites-enabled/the2030group.com.conf:
Invalid command 'php_admin_value', perhaps misspelled or defined by a module not included in the server configuration
Action 'configtest' failed.
The Apache error log may have more information.


Re-loading Webmin ..
.. done

Restore completed successfully.

Submitted by JamieCameron on Tue, 03/21/2017 - 01:28

Are you perhaps copying this backup to another system? If so, make sure you also copy the .dom and .info files from the same directory on the source system - these contain metadata that Virtualmin uses to determine the backup contents.

Submitted by Jason at eROI on Tue, 03/21/2017 - 14:35 Pro Licensee

Hello Jamie,

Yes I am. AND that's the problem: I wasn't copying the dom and info files.

Interestingly enough, the gui restores I was testing ALSO didn't have the dom and info files available...

I will play with this a bit more, and let you know if this is resolved.

Thanks for your time!

Submitted by Jason at eROI on Tue, 03/21/2017 - 19:39 Pro Licensee

Hello Jamie,

Go ahead and close this as "resolved", or is that up to me?

--jason

Submitted by Jason at eROI on Tue, 03/21/2017 - 19:40 Pro Licensee

Status: Active » Closed (works as designed)