Automatic restore by cron script almost 100% usage of CPU

Submitted by skpsi on Mon, 09/26/2016 - 08:40 Pro Licensee

Hi . I created automatic restoring very simple script like:

***********
#!/bin/bash
/usr/sbin/virtualmin restore-domain --all-features  --all-domains --source  /home/backups/server/backup_from_master_server.tar.gz
*********

i add into cron job with root permission to executed it . Cron job starts every 8am 10 am 12 am 2 pm 4pm 6pm 8 pm 10 pm . File backup_from_master_server.tar.gz isnt big - about 3GB .

I notice that when script is executed it's stays in processes and do i dont know what because when i go into restored domain i have fresh data what is expected and CPU is near 90% with gzip process :

12717 root 99.0 % gzip -d -c
12548 root 98.6 % gzip -d -c
12070 root 24.9 % gzip -d -c
12509 root 22.0 % gzip -d -c
12501 root 7.3 % /bin/tar xvfX - /home/temp/238214_10327_5_restore-domain.pl
12071 root 6.2 % /bin/tar xf -
12510 root 5.6 % /bin/tar xf -
12549 root 4.2 % /bin/tar tf -
12718 root 4.1 % /bin/tar tf -
12716 root 4.0 % cat /home/backups/server//backup_from_master_server.tar.gz
12547 root 3.9 % cat /home/backups/server//backup_from_master_server.tar.gz
12719 root 3.2 % [kworker/u16:4]
12747 root 2.6 % /usr/share/webmin/proc/index_cpu.cgi
12499 root 2.5 % cat /home/temp/103890_10327_3_restore-domain.pl/restore_domain_dir
12514 root 2.4 % /usr/share/webmin/virtual-server/restore-domain.pl
30244 root 2.3 % [kworker/u16:2]
9544 root 2.1 % [kworker/u16:3]
11540 root 2.0 % [kworker/u16:0]
12500 root 1.5 % cat
28330 root 1.3 % [kworker/u16:1]
12069 root 0.8 % cat /home/backups/server//backup_from_master_server.tar.gz
12508 root 0.8 % cat /home/backups/server//backup_from_master_server.tar.gz
12075 root 0.7 % /usr/share/webmin/virtual-server/restore-domain.pl
12720 root 0.5 % sshd: root [priv]
12036 root 0.4 % /usr/share/webmin/virtual-server/restore-domain.pl
58 root 0.3 % [kswapd0]
11958 root 0.3 % /usr/share/webmin/virtual-server/restore-domain.pl
10327 root 0.2 % /usr/share/webmin/virtual-server/restore-domain.pl
11272 root 0.2 % /usr/share/webmin/virtual-server/restore-domain.pl
11515 root 0.2 % /usr/share/webmin/virtual-server/restore-domain.pl
10114 root 0.1 % /usr/share/webmin/virtual-server/restore-domain.pl
10796 root 0.1 % /usr/share/webmin/virtual-server/restore-domain.pl

That is not good behavior as it expected whit cron job

Status: 
Active

Comments

Submitted by andreychek on Mon, 09/26/2016 - 09:30

Howdy -- hmm, can you show us the entry from cron that root has for that script?

For example, does it show up when you run "crontab -l"?

Submitted by skpsi on Mon, 09/26/2016 - 12:36 Pro Licensee

For example no i have running 10 restore-domain.pl in processes .

root@ws2:~# crontab -l
@daily /etc/webmin/backup-config/backup.pl 147214122417276
0 4 * * * /root/restore_ws1.sh > /root/restore.log 2>&1 #resotring ws1 z lokalnych zasobów z bieżacą datą
* 7,9,11,13,15,17,19,21,23 * * * /root/restore_client1.sh > /root/restore_client1.log 2>&1
(...)
awstats and webanalizer entry

and log restore_client1.log with lastes modify date 19:33 26 Sep

Checking for missing features ..
.. all features in backup are supported

Checking for errors in backup ..
.. no errors found

Starting restore..
Extracting backup archive file ..
.. done

Submitted by andreychek on Mon, 09/26/2016 - 16:07

How long does it take to perform a backup of those particular domains?

I'm wondering if the backups are "stacking up" during the restore process... are they perhaps particularly large accounts?

Submitted by JamieCameron on Mon, 09/26/2016 - 18:03

Also, running multiple backups or restores at the same time is probably counter-productive. You should run them in series rather than parallel.

Submitted by skpsi on Tue, 09/27/2016 - 01:58 Pro Licensee

Same backup takes about 5-7 minutes on master server then file is copy to slave server and wait till next full hour and restoring takes about 30min . So there is no chance that restore file backup_from_master_server.tar.gz was overwrited during restoring process

Submitted by JamieCameron on Tue, 09/27/2016 - 23:24

By the way, what are you trying to do here? Are you just taking backups, or are you use the backup and restore process to replicate domains between machines?

Submitted by skpsi on Wed, 09/28/2016 - 07:53 Pro Licensee

Ok, Backup domain are perform on one machine and send to another via ssh and then restore by my script pasted on first post . Now croin job are disabled ( from yesterday ) and in processes i have a least 3 processes restore-domain.pl.

Submitted by JamieCameron on Wed, 09/28/2016 - 16:31

Maybe there are previous restores still running from the day before? If you kill them and re-try the sync, does it help?

Also, you should look into the transfer-domain API command, as it can do a backup and remote restore in a single operation.

Submitted by skpsi on Thu, 09/29/2016 - 02:51 Pro Licensee

Now i dont have any restoring process. I try use transfer-domain i see how it works. But why restore-domain processes remain and consumption of cpu is near 100% that is mystery