Backup logs stopped working

Submitted by responsiveny on Sun, 08/31/2014 - 19:29

Backups started failing due to reaching limit of backup processes (3). I did see these backup processes running. So I rebooted the VM server. Now, backups are working again but the logs are not updating in the GUI. How to troubleshoot backup logs? Thanks

Status: 
Closed (fixed)

Comments

Submitted by JamieCameron on Sun, 08/31/2014 - 19:53

Are you sure that some backups have completed? Only complete backups show up in the logs..

Submitted by responsiveny on Sun, 08/31/2014 - 19:55

I do see the files in the backup destination, but maybe the backup didn't complete? How can I tell if the backup process is still running and if it is, how to troubleshot why it's not completing.

Submitted by JamieCameron on Sun, 08/31/2014 - 21:55

Check if there is still a backup.pl process running on the system.

Submitted by responsiveny on Sun, 08/31/2014 - 22:00

No backup processes running.

Submitted by JamieCameron on Sun, 08/31/2014 - 22:40

Try running the backup manually (via the UI) and let me know if it completes or not. It may be that the backup is dieing part way though..

Submitted by responsiveny on Sun, 08/31/2014 - 23:46

The manual job completed when run from the GUI and it shows up in the "Backups Logs" section of the GUI.

Submitted by responsiveny on Sun, 08/31/2014 - 23:48

Now this is interesting. The manual backup shows as completed successfully in the log, but the process still shows in the process list:

/usr/libexec/webmin/virtual-server/backup.cgi

Submitted by responsiveny on Mon, 09/01/2014 - 09:25

A scheduled full backup ran last night, data seems to be there but no log in GUI. I can't be 100% sure that it completed successfully... is there another place to look for the log?

Submitted by JamieCameron on Mon, 09/01/2014 - 13:22

Check the cron log, if your system has one - it will be named like /var/log/cron

Submitted by responsiveny on Tue, 09/02/2014 - 13:13

The backups appear to have run last night because I see the backup files in the destination. However, nothing was added to the backup log in the GUI, so I cannot actually read the log.

in cron, there was this:

Sep  2 04:30:01 kickback crond[31053]: (root) CMD (/etc/webmin/virtual-server/backup.pl --id 13653903586283)

and there are no backup processes running now.

Submitted by JamieCameron on Tue, 09/02/2014 - 17:56

Can you try SSHing in as root and running the command /etc/webmin/virtual-server/backup.pl --id 13653903586283 , and see if it crashes?

Submitted by responsiveny on Tue, 09/02/2014 - 21:20

Is there supposed to be any output? It's running but no output to the console.

Submitted by JamieCameron on Tue, 09/02/2014 - 21:37

If it works, there should be no output.

If it fails, it should output some error message - please post that here.

Submitted by responsiveny on Tue, 09/02/2014 - 23:15

it's running for a couple of hours, no output and no files written yet. backup.pl is running in the console. I can keep it running and see what happens in a few more hours.

Submitted by responsiveny on Wed, 09/03/2014 - 00:49

got the error finally. This server was recently rebooted by the way.

[root@kickback ~]# /etc/webmin/virtual-server/backup.pl --id 13653903586283
 
 
Error: Failed to open /etc/webmin/virtual-server/backuplogs/1409722236-26417-1 for writing : Too many open files
Error
-----
Failed to open /etc/webmin/virtual-server/backuplogs/1409722236-26417-1 for writing : Too many open files

Submitted by responsiveny on Wed, 09/03/2014 - 00:50

finally got the error

[root@kickback ~]# /etc/webmin/virtual-server/backup.pl --id 13653903586283

Error: Failed to open /etc/webmin/virtual-server/backuplogs/1409722236-26417-1 for writing : Too many open files

Error

Failed to open /etc/webmin/virtual-server/backuplogs/1409722236-26417-1 for writing : Too many open files

Submitted by andreychek on Wed, 09/03/2014 - 09:27

What is the output of this command:

sysctl fs.file-max

Submitted by responsiveny on Wed, 09/03/2014 - 10:52

sysctl fs.file-max

fs.file-max = 391269

Submitted by JamieCameron on Wed, 09/03/2014 - 17:12

You could try increasing this with a command like sysctl -w fs.file-max=1000000

Submitted by responsiveny on Wed, 09/03/2014 - 18:05

I tried that, but this commend seems to show the limit on open files is 1024?

[root@kickback ~]# ulimit -Hn 1024

Submitted by responsiveny on Wed, 09/03/2014 - 18:06

We'll see if the backup logs work tonight.

Submitted by responsiveny on Thu, 09/04/2014 - 10:25

It appears that the backup ran but again no log in GUI

Submitted by JamieCameron on Thu, 09/04/2014 - 12:08

Does a manual run with /etc/webmin/virtual-server/backup.pl --id 13653903586283 still fail due to lack of file descriptors?

Submitted by responsiveny on Sun, 09/07/2014 - 22:16

The logs started working again. Maybe it was the max open files fix that worked. You can close this, thanks!

Submitted by Issues on Sun, 09/21/2014 - 22:47

Automatically closed -- issue fixed for 2 weeks with no activity.