Backup logs stopped working

Backups started failing due to reaching limit of backup processes (3). I did see these backup processes running. So I rebooted the VM server. Now, backups are working again but the logs are not updating in the GUI. How to troubleshoot backup logs? Thanks

Status: 
Closed (fixed)

Comments

Are you sure that some backups have completed? Only complete backups show up in the logs..

I do see the files in the backup destination, but maybe the backup didn't complete? How can I tell if the backup process is still running and if it is, how to troubleshot why it's not completing.

Check if there is still a backup.pl process running on the system.

No backup processes running.

Try running the backup manually (via the UI) and let me know if it completes or not. It may be that the backup is dieing part way though..

The manual job completed when run from the GUI and it shows up in the "Backups Logs" section of the GUI.

Now this is interesting. The manual backup shows as completed successfully in the log, but the process still shows in the process list:

/usr/libexec/webmin/virtual-server/backup.cgi

A scheduled full backup ran last night, data seems to be there but no log in GUI. I can't be 100% sure that it completed successfully... is there another place to look for the log?

Check the cron log, if your system has one - it will be named like /var/log/cron

The backups appear to have run last night because I see the backup files in the destination. However, nothing was added to the backup log in the GUI, so I cannot actually read the log.

in cron, there was this:

Sep  2 04:30:01 kickback crond[31053]: (root) CMD (/etc/webmin/virtual-server/backup.pl --id 13653903586283)

and there are no backup processes running now.

Can you try SSHing in as root and running the command /etc/webmin/virtual-server/backup.pl --id 13653903586283 , and see if it crashes?

Is there supposed to be any output? It's running but no output to the console.

If it works, there should be no output.

If it fails, it should output some error message - please post that here.

it's running for a couple of hours, no output and no files written yet. backup.pl is running in the console. I can keep it running and see what happens in a few more hours.

got the error finally. This server was recently rebooted by the way.

[root@kickback ~]# /etc/webmin/virtual-server/backup.pl --id 13653903586283
 
 
Error: Failed to open /etc/webmin/virtual-server/backuplogs/1409722236-26417-1 for writing : Too many open files
Error
-----
Failed to open /etc/webmin/virtual-server/backuplogs/1409722236-26417-1 for writing : Too many open files

finally got the error

[root@kickback ~]# /etc/webmin/virtual-server/backup.pl --id 13653903586283

Error: Failed to open /etc/webmin/virtual-server/backuplogs/1409722236-26417-1 for writing : Too many open files

Error

Failed to open /etc/webmin/virtual-server/backuplogs/1409722236-26417-1 for writing : Too many open files

What is the output of this command:

sysctl fs.file-max

sysctl fs.file-max

fs.file-max = 391269

You could try increasing this with a command like sysctl -w fs.file-max=1000000

I tried that, but this commend seems to show the limit on open files is 1024?

[root@kickback ~]# ulimit -Hn 1024

We'll see if the backup logs work tonight.

It appears that the backup ran but again no log in GUI

Does a manual run with /etc/webmin/virtual-server/backup.pl --id 13653903586283 still fail due to lack of file descriptors?

The logs started working again. Maybe it was the max open files fix that worked. You can close this, thanks!

Automatically closed -- issue fixed for 2 weeks with no activity.