Error reading response length from fastrpc.cgi in firewall cluster

First thing to check - does the Linux Firewall module work OK on the remote servers?

Actually, I see the cause of this - it will be fixed in the next Webmin package release.

Automatically closed - issue fixed for 2 weeks with no activity.

I have been randomly getting this error and ignoring it thinking its intermittent but it's starting to become more regular now in my scheduled daily backups. It has only appeared since I added backup to Webmin Server (my newly spun up hot backup) as a destination.

My pattern seems to be that it does first few virtual servers without error and then gets to our largest one (data wise - about 2.5 GB tar) and it throws this error.

My destinations are

1. Save to local diretcory
2. Save To Webmin Server
3. Command to run after backup (rclone command to backup to our G Suite drive account)

I have set backup to "Continue, but don't purge old backups" so it completes steps 1 & 3 above. Step 2 stops when it reaches the large domain (and throws above error).

My Specs CentOS 7.7 on source and destination (destination running on Hyper-V VM) Virtualmin 6.08 Pro

Does anything get logged to /var/webmin/miniserv.error on the remote system? Also , how much RAM does the remote server have?

It's a VM which I have only assigned 1GB to - I could up to see if that helps. I see no errors related to this activity in the miniserv.error.

miniserv.log has a single entry root [11/Jan/2020:00:00:08 -0500] "POST /fastrpc.cgi HTTP/1.0" 200 47 That appears to indicate all was ok (this was one of the dates that had an error, last couple of nights have been ok

One thing to try would be increasing the RAM on the VM to at least 3 GB, and see if that helps with a 2.5 GB backup.

Thanks I upped memory to 2GB and did have not experienced the error since. I will consider this closed at this time. Thx

Wow, unbelievable just as I updated this thread I got my latest backup email confirmation (runs at midnight EST) and it failed again - first time in about a week. This time it was an incremental backup which typically max out at about 800 MB so this seems to be an issue beyond RAM allocation.

As a further followup up my full backups I switched the second destination to use SSH to the hot backup VM, instead of Webmin Server. I will do the same now for the incremental backups and see if that cures it. What module is this error related to so I can look possibly at some other logs to debug?

You might also check /var/log/messages on the destination system for any reports of OOMs.

Also, make sure that TCP ports 10000-10100 are open on the destination server, as Webmin uses that range for RPCs.

Firewall ports no issue (using default install which opens this range). If firewall was an issue I would expect it would never work at all, not just bomb on large backups archives.

Nothing in messages log relating to memory issues

I have switched to using SSH on full backup also since my last post, which completed successfully last Sunday evening so it seems it is isolated to Webmin server backup destinations.

Is this a problem that only happens for very large backups?

Yes it appears so, not sure what you classify as "very large" though. It seems more of some type of timeout issue. I checked back in my logs and it is always the same, when there is a failure.

First several virtual servers backup fine (local and then to webmin server)

Uploading archive to Webmin server 192...* .. .. done

Then (not always the same virtual server but yes seemingly always ones over 500 MB in size)

Uploading archive to Webmin server 192...* .. .. upload failed! Error reading response length from fastrpc.cgi :

Then all subsequent virtual servers will fail with same error so once it fails it never recovers.

As I said since I updated this second destination to use SSH (to exact same destination server IP where I changed nothing) instead of Webmin Server I have had zero failures. Seems like there needs to be more logging added somewhere for this fastrpc script to trace the source of the issue. Not sure how long its been an issue as I have never before, in all these years, used backup to Webmin Server. Ironically I thought it would be a more stable option :)

I wonder if some timeout on the upload is being hit.

Does it fail after exactly the same amount of time on every backup?

BTW, I found some inefficiencies in the code when uploading these backups - they will be fixed in the next Webmin release.