Backups could optionally avoid temporary local files when piping to remote server (through e.g. scp) is possible

Submitted by beat on Thu, 11/22/2018 - 08:23

This is related to my bug-report https://www.virtualmin.com/node/59538 (Backups that fail for lack of space don't clean-up the /home/domain/.backup or /home/domains/../.backup file and fail on next backups too):

When copying by SCP to a remote server is setup, optionally it would be nice to pipe directly the files to the remote server. At least for files and database backups (e.g. piping tar|gz|scp resp. mysqldump|gz|scp) directly. This would avoid having a lot of local spare storage, and more importantly avoid completely the risk of not having enough local space to backup, and then when everything is crashed to be out of available backups..

Just an idea as a FR. :-) Thoughts ?

Related issues: 
#59538: Backups that fail for lack of space don't clean-up the /home/domain/.backup or /home/domains/../.backup file and fail on next backups too
Status: 
Active

Comments

Submitted by JamieCameron on Fri, 11/23/2018 - 14:32

That's not a bad idea, although it isn't possible to pipe a file into scp (as far as I know) - instead Virtualmin would need to pipe into a command like ssh user@host "cat >destfile.tgz" , which has the down-side that it won't work for scp-only remote users.

Submitted by beat on Fri, 11/23/2018 - 18:37

Btw, as big additional benefit to piping directly to the remote server, it would divide by 3 the disk-access-load on the backuped server! This is an even bigger reason that this feature could turn out to be very valuable, as Virtualmin backups are very disk-intensive and can generate intensive loads, even on local SSDs.

Indeed, you can't pipe to scp directly, as it needs file-size in advance. But:

If the remote server runs Virtualmin GPL, an open-source/open-standards but specific secure and encrypted communication channel could be established.

Btw.2. For good reason of a backup surviving any catastrophes, including economic collapse, it is crucial that this backup server functionality can be run on Virtualmin GPL.

Submitted by JamieCameron on Fri, 11/23/2018 - 18:48

I'll put this on my TODO list, as we do this kind of streaming backup already in Cloudmin so it should be possible in theory (if the destination user has the right privileges).

Submitted by beat on Fri, 11/23/2018 - 19:45

Very Cool! Thanks Jamie!

Here also some thoughts on these priviledges needed regarding the files-transfers and backups.

Same as you, I would love to avoid having to give ssh priviledges (in addition of scp-only) on the backup server to each server to be backed-up.

Right now, Virtualmin-backups are pushed from each server to the backup server onto a large harddisk. I have thus created a scp-only account on the backup server for each remote server to backed-up, and created its own chrooted directory on the backup server. That way if one server gets compromised, it doesn't access the other ones, and can't run software on the backup server nor access data of the other servers. However, scheduling backups to not happen simultaneously requires to check the amount of time needed by each server and to schedule them to not overlap, otherwise the network connection must be shared, and disk access on the spinning disk become way slower, overlapping even more servers. And then redo those timings and schedulings from time to time as data grows or shrinks.

My second backup method uses rdiff-backup, initiated from the backup server, thus it is a PULL backup method. A script is backing up one server after the other, continuously from the backup server. Additional benefit, no server has access to the backup server, and risk of compromise of the backup server which is off-site and highly secured is lower because of less software running on it. rdiff-backup has proven to be very reliable over last 10 years that i'm using it daily.

Rdiff-backup opens a remote ssh session from the backup-server with a rdiff-backup "server" program running on the server to be backed up. But the other way around would work too.

Just as thought, would be nice that the Virtualmin backups could also be initiated from the backup server's Virtualmin GPL installation ? That's where a Virtualmin-own open-source protocol could also make sense.

Submitted by JamieCameron on Fri, 11/23/2018 - 23:24

I'm reluctant to create my own file transfer protocol, as I doubt I could solve all the security and network issues that SSH has already solved!

Submitted by beat on Sat, 11/24/2018 - 17:34

Fully agree, it would be stupid to recreate something existing.

Maybe I wasn't specific enough: Idea was not to invent a new protocol, but to use an existing one (e.g. ssh, sftp, rsync, rdiff-backup, ...) user-selectable, with proven open-source code and piping capability, just under tight and authenticated control of virtualmin (authentication with existing protocols and proven code) ;-)

Submitted by JamieCameron on Sun, 11/25/2018 - 09:53

Oh yeah, what I'd do is just use ssh since it's the most widely supported option, possibly with a choice to fall back to scp.

Submitted by beat on Fri, 03/08/2019 - 00:52

Any news on this feature ? :-)

Still having daily massive disk overloads on a server due to the unneeded local intermediary storage.

Submitted by JamieCameron on Sat, 03/09/2019 - 18:42

Still on my TODO list. It's do-able, but non-trivial as it changes a lot about how backups are executed.