ZSTD compression for backup

Submitted by hregis on Thu, 01/07/2021 - 08:57 Pro Licensee

Hello

can you add the ZSTD compression for the backup please ?

Thank you

Status: 
Active
Virtualmin version: 
6.14
Webmin version: 
1.962
Related packages versions: 
Pro

Comments

Submitted by JamieCameron on Fri, 01/08/2021 - 01:11

Is it significantly better than gzip and bzip2 ?

Submitted by hregis on Fri, 01/08/2021 - 02:42 Pro Licensee

Hello Jamie,
it's clearly more efficient !

compression with the default level (gzip: 6, bzip2: 6 and zstd: 3)
zstd have more options to greatly improved compression
see https://www.manpagez.com/man/1/zstd/

-rw-r--r-- 1 root root 3,0G janv. 22 2020 aeb_dol1.sql
-rw-r--r-- 1 root root 3,0G janv. 8 08:25 aeb_dol2.sql
-rw-r--r-- 1 root root 3,0G janv. 8 08:26 aeb_dol3.sql

gzip aeb_dol1.sql
113,04s user 2,34s system 63% cpu 3:01,26 total

bzip2 aeb_dol2.sql
426,09s user 2,77s system 84% cpu 8:27,17 total

zstd -z -q --rm aeb_dol3.sql
9,31s user 2,12s system 16% cpu 1:09,08 total

files after compression:

-rw-r--r-- 1 root root 1,8G janv. 22 2020 aeb_dol1.sql.gz
-rw-r--r-- 1 root root 1,7G janv. 8 08:25 aeb_dol2.sql.bz2
-rw-r--r-- 1 root root 1,6G janv. 8 08:26 aeb_dol3.sql.zst

decompression:

gzip -d aeb_dol1.sql.gz
20,99s user 2,41s system 24% cpu 1:34,63 total

bzip2 -d aeb_dol2.sql.bz2
163,77s user 3,65s system 69% cpu 4:02,01 total

zstd -d -q --rm aeb_dol3.sql.zst
3,01s user 2,41s system 6% cpu 1:28,87 total

Submitted by JamieCameron on Sat, 01/09/2021 - 20:28

Wow, that is quite efficient.

Are the zstd compression commands included with any common Linux distributions?

Submitted by hregis on Sun, 01/10/2021 - 03:44 Pro Licensee

to my knowledge : debian, ubuntu, centos, fedoraproject...

Ilia's picture
Submitted by Ilia on Sun, 01/10/2021 - 07:15

Stats look too good to be true!

Jamie, I checked and indeed all distros have it and it can be installed by package name as zstd in CentOS 7-8, Debian 9-10, and Ubuntu 18-20.

Submitted by JamieCameron on Sun, 01/10/2021 - 23:23

OK - I'll put this on my TODO list to investigate.

Does the tar command have built-in support for the ZSTD format, like it does for gzip?

Submitted by hregis on Mon, 01/11/2021 - 01:30 Pro Licensee

yes you can use :

tar --zstd -cf .......
tar --zstd -xf .......