Compressing multiple files or directories with XZ and TAR

XZ is one of the best compression tools I’ve seen, it’s compressed files so big to a fraction of their size. I’ve had an 16 GB SQL dump, and it managed to compress it down to 263 MB.

I’ve needed to compress several files using XZ. And as XZ compresses single files we are gonna have to use TAR to do it And since the newer version of XZ supports multi threading, we can speed up compression quite a bit

Since the XZ utils in Debian/Ubuntu is an old version I’ve created my own backports that I can use.

echo "deb http://packages.matoski.com/ debian main" | sudo tee /etc/apt/sources.list.d/packages-matoski-com.list
curl -s http://packages.matoski.com/keyring.gpg | sudo apt-key add -
sudo apt-get update
sudo apt-get install xz-utils

This should give you a newer version of XZ that supports multi threading.

Let’s have a directory with a lot of files and subdirectories, to compress it all we would do

tar -cf - directory/ | xz -9e --threads=8 -c - > tarfile.tar.xz

On the newer versions of tar you can also do the following too

XZ_OPT="-9e --threads=8" tar cJf tarfile.tar.xz directory

Or if you want you can do it separately with a progress bar like this.

tar -cf - directory/ | (pv -s $(du -sb directory/ | awk '{print $1}') -p --timer --rate --bytes > tarfile.tar)
xz -9ev --threads=8 tarfile.tar

This will give you a nice progress bar while it’s creating the archive firstly and then compressing it with XZ