Backups are important. Very important. Fortunately, the combined power of zfs, rsync, and a few shell scripts makes things a breeze.

I generally try my best to follow a 3 2 1 rule for backups. As it currently stands, I have a FreeBSD server running the latest -RELEASE channel (12.2-RELEASE as of this writing) with zfs on all filesystems. Zfs pool is divided into volumes for photos, documents, videos, etc.

TODO Offsite backups with ZFS

Jails

Backing up jails with zfs snapshots takes a toll on storage as, for example, my unifi controller jail takes up 30GB after only a few months due to the logs it collects. Log data is not important enough for me to keep as archive, so I'll have to be selective about what I'm backing up.

For jails, my strategy involves listing all configuration files I care about in a file in the root of the jail and having rsync collect all jail-related config files in a central location (a zfs volume somewhere) neatly beside all the other files I want to backup.

There are generally 3 pieces of information I need to have to be able to fully restore a jail:

1. service-specific configuration files
2. rc.conf
3. pkgs

to get a list of manually installed (non-automatic), use pkg --chroot /path/to/jail/root query -e '%a = 0' %o (see pkg(8) and pkg-query(8)). In my case, I have rsyslog and ssmtp installed in all my jails with uniform configs, so I'll ignore those. I also don't use bitlbee anymore.

irc/bitlbee
ports-mgmt/dialog4ports
devel/libuv
ports-mgmt/pkg
sysutils/rsyslog8
mail/ssmtp
security/sudo
editors/vim-lite
irc/znc
shells/zsh

Restoring jails with ansible

I'll be using /mpool/scripts/backups/jail-deployment as the root of my ansible configuration files for jail deployment. I'm using ansible 2.9.7 on FreeBSD 12.2-RELEASE.

Setting up ansible

Ansible has support to run tasks in jails which we will use to setup the jails. You can also use ssh if your controller is remote, but I'll be using ansible on the host itself to keep things tidy. As jail connections use ansible connection plugins, a bit of setup is required.

configurations for jails goes into ansible.cfg. I had to specify python3 as ansible was only looking for specific python3 versions.

ansible.cfg

[defaults]
inventory = hosts
interpreter_python = python3

My hosts file looks as follows.

hosts

znc   ansible_connection=jail ansible_jail_host=/jails/znc   ansible_jail_user=root
db    ansible_connection=jail ansible_jail_host=/jails/db    ansible_jail_user=root
web   ansible_connection=jail ansible_jail_host=/jails/web   ansible_jail_user=root
...

site.yml

- import_playbook: znc/site.yml

znc/site.yml

- name: setup ZNC jail
  hosts: znc
  tasks:
    - name: test connection
      ping:

Make sure the jail is running. And test it.

ansible-playbook site.yml

And it fails! FreeBSD doesn't include python in its base install. To fix it, install python using pkg in the jail's root.

pkg --rootdir /jails/znc install python3

From here on, ansible can be used as usual to configure any of the jails. How to use ansible is left as an exercise to the reader.

Databases

postgresql

postgresql provides pg_basebackup which is very handy. Postgres also has pg_dumpall. See their respective documentation for further info. I ended up using pg_basebackup since it backs up the entire $PGDATA directory including configuration files and WAL files.

mkdir postgres_backup
pg_basebackup -D postgres_backup --format=tar --gzip --progress

To see what has been backed up, use tar.

tar -t -f base.tar.gz

To restore the database, untar base.tar.gz into $PGDATA (on FreeBSD it's /var/db/postgres/dataXX/ and untar pg_wal.tar.gz into $PGDATA/pg_wal.

mariadb

mariadb provides mysqldump which is equally as handy. my.cnf has to be backed up separately. I couldn't find an equivalent to pg_basebackup for mysql.

mysqldump -u root -p -x -A | gzip > mariadb_backup.gz

To restore, start mysql-server, and restore.

gunzip mariadb_backup.gz
mysql -u root < mariadb_backup

To user mysqldump without entering a password (i.e. for scripted backups), add the following to my.cnf. On FreeBSD, it's /usr/local/etc/mysql/my.cnf.

[mysqldump]
user=root
password=<password>

Making things regular

To perform regular backup, I have written a script for each component that I want backed up. These "components" include either full jail configurations, or single databases.

I then symlink the script to /usr/local/etc/periodic/weekly/ which then runs the backups every week. A sample script is as follows.

#!/bin/sh

set -o pipefail
set -o nounset

NAME="db"
COMPONENT="postgresql"

BASEDIR="/mpool/jail_backup/$NAME"
JAILDIR="/zroot/iocage/jails/$NAME/root"

BACKUPDIR="$BASEDIR/$(date +'%Y%m%d')/$COMPONENT"
mkdir -p "$BACKUPDIR"

# backup postgresql
iocage exec db -- pg_basebackup -D "/pgbackup" --format=tar --gzip
rsync -q --remove-source-files --recursive "$JAILDIR/pgbackup/" "$BACKUPDIR/"