Introduction

There are two kinds of people: those who backup, and those who have yet to lose all their data.

I have no idea who said that, but I know that those are words of a wise person. I lost my data a few times, and since then I try to back up my digital belongings. I am not exactly successful every time (hence a few times, not once), but each time has taught me something.

I went from burning things into CDs and later DVDs, through storing copies on a second computer, then to a server in my local network. All of those I did manually, using different methods, but at one point Iā€™ve discovered Clonezilla and used it thereafter.

The problem with all those methods is that I couldnā€™t easily automate them. And since I am very awful at keeping tabs on my schedules, big or small, important or not, I sometimes lost data just because my backups werenā€™t recent enough.

Lately Iā€™ve upgraded to cloud storage with restic. I still havenā€™t got a proper 3-2-1 strategy implemented, because I only have two copies of the data, but Iā€™m getting there.

Doing the actual work

Using restic is easy; you have to create a repository that will contain your files: restic init --repo /path/to/repository and then you can start backing up with restic repo /path/to/repository backup /directory/to/backup. Very easy to use šŸ™‚ Each repository has to be defined before you can actually do operations on it, so itā€™s time to do that now.

I am using Backblazeā€™s B2 cloud storage to contain my repository, so there will be some environment variables needed, but weā€™ll get to that.

Defining systemd unit

First, you need to start with a systemd unit file that will launch the process. This is one that I am using for backing up my $HOME directory:

# ~/.config/systemd/user/restic-home-backup.service
[Unit]
Description=Restic backup service of the home directory
OnFailure=failure-user-notification@%n.service
[Service]
Type=oneshot
ExecStart=restic backup --verbose --one-file-system --tag systemd.timer.home $EXCLUDES $PATH
ExecStartPost=/bin/bash --login -c 'restic forget --verbose --tag systemd.timer.home --group-by "paths,tags" --keep-daily $KEEP_DAYS --keep-weekly $KEEP_WEEKS --keep-monthly $KEEP_MONTHS --keep-yearly $KEEP_YEARS && %h/scripts/add-notification.sh normal "home backup finished"'
EnvironmentFile=%h/.config/restic-backups/home.conf

This is a one-time service that runs and then exits. If it fails, it invokes the failure-user-notification service, and if everything goes okay, then it tells restic to prune some old backups, to not inflate my bill on B2 too much šŸ˜€ After that it sends me a desktop notification that the whole process went okay.

It also contains a flag that prevents restic from backing up anything from resources mounted somewhere in the source directory, so you accidentally donā€™t get the contents of you 1 TiB external drive that your OS for some weird reason mounted in your $HOME. The variables are defined in a file that contains:

# ~/.config/restic-backups/home.conf
PATH="/home/kamil"
EXCLUDES="--exclude-file /home/kamil/.config/restic-backups/home-excludes --exclude-if-present .backup-ignore"
KEEP_DAYS=7
KEEP_WEEKS=4
KEEP_MONTHS=6
KEEP_YEARS=3
B2_ACCOUNT_ID=redacted
B2_ACCOUNT_KEY=redacted
RESTIC_REPOSITORY=redacted
RESTIC_PASSWORD=redacted

B2_ACCOUNT_ID is the keyID of your Master Application Key or one of your Application Keys; B2_ACCOUNT_KEY is the applicationKey you see only once, after you generate a new key. This tells the unit to backup my home directory and exclude everything that I included in the file, and anything that has a .backup-ignore file in it. The exclude file should just contain paths that you donā€™t want in the backup, one per line like so:

/one/excluded/path
/another/one
/and/one/more

But itā€™s not everything; over time restic accumulates data that isnā€™t really needed and can be removed. To get rid of that too, I decided to create an additional service:

# ~/.config/systemd/user/restic-home-backup-pruning.service
[Unit]
Description=Prune restic backups of the home directory
OnFailure=failure-user-notification@%n.service
[Service]
Type=oneshot
ExecStart=restic prune
ExecStartPost=/bin/bash --login -c 'restic prune --verbose && %h/scripts/add-notification.sh normal "home backup pruning finished"'
EnvironmentFile=%h/.config/restic-backups/home.conf

The only thing thatā€™s left is this weird failure-user-notification service that those above units mention. This one is special, but Iā€™ll describe why later on.

# ~/.config/systemd/user/failure-user-notification@.service
[Unit]
Description=Send a notification to the user
[Service]
Type=oneshot
ExecStart=/home/kamil/scripts/add-notification.sh critical "%i failed"

After each modification you need to run systemctl --user daemon-reload so that systemd reads their new contents. If you want to test things out, run systemctl --user start restic-home-backup.service and see what happens. To inspect any errors, just use journalctl --user -u restic-home-backup.service with maybe an additional -f flag to follow the output live.

Automating the processes

This is fine and all, but as I have mentioned, these units run one time and do nothing more. This is why we need timers!

# ~/.config/systemd/user/restic-home-backup.timer
[Unit]
Description=Backup with restic daily
[Timer]
OnCalendar=daily
Persistent=true
[Install]
WantedBy=timers.target

This little piece of code tells systemd to run a service daily at 02:00, or at the first moment possible (thanks to Persistent=true) whatever happens first. It is useful if you donā€™t keep your computer on at such times, which is probably true for many. Now we only need to define a timer for the pruning unit:

# ~/.config/systemd/user/restic-home-backup-pruning.timer
[Unit]
Description=Prune data from the restic repository monthly
[Timer]
OnCalendar=*-*-01 16:00:00
Persistent=true
[Install]
WantedBy=timers.target

This tells systemd to run this at the start of each month at 16:00. I didnā€™t use the monthly, since this would often result in the both timers starting together. This leads to one of the jobs failing, because restic canā€™t do two things at once with a repository.

To actually make things automatic, we just need to inform systemd of the changes using systemctl --user daemon-reload and then enabling the timers with systemctl --user enable restic-home-backup.timer and systemctl --user enable restic-home-backup-pruning.timer.

Backups of the filesystem root

Here I wonā€™t include all of the nitty-gritty details, since the unit files are very similar, but use the path to your $HOME instead of %h in the unit files. I personally created them in ~/.config/systemd/user/ too and just later on linked to /etc/systemd/system/ (you need to link the failure-user-notification@.service too!). Then use systemctl as root and without the --user flag.

Notifications

As I promised, I need to explain that weird failure-user-notification@.service.

This is a kinda hacky solutions to my problem, but I thought about it a 4 AM on Sunday, itā€™s working, so I didnā€™t change it šŸ˜… I wanted to have some notification about the status of my daily backup processes. I didnā€™t want to use mail and sendmail, because I have a working setup for desktop notifications that I could leverage with notify-send. In the systemd unit owned by my user I could use it just fine and the notifications were visible, but that didnā€™t work for the ones owned by root.

I played around with different ideas with no success, but then the *NIX muse whispered into my ear a single word: tail. The idea immediately crystalized in my head: Iā€™ll just write the notification to a file that is constantly watched by tail -f that then forwards the input to notify-send. I only need to run the script as my user, and everything will be working.

I prepared the needed files and directories:

# shell
mkdir ~/.cache/system-notifications/
touch ~/.cache/system-notifications/{normal,critical}

I wrote a script to loop over the files I created:

#!/bin/bash
# ~/scripts/notifications.sh

loop_over() {
    urgency=$(basename "$1")
    while read -r line
    do
        notify-send --urgency "$urgency" "$line"
    done < <(tail -f "$1" -n 0)
}

for file in /home/kamil/.cache/system-notifications/*
do
    loop_over "$file" &
done

Finally I just wrote script to add apropriate notifications:

#!/bin/bash
# ~/scripts/add-notification.sh

args_length=${#@}
level=$1
message=${*: 2:$args_length}

echo "$message" >> /home/kamil/.cache/system-notifications/"$level"

For it to actually work, you need to launch notifications.sh first, so I added ~/bin/notifications.sh & to my ~/.profile.

To use it, just invoke it like so: ~/scripts/add-notification.sh critical some critical notification.

Wrapping things up

There it is, my backup solution using restic, systemd and a bit of pipes magic to spice things up. I have been using it for half a month already, and everything works properly. The backups are being restored properly and it is working fine without any involvement on my end.

Tags: english technical tutorial linux

Posted on: 2021-04-17, last edited on: 2021-12-25, written by: Kavelach