Introduction

There are two kinds of people: those who backup, and those who have yet to lose all their data.

I have no idea who said that, but I know that those are words of a wise person. I lost my data a few times, and since then I try to back up my digital belongings. I am not exactly successful every time (hence a few times, not once), but each time has taught me something.

I went from burning things into CDs and later DVDs, through storing copies on a second computer, then to a server in my local network. All of those I did manually, using different methods, but at one point I’ve discovered Clonezilla and used it thereafter.

The problem with all those methods is that I couldn’t easily automate them. And since I am very awful at keeping tabs on my schedules, big or small, important or not, I sometimes lost data just because my backups weren’t recent enough.

Lately I’ve upgraded to cloud storage with restic. I still haven’t got a proper 3-2-1 strategy implemented, because I only have two copies of the data, but I’m getting there.

Doing the actual work

Using restic is easy; you have to create a repository that will contain your files: restic init --repo /path/to/repository and then you can start backing up with restic repo /path/to/repository backup /directory/to/backup. Very easy to use 🙂 Each repository has to be defined before you can actually do operations on it, so it’s time to do that now.

I am using Backblaze’s B2 cloud storage to contain my repository, so there will be some environment variables needed, but we’ll get to that.

Defining systemd unit

First, you need to start with a systemd unit file that will launch the process. This is one that I am using for backing up my $HOME directory:

# ~/.config/systemd/user/restic-home-backup.service
[Unit]
Description=Restic backup service of the home directory
OnFailure=failure-user-notification@%n.service
[Service]
Type=oneshot
ExecStart=restic backup --verbose --one-file-system --tag systemd.timer.home $EXCLUDES $PATH
ExecStartPost=/bin/bash --login -c 'restic forget --verbose --tag systemd.timer.home --group-by "paths,tags" --keep-daily $KEEP_DAYS --keep-weekly $KEEP_WEEKS --keep-monthly $KEEP_MONTHS --keep-yearly $KEEP_YEARS && %h/scripts/add-notification.sh normal "home backup finished"'
EnvironmentFile=%h/.config/restic-backups/home.conf

This is a one-time service that runs and then exits. If it fails, it invokes the failure-user-notification service, and if everything goes okay, then it tells restic to prune some old backups, to not inflate my bill on B2 too much 😀 After that it sends me a desktop notification that the whole process went okay.

It also contains a flag that prevents restic from backing up anything from resources mounted somewhere in the source directory, so you accidentally don’t get the contents of you 1 TiB external drive that your OS for some weird reason mounted in your $HOME. The variables are defined in a file that contains:

# ~/.config/restic-backups/home.conf
PATH="/home/kamil"
EXCLUDES="--exclude-file /home/kamil/.config/restic-backups/home-excludes --exclude-if-present .backup-ignore"
KEEP_DAYS=7
KEEP_WEEKS=4
KEEP_MONTHS=6
KEEP_YEARS=3
B2_ACCOUNT_ID=redacted
B2_ACCOUNT_KEY=redacted
RESTIC_REPOSITORY=redacted
RESTIC_PASSWORD=redacted

B2_ACCOUNT_ID is the keyID of your Master Application Key or one of your Application Keys; B2_ACCOUNT_KEY is the applicationKey you see only once, after you generate a new key. This tells the unit to backup my home directory and exclude everything that I included in the file, and anything that has a .backup-ignore file in it. The exclude file should just contain paths that you don’t want in the backup, one per line like so:

/one/excluded/path
/another/one
/and/one/more

But it’s not everything; over time restic accumulates data that isn’t really needed and can be removed. To get rid of that too, I decided to create an additional service:

# ~/.config/systemd/user/restic-home-backup-pruning.service
[Unit]
Description=Prune restic backups of the home directory
OnFailure=failure-user-notification@%n.service
[Service]
Type=oneshot
ExecStart=restic prune
ExecStartPost=/bin/bash --login -c 'restic prune --verbose && %h/scripts/add-notification.sh normal "home backup pruning finished"'
EnvironmentFile=%h/.config/restic-backups/home.conf

The only thing that’s left is this weird failure-user-notification service that those above units mention. This one is special, but I’ll describe why later on.

# ~/.config/systemd/user/failure-user-notification@.service
[Unit]
Description=Send a notification to the user
[Service]
Type=oneshot
ExecStart=/home/kamil/scripts/add-notification.sh critical "%i failed"

After each modification you need to run systemctl --user daemon-reload so that systemd reads their new contents. If you want to test things out, run systemctl --user start restic-home-backup.service and see what happens. To inspect any errors, just use journalctl --user -u restic-home-backup.service with maybe an additional -f flag to follow the output live.

Automating the processes

This is fine and all, but as I have mentioned, these units run one time and do nothing more. This is why we need timers!

# ~/.config/systemd/user/restic-home-backup.timer
[Unit]
Description=Backup with restic daily
[Timer]
OnCalendar=daily
Persistent=true
[Install]
WantedBy=timers.target

This little piece of code tells systemd to run a service daily at 02:00, or at the first moment possible (thanks to Persistent=true) whatever happens first. It is useful if you don’t keep your computer on at such times, which is probably true for many. Now we only need to define a timer for the pruning unit:

# ~/.config/systemd/user/restic-home-backup-pruning.timer
[Unit]
Description=Prune data from the restic repository monthly
[Timer]
OnCalendar=*-*-01 16:00:00
Persistent=true
[Install]
WantedBy=timers.target

This tells systemd to run this at the start of each month at 16:00. I didn’t use the monthly, since this would often result in the both timers starting together. This leads to one of the jobs failing, because restic can’t do two things at once with a repository.

To actually make things automatic, we just need to inform systemd of the changes using systemctl --user daemon-reload and then enabling the timers with systemctl --user enable restic-home-backup.timer and systemctl --user enable restic-home-backup-pruning.timer.

Backups of the filesystem root

Here I won’t include all of the nitty-gritty details, since the unit files are very similar, but use the path to your $HOME instead of %h in the unit files. I personally created them in ~/.config/systemd/user/ too and just later on linked to /etc/systemd/system/ (you need to link the failure-user-notification@.service too!). Then use systemctl as root and without the --user flag.

Notifications

As I promised, I need to explain that weird failure-user-notification@.service.

This is a kinda hacky solutions to my problem, but I thought about it a 4 AM on Sunday, it’s working, so I didn’t change it 😅 I wanted to have some notification about the status of my daily backup processes. I didn’t want to use mail and sendmail, because I have a working setup for desktop notifications that I could leverage with notify-send. In the systemd unit owned by my user I could use it just fine and the notifications were visible, but that didn’t work for the ones owned by root.

I played around with different ideas with no success, but then the *NIX muse whispered into my ear a single word: tail. The idea immediately crystalized in my head: I’ll just write the notification to a file that is constantly watched by tail -f that then forwards the input to notify-send. I only need to run the script as my user, and everything will be working.

I prepared the needed files and directories:

# shell
mkdir ~/.cache/system-notifications/
touch ~/.cache/system-notifications/{normal,critical}

I wrote a script to loop over the files I created:

#!/bin/bash
# ~/scripts/notifications.sh

loop_over() {
    urgency=$(basename "$1")
    while read -r line
    do
        notify-send --urgency "$urgency" "$line"
    done < <(tail -f "$1" -n 0)
}

for file in /home/kamil/.cache/system-notifications/*
do
    loop_over "$file" &
done

Finally I just wrote script to add apropriate notifications:

#!/bin/bash
# ~/scripts/add-notification.sh

args_length=${#@}
level=$1
message=${*: 2:$args_length}

echo "$message" >> /home/kamil/.cache/system-notifications/"$level"

For it to actually work, you need to launch notifications.sh first, so I added ~/bin/notifications.sh & to my ~/.profile.

To use it, just invoke it like so: ~/scripts/add-notification.sh critical some critical notification.

Wrapping things up

There it is, my backup solution using restic, systemd and a bit of pipes magic to spice things up. I have been using it for half a month already, and everything works properly. The backups are being restored properly and it is working fine without any involvement on my end.

Tags: english technical tutorial linux

Posted on: 2021-04-17, last edited on: 2021-12-25, written by: Kavelach