0 Rsync backups that restore: baseline and verification (2025) - kevwells.com

Rsync backups that restore: baseline and verification (2025)

by

TL;DR:

  • Back up to dated snapshots using --link-dest so each day looks full but uses hard links.

  • Keep an exclude file, don’t recurse into /proc, /sys, etc., and avoid infinite loops by excluding the backup path itself.

  • After every run, prove you can restore one file and one directory.

  • Rotate snapshots (e.g., keep 30 dailies, 12 monthlies). Log everything.

1) Scope and baseline

This guide covers Linux server backups with rsync (local or over SSH). Two common patterns:

  • Data-only (recommended for most): /etc, /home, app data (e.g., /var/www, /srv/*).

  • Whole-system: root filesystem without pseudo-filesystems and caches.

Either way, snapshots are dated directories plus a latest symlink:

/mnt/backup/hostname/
├── 2025-08-19/
├── 2025-08-20/
└── latest -> 2025-08-20

2) Exclude file (save as /etc/rsync/excludes.txt)

For whole-system style. Trim if you’re doing data-only.

# Pseudo and runtime
/dev/*
/proc/*
/sys/*
/run/*
/tmp/*
/var/tmp/*
/lost+found

# Caches and transient
/var/cache/*
/var/lib/apt/lists/*
/var/log/journal/*

# Mount points (avoid recursing into other filesystems)
#/mnt/*
#/media/*
#/backup/*
# Adjust to your environment:
#/var/lib/docker/*
#/var/lib/containerd/*

# Swap / image junk
/swapfile
*.iso

# IMPORTANT: exclude your backup destination to avoid recursion (adjust to your path)
#/mnt/backup/*

Uncomment paths you actually have. If you do data-only, you don’t need most of this—just back up the selected trees.

3) First full snapshot (local disk)

Choose a destination (example): /mnt/backup/$(hostname -s).

sudo mkdir -p /mnt/backup/$(hostname -s)
sudo chown -R root:root /mnt/backup
DATE=$(date +%F)
DEST="/mnt/backup/$(hostname -s)/$DATE"

# Example: whole-system style (stays on one filesystem; excludes below)
sudo rsync -aHAXx --numeric-ids \
--info=stats2,progress2 \
--exclude-from=/etc/rsync/excludes.txt \
/ "$DEST/"

# Update the 'latest' symlink atomically
cd "$(dirname "$DEST")" && sudo ln -sfn "$DATE" latest

Flags explained (brief):

  • -aHAXx → archive, hardlinks, ACLs, xattrs, stay on this filesystem.

  • --numeric-ids → preserve uids/gids numerically (safer across hosts).

  • --info=stats2,progress2 → usable logging without chatty noise.

Data-only variant (back up only key trees):

sudo rsync -aHAX --numeric-ids \
--info=stats2,progress2 \
/etc/ /home/ /var/www/ /srv/ "$DEST/"
cd "$(dirname "$DEST")" && sudo ln -sfn "$DATE" latest

4) Daily snapshots with --link-dest (incrementals that look full)

Create /usr/local/sbin/rsync-snapshot.sh:

#!/usr/bin/env bash
set -Eeuo pipefail

SRC="/"
BASE="/mnt/backup/$(hostname -s)"
DATE=$(date +%F)
DEST="$BASE/$DATE"

# Previous snapshot if exists
LINKDEST=""
if [[ -L "$BASE/latest" ]]; then
LINKDEST="--link-dest=$BASE/latest"
fi

# Ensure destination dir
mkdir -p "$DEST"

# Whole-system style (edit for data-only; remove -x if you want to traverse mounts)
rsync -aHAXx --numeric-ids \
--delete --delete-excluded \
--info=stats2 \
--exclude-from=/etc/rsync/excludes.txt \
$LINKDEST \
"$SRC" "$DEST/"

ln -sfn "$DATE" "$BASE/latest"

# Simple retention: keep last 30 dailies
find "$BASE" -maxdepth 1 -type d -name "20[0-9][0-9]-*" -mtime +30 -exec rm -rf {} +

# Log a marker
echo "$(date -Is) snapshot $DATE complete" >> /var/log/rsync-backup.log

sudo install -m 0750 /usr/local/sbin/rsync-snapshot.sh /usr/local/sbin/rsync-snapshot.sh

Schedule (choose one):

  • cron: sudo crontab -e

    # Daily at 02:30
    30 2 * * * /usr/local/sbin/rsync-snapshot.sh

  • systemd timer (more reliable):

    • /etc/systemd/system/rsync-snapshot.service

      [Unit]
      Description=Daily rsync snapshot
      [Service]
      Type=oneshot
      ExecStart=/usr/local/sbin/rsync-snapshot.sh

    • /etc/systemd/system/rsync-snapshot.timer

      [Unit]
      Description=Run rsync snapshot daily
      [Timer]
      OnCalendar=*-*-* 02:30:00
      Persistent=true
      [Install]
      WantedBy=timers.target

    • Activate:

      sudo systemctl daemon-reload
      sudo systemctl enable --now rsync-snapshot.timer

--delete removes files deleted at source in the current snapshot; prior snapshots retain them thanks to hard links. Review your comfort level before enabling.

5) Offsite (over SSH)

Back up to a remote host backup@backupbox into /srv/backup/$(hostname -s)/$DATE/:

DATE=$(date +%F)
BASE="/srv/backup/$(hostname -s)"
DEST="$BASE/$DATE"
SSH_OPTS="-i /root/.ssh/backup_ed25519 -o BatchMode=yes -o StrictHostKeyChecking=yes"

# Ensure base and linkdest exist remotely
ssh -o "IdentitiesOnly=yes" $SSH_OPTS backup@backupbox "mkdir -p '$DEST' '$BASE'"

LINKDEST=""
if ssh $SSH_OPTS backup@backupbox "[ -L '$BASE/latest' ]"; then
LINKDEST="--link-dest=$BASE/latest"
fi

sudo rsync -aHAXx --numeric-ids \
--delete --delete-excluded \
--info=stats2 \
--exclude-from=/etc/rsync/excludes.txt \
-e "ssh $SSH_OPTS" \
/ backup@backupbox:"$DEST/"

ssh $SSH_OPTS backup@backupbox "ln -sfn '$DATE' '$BASE/latest'"

Hardening tip: create a backup user on the remote box with limited permissions and an SSH forced-command if you want to restrict what rsync can do.

6) Verify restores (every run, not once a year)

Single file

# Compare before overwrite (diff exits 0 if identical)
sudo diff -u /etc/ssh/sshd_config /mnt/backup/$(hostname -s)/latest/etc/ssh/sshd_config || true

# Restore
sudo rsync -aHAX /mnt/backup/$(hostname -s)/latest/etc/ssh/sshd_config /etc/ssh/sshd_config

Directory

sudo rsync -aHAX --delete /mnt/backup/$(hostname -s)/latest/etc/ /etc/

Whole snapshot dry-run (sanity check)

sudo rsync -aHAXxn --checksum --delete \
--exclude-from=/etc/rsync/excludes.txt \
/ /mnt/backup/$(hostname -s)/latest/

On SELinux systems, follow restores with restorecon -R /path as required.

7) Logging and rotation

Add --log-file=/var/log/rsync-backup.log to your rsync command(s) if you want full logs, then set logrotate:

/etc/logrotate.d/rsync-backup:

/var/log/rsync-backup.log {
weekly
rotate 12
compress
missingok
notifempty
create 0640 root adm
}

8) Common pitfalls (avoid these)

  • Trailing slash semantics: rsync SRC/ DEST/ copies contents of SRC; rsync SRC DEST/ creates a subdir DEST/SRC. Be explicit.

  • Recursion into backups: always exclude the backup path (or keep destination on a different filesystem/mount).

  • Permissions not preserved: use -a, and include -A -X -H where you care about ACLs, xattrs, hard links.

  • UID/GID mismatch: add --numeric-ids.

  • No restore proof: schedule a monthly restore test to a temporary path and document the steps.

9) Minimal checklist

  • Exclude file in place and correct for your system.

  • First snapshot completes; latest symlink points to it.

  • Daily job scheduled (cron or systemd timer).

  • One file restored and verified.

  • One directory restored and verified.

  • Retention removes old snapshots (confirm).

  • Offsite copy (SSH) working, or at-rest encryption on the backup disk (e.g., LUKS).

Security gaps in Linux and cloud systems risk downtime, data compromise, lost business — and compliance failures.

With 20+ years’ experience and active UK Security Check (SC) clearance, I harden Linux and cloud platforms for government, corporate, and academic sectors — ensuring secure, compliant, and resilient infrastructure.

Contact for Contract Assignments

No related posts.