Building an Automated Backup & Restore System for a Linux Web Stack

by

Lab Projects · by Kevin Wells

Executive summary

This project documents a production-grade backup and restore system for a small Linux web stack hosting multiple WordPress sites.

The design goal was reliability over cleverness: full nightly snapshots, 7-day retention, fast verification, and a restore process you can execute at 03:00 without problems. 

Objectives

  • Back up system configuration so a bare-metal rebuild is painless.
  • Back up each website’s webroot and database as atomic units.
  • Write to an encrypted target only. Abort if the target is not mounted.
  • Keep retention simple and tight: 7 days of nightly snapshots.
  • Make restores repeatable and boring. Test them.

What is backed up

System core

  • /etc (Apache vhosts, PHP, certificates, network, SSH, firewall, cron, logrotate, etc.)
  • /usr/local/sbin and /usr/local/bin (operational scripts)
  • /opt and /root (service installs and admin notes)
  • Package state snapshots: dpkg --get-selections, apt-mark showmanual, snap list

Websites (multi-site)

  • Webroot per site (e.g. /var/www/<site>) with caches excluded
  • Database per site via mysqldump with safe flags (--single-transaction, --skip-lock-tables, etc.)

Not backed up: transient caches, package caches, and bulky logs unless needed for forensics.

Destination and retention

  • Encrypted target mounted at a secure path (example: /mnt/secure-backups).
  • Directory layout: 
    /mnt/secure-backups/SYSTEMBACKUPS/
├── system/<host>/
└── web/<host>/<site>/
  • Retention: strict 7 days (time-based prune).

Scheduling

All jobs are defined in /etc/cron.d with flock locks and dedicated logs.

# /etc/cron.d/backup-platform
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MAILTO=root

# 01:05 - system core
5 1 * * *  root flock -n /run/lock/backup_system_core.lock /usr/local/sbin/backup_system_core.sh >>/var/log/backup_system_core.log 2>&1

# 01:25 - websites (multi-site)
25 1 * * * root flock -n /run/lock/backup_websites.lock /usr/local/sbin/backup_websites.sh >>/var/log/backup_websites.log 2>&1

Logs rotated daily for 7 copies:

# /etc/logrotate.d/backup-platform
/var/log/backup_system_core.log /var/log/backup_websites.log {
  daily
  rotate 7
  compress
  missingok
  notifempty
  create 0640 root adm
}

Implementation details

Site definitions

Each website has a small .env file under /etc/backup/sites.d. One site per file.

# /etc/backup/sites.d/example.com.env
SITE_NAME="example.com"
WEBROOT="/var/www/example.com"
DB_NAME=""              # empty - auto-detect from wp-config.php if WordPress
DB_DUMP_USER="root"     # use root client defaults for non-interactive dumps
# Optional for remote hosts:
# SSH_HOST="vps.example.net"
# SSH_USER="kevin"
# SSH_PORT=22

Credentials

Database dump authentication uses the root client defaults file. No plaintext passwords in scripts.

# /root/.my.cnf (0600, root:root)
[client]
user=root
password=<REDACTED>

System core script (outline)

#!/usr/bin/env bash
set -Eeuo pipefail
RETENTION_DAYS=7
DEST="/mnt/secure-backups/SYSTEMBACKUPS/system/$(hostname -s)"
mountpoint -q /mnt/secure-backups || { echo "Target not mounted"; exit 3; }
mkdir -p "$DEST"
dpkg --get-selections > "$DEST/dpkg-selections.$(date -u +%Y%m%dT%H%M%SZ).txt"
apt-mark showmanual > "$DEST/apt-manual.$(date -u +%Y%m%dT%H%M%SZ).txt" || true
snap list > "$DEST/snap-list.$(date -u +%Y%m%dT%H%M%SZ).txt" 2>/dev/null || true
tar -cpf "$DEST/etc.$(date -u +%Y%m%dT%H%M%SZ).tar" --xattrs --acls -C / etc
gzip -9 "$DEST"/etc.*.tar
# ... also /usr/local, /opt, /root
find "$DEST" -type f -mtime +$RETENTION_DAYS -delete

Websites script (outline)

  • Reads all /etc/backup/sites.d/*.env.
  • For each site: archive webroot and dump DB if detected.
  • Writes DB to a temp file and moves on success to avoid zero-byte dumps.
#!/usr/bin/env bash
set -Eeuo pipefail
RETENTION_DAYS=7
BASE="/mnt/secure-backups/SYSTEMBACKUPS/web/$(hostname -s)"
mountpoint -q /mnt/secure-backups || { echo "Target not mounted"; exit 3; }
for cfg in /etc/backup/sites.d/*.env; do
  source "$cfg"
  dest="$BASE/$SITE_NAME"; mkdir -p "$dest"
  # webroot with sensible excludes
  tar --exclude='wp-content/cache/*' --exclude='wp-content/uploads/cache/*' \
      -cpf "$dest/$SITE_NAME.webroot.$(date -u +%Y%m%dT%H%M%SZ).tar" -C / "${WEBROOT#/}"
  gzip -9 "$dest/$SITE_NAME.webroot."*.tar
  # DB autodetect for WordPress
  if [[ -f "$WEBROOT/wp-config.php" ]]; then
    DBN=$(sed -n "s/.*DB_NAME.*'\\([^']*\\)'.*/\\1/p" "$WEBROOT/wp-config.php" | head -n1)
    DBH=$(sed -n "s/.*DB_HOST.*'\\([^']*\\)'.*/\\1/p" "$WEBROOT/wp-config.php" | head -n1)
    DUMP_USER="${DB_DUMP_USER:-root}"
    if [[ -n "$DBN" ]]; then
      tmp="$dest/$SITE_NAME.$DBN.$(date -u +%Y%m%dT%H%M%SZ).sql.tmp"
      mysqldump -h "${DBH:-localhost}" -u "$DUMP_USER" --single-transaction --skip-lock-tables --routines --events "$DBN" > "$tmp"
      gzip -9 "$tmp" && mv -f "$tmp.gz" "${tmp%.tmp}.gz"
    fi
  fi
  find "$dest" -type f -mtime +$RETENTION_DAYS -delete
done

Note: paths and timings in this article are examples. Adjust to your environment.

Verification and health checks

# Confirm destination is mounted
mountpoint -q /mnt/secure-backups

# Run both jobs on demand
sudo /usr/local/sbin/backup_system_core.sh
sudo /usr/local/sbin/backup_websites.sh

# Inspect artifacts
tree -L 2 /mnt/secure-backups/SYSTEMBACKUPS/system/$(hostname -s)/
tree -L 2 /mnt/secure-backups/SYSTEMBACKUPS/web/$(hostname -s)/

# Sanity check SQL header
zcat /mnt/secure-backups/SYSTEMBACKUPS/web/$(hostname -s)/example.com/*.sql.gz | head -n 20

Restore playbooks

Full host rebuild

  1. Install base OS, mount encrypted backup target.
  2. Restore selective /etc content: Apache vhosts, PHP config, certificates, SSH, firewall, cron.
  3. Reinstall packages using the saved lists if you want to accelerate base setup.
  4. For each site: restore webroot tar, set ownership to web user, import SQL dump.
  5. Validate: apachectl -t, site logins, permalinks, media.

Single site recovery

  1. Restore that site’s webroot tar.
  2. Import the matching SQL dump.
  3. Reload Apache. Test.

Rule of thumb: restore the last good snapshot. If you need point-in-time precision, add binary log capture to your MariaDB stack as a complementary enhancement.

Security notes

  • Use an encrypted destination and keep it physically controlled.
  • Store DB client credentials only in the root defaults file with strict permissions.
  • Avoid embedding secrets in scripts or .env files.
  • Sanitise all documentation before publishing. This article omits environment-specific secrets by design.

Results and lessons learned

  • Full nightly snapshots are predictable and easy to restore under pressure.
  • Mount guards prevent writing “backups” to the wrong disk. Obvious, yet commonly missed.
  • Atomic SQL dumps (write to temp, then move) eliminate zero-byte artifacts.
  • Test restores monthly. If you never test, you do not have backups — you have files you hope are useful.

Next steps

  • Optional: add MariaDB binary logs for point-in-time recovery.
  • Optional: add a deduplicated off-site tier with Borg or Restic (encrypted) if your threat model demands it.
  • Automate a monthly restore to a dev environment and report success or failure.
Copyright © Kevin Wells. This article is part of the Lab Projects series at kevwells.com.