How to Test That Your VPS Backup Actually Restores

Published on

Run a restore drill on a spare VPS or in a scratch folder: fetch the backup, check checksums and databases, start the app, time it and write a runbook.

A VPS panel showing disk usage at 0.96 of 120 GB and a bandwidth meter with IN and OUT at 0%

Your backup job reports success every night. That only proves something was written somewhere. It doesn't prove you can get the data back without the old server, that it holds everything your app needs, or how long your site would be down while you rebuild. A restore drill answers all three: you restore last night's backup somewhere safe, check it, start the app on it and time every step. This guide runs one on Ubuntu 24.04 and Debian 12 and ends with a runbook for the day it's real.

Quick answer: Restore the backup where it can't touch live data: a spare VPS, or a scratch folder such as /srv/restore on the live server, never over /. Fetch it from the off-site copy using only the credentials in your password manager. Check the restored files against a checksum list made at backup time with sha256sum -c, load the database dumps and look at the newest row, and run pg_amcheck, mariadb-check or SQLite's PRAGMA integrity_check on database files you restored as files. Then start the app, open it under its real hostname with curl --resolve, compare the total time with how long you can afford to be down, and write the steps into a runbook you keep off the server.

What a Drill Proves

Each check answers a different question, and only the later ones show that the backup is usable:

CheckWhat it shows
restic check, gzip -tThe backup can be read, not that it's complete
Checksums against a list made at backup timeThe restored files match the originals
Loading the dumps, integrity checksThe data loads and is consistent, and how old it is
The app runs on the copyNothing it needs was left out
A timestamp on every stepHow long recovery really takes

PostgreSQL's manual makes the same point about its own backup checker: even if you use pg_verifybackup, "you should still perform test restores and verify that the resulting databases work as expected and that they appear to contain the correct data."

Write two targets down before you start. NIST calls them the recovery point objective, the point in time your data must be restored to, and the recovery time objective, how long recovery may take before it hurts the business. In plain words: how much recent data you can lose, and how long you can be down.

Step 1: Pick a Target and Start a Log

Restore onto a spare VPS if you can. It's the only test that proves you can rebuild without the old server, and it keeps the restore away from your live data. Give it the same OS release as the live server and enough disk for the restored data. A Space-Node VPS is KVM with full root access, and you choose Ubuntu 24.04 LTS or Debian on the order form, so a drill server can match yours.

No spare server? Restore into a scratch folder and scratch databases on the live one, but check the space first, because a full disk takes the live site down too. In a root shell with your restic settings loaded, as in our restic guide:

restic stats latest     # "Total Size" is what a full restore writes
df -h /srv

By default, restic stats counts the size of the restored files. Then start a log. This function writes a UTC timestamp and a label to ~/drill.log; call it after every step:

mark() { echo "$(date -u +%T) $*" | tee -a ~/drill.log; }
mark "start"

Step 2: Fetch the Backup as if the Server Were Gone

Use only what you'd still have after losing the server: the off-site copy and the credentials in your password manager. Don't copy keys or passwords over from the live VPS; if something is missing, that's your first finding. On a drill server, set up restic as the restic guide's section on losing the VPS describes, then restore:

restic snapshots --latest 1
restic restore latest --target /srv/restore --verify
mark "backup restored"

--latest 1 lists the newest snapshot of each host and path. --target puts the files under /srv/restore with their full paths, and --verify then checks the content of every restored file. Never use --target / in a drill: the restore docs say restore overwrites existing files by default. Keep tar archives? gzip -t checks a compressed file's integrity before you extract it.

Step 3: Check the Files

restic's checks prove that the restore matches the backup, not that the backup holds the right files. For that, let the backup job write a checksum list of your important folders just before each run. With the restic service from our guide, add this line above its ExecStart= lines:

ExecStartPre=-/bin/sh -c 'cd /var/www && find . -type f -print0 | xargs -0 sha256sum > /var/backups/www.sha256'

The - makes systemd treat a failure of this line as success, so the backup still runs (systemd.service). Hashing reads every file, so with big upload folders, list only the folders that matter. In the drill, check the restored files against the restored list:

cd /srv/restore/var/www
sudo sha256sum --quiet -c /srv/restore/var/backups/www.sha256 && echo ALL MATCH
sudo find /srv/restore \( -nouser -o -nogroup \) | head

With --quiet, sha256sum prints only the files that fail. FAILED means the content differs from when the list was made. FAILED open or read means the file isn't in the backup, usually because of an include or exclude rule, and the check ends with sha256sum: WARNING: 1 listed file could not be read. A file that changed between the list and the backup, such as a log, fails too, so leave those folders out of the list. The find line prints files whose numeric owner has no user on this server: restic restores owners by their numeric IDs (node.go), which matters on a new server.

Step 4: Rebuild the Server Around the Data

Skip this step when you restored into a scratch folder on the live server. On a new server, copy back only what you need from /srv/restore, as the restic guide explains, so the new server keeps its own disk and network settings. Work in this order.

Packages. Let the backup job save your package list, with a second line like the one in Step 3: ExecStartPre=-/bin/sh -c 'apt-mark showmanual > /var/backups/apt-manual.txt'. On the new server, add your third-party repositories (Docker, the PostgreSQL apt repository) as their install guides say, then reinstall the list. A missing repository stops apt-get with E: Unable to locate package.

sudo apt-get update
sudo xargs -a /srv/restore/var/backups/apt-manual.txt apt-get install -y

Users. Debian Policy keeps IDs 0 to 99 the same on every Debian system, but users that packages create, such as postgres, get a free ID between 100 and 999 that can differ per server. Compare the two lists; lines starting with < come from the old server, > from the new one:

diff <(cut -d: -f1,3 /srv/restore/etc/passwd | sort) <(cut -d: -f1,3 /etc/passwd | sort)
sudo useradd -m -u 1001 -s /bin/bash appuser

Create your own accounts with their old ID, as in the second line. Where a package user got another ID, fix the owner of its restored files after copying, for example sudo chown -R postgres:postgres on its folder.

Configuration. Keep the copy from acting as the live server: a restored backup timer would add the drill server's files to your real repository, and cron jobs may send mail or call other services. Stop cron first, then copy back per app: nginx, your systemd units, certificates, crontabs and app folders. A trailing slash on the source makes rsync copy the folder's contents. Then disable the backup timer you restored:

sudo systemctl disable --now cron.service
sudo rsync -a /srv/restore/etc/nginx/ /etc/nginx/     # and so on, per app
sudo systemctl daemon-reload
sudo systemctl disable --now restic-backup.timer
systemctl list-timers
mark "server rebuilt"

In a real recovery, enable cron and the backup timer again at the end.

Step 5: Load and Check the Databases

Load each dump the way our PostgreSQL and MariaDB guides do: into a scratch database on the live server, as in their monthly test, or for real on the drill server. A load without errors is the first check. The second is age, because the newest row shows how much a disaster right now would cost you:

sudo -u postgres psql -d restore_test -c 'SELECT max(created_at) FROM orders;'
mark "databases loaded"

Use your own database, table and timestamp column. If the answer is two days old, the nightly dump stopped running, and the drill found it before a disaster did.

Database files restored as files, from a disk snapshot, a copied data folder or pg_basebackup, need a structure check once the database server runs on them. Debian and Ubuntu don't put pg_amcheck on the PATH, so call it by its full path, with 15 instead of 16 on Debian 12:

sudo -u postgres /usr/lib/postgresql/16/bin/pg_amcheck --install-missing --all
sudo mariadb-check --all-databases --silent
sudo apt-get install sqlite3
sudo sqlite3 /srv/restore/srv/myapp/data.db 'PRAGMA integrity_check;'

pg_amcheck checks tables and B-tree indexes, and --install-missing adds the amcheck extension it needs. The amcheck docs name "faulty and ill-conceived backup and restore tools" among the causes of corruption seen in production, and warn that amcheck "can only prove the presence of corruption; it cannot prove its absence". mariadb-check runs CHECK TABLE across all databases, needs the server running, and with --silent prints only errors. SQLite's integrity_check returns a single ok when it finds nothing. Its corruption guide explains why a copied file can fail: a copy made during a transaction "might contain some old and some new content", and a -wal or -journal file must stay with its database.

Step 6: Start the App and Use It

A scratch-folder drill on the live server ends at Step 5, because a second copy of the app there would clash with the live one. On the drill server, start the app and read its log; for Docker, restore the volumes and start the stack as in our Docker volumes guide:

sudo systemctl enable --now nginx myapp
sudo journalctl -u myapp -n 50 --no-pager
mark "app started"

Now test it under its real name without touching DNS. From your own computer, with the drill server's IP address:

curl -sS -o /dev/null -w '%{response_code}\n' --resolve example.com:443:203.0.113.50 https://example.com/

--resolve makes curl use that address instead of the DNS answer; add -k if you didn't restore the TLS certificate. For a browser, add the line 203.0.113.50 example.com to your computer's hosts file and remove it afterwards. Log in, open the newest order or post and an uploaded file, then run mark "site works" on the drill server. The drill server holds a copy of your real data, so give it the same firewall as the live server and delete it when you're done.

Step 7: Time It and Write the Runbook

cat ~/drill.log shows when each step ended. The time from start to site works is your measured recovery time; compare it with how long you can be down. If you'd order a new server in a real disaster, add the time until it's ready and the DNS change our website move guide covers. If the total is too long, the log shows which step to shorten.

Then write the runbook while the details are fresh, and keep it where you can reach it without the server, such as your password manager's notes or a private repository:

RESTORE RUNBOOK: shop.example.com
Last drill: <date>, <total time>, <who>, <result>, newest data from <time>
Live server: <OS release>, <plan>, app in /srv/shop, PostgreSQL 16
Backups: restic sftp:backupbox:/srv/restic/vps1 at 04:30, dumps at 03:30
Secrets: password manager entries "restic vps1", "backupbox SSH", "DNS"
Steps, with the exact commands and how long each took:
1. Order a VPS with the same release, set up restic (restic guide)
2. restic restore latest --target /srv/restore --verify
3. ...
Known problems: postgres has another UID on a new server, chown after copy
Contacts: hosting support, DNS provider, app developer

Run the drill again after big changes, such as a new app, a new backup tool or an OS upgrade, and update the runbook each time. The runbook is one part of a disaster recovery plan; our disaster recovery checklist adds recovery targets, account access, contacts and the first hour of an outage.

If It Still Fails

  • sha256sum: WARNING: 12 computed checksums did NOT match. If your app changes those files, such as caches or logs, leave their folders out of the list. If they never change, the copy is damaged: restore from an older snapshot and read all stored data back with restic check --read-data.
  • FAILED open or read. The file isn't in the backup. Compare your include and exclude lists with the folder, as in Step 4 of the restic guide.
  • Permission denied in the app's log. The files belong to another ID on this server. Compare the users as in Step 4 and fix the owners.
  • The app starts empty. It reads another path, database or Docker volume than the one you restored. Compare its config with the restored copy; the Docker guide explains how Compose names volumes.
  • The disk fills up during the restore. Compare restic stats latest with df -h first, or restore one folder at a time with --include.
  • The newest row is days old. The dump job stopped. systemctl status on its service and the journal show why.

FAQ

How often should I run a restore drill? Restore one database and one folder every month, as our database and restic guides do, and run a full drill on a spare server a few times a year and after big changes.

Can I test on the live server? For files and databases, yes: restore into /srv/restore and scratch databases after checking the free space. It can't prove that you can rebuild without that server, though, so run a full drill elsewhere now and then.

Isn't restic check enough? No. It proves the repository is intact and readable. It can't show that the backup holds the right folders, that last night's dump is in it or that your app starts on it.

Does the drill server need to be as big as the live one? It needs enough disk for the restored data and enough memory to run the app. For the timing, use the same size as the live server, so the result matches what a real recovery would take.

Are backups included with a Space-Node VPS? Not in the plan price; you can arrange them through a support ticket. Whoever makes your backups, ask how a restore works, in place or onto another server, and drill it the same way.

Sources

All opened on 31 October 2026: