Ping Clock Ping Clock
Sign up

Get an alert when a backup doesn't happen

Free for up to 20 jobs.

The worst time to find out a backup stopped is when you need to restore it. Backups fail for dull reasons: a full disk, an expired storage credential, a password change, a cron entry lost in a server rebuild. None of these make noise. A heartbeat check does.

The basic setup

  1. Create a check with the backup's schedule, for example daily at 01:00 in your server's time zone.
  2. Set the grace time a bit longer than the backup usually takes. If it normally finishes by 01:40, a 90-minute grace is safe.
  3. Add the ping after the backup command, using && so it only pings on success.
0 1 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1 && curl -fsS -m 10 --retry 5 https://www.pingclock.com/ping/YOUR-CHECK-ID

If the backup fails, there's no ping. If cron never starts it, there's no ping. If the server is off, there's no ping. Each case ends in an alert.

Common backup tools

Database dumps (pg_dump, mysqldump): make sure the script exits non-zero on failure. With pipes, a failing dump can be hidden by a successful gzip. Use set -o pipefail in bash.

restic and borg: both return non-zero exit codes on errors, so && curl ... works directly. Consider a second check for the periodic check or prune job.

rsync: exit code 24 ("some files vanished") is often harmless; other non-zero codes are not. Handle 24 explicitly in your script if your data changes during the copy.

Cloud provider snapshots: if they're scheduled by the provider rather than by you, a heartbeat can't be attached directly. Run a small script that lists the latest snapshot and pings only if one exists from today.

Go beyond "it ran"

A job that exits 0 can still produce a useless backup. Add cheap sanity checks before the ping:

  • Size check. Fail if the file is suspiciously small compared with last time.
  • Age check. Confirm the newest file in the destination is from this run.
  • Integrity check. For archives, test that they open. For dumps, check the last line looks like a complete dump.
#!/bin/bash
set -euo pipefail
pg_dump mydb | gzip > /backups/mydb-$(date +%F).sql.gz
size=$(stat -c%s /backups/mydb-$(date +%F).sql.gz)
[ "$size" -gt 1000000 ] || exit 1

Capture the reason for failures

On Pro, POST the tail of the log with a fail ping, so the alert shows why:

/usr/local/bin/backup.sh > /tmp/backup.out 2>&1
code=$?
tail -n 20 /tmp/backup.out | curl -fsS -m 10 --data-binary @- https://www.pingclock.com/ping/YOUR-CHECK-ID/$code

Test restores

Monitoring tells you a backup was made. Only a restore tells you it's usable. Schedule a regular restore test into a scratch database or folder, and give that job its own heartbeat check.

Where to send backup alerts

Somewhere a person will act on it the same day. A shared Slack or Teams channel works better than one person's inbox, because people go on holiday.

Questions

What if the backup runs from a NAS or appliance that can't run curl?

If it can send an email when a job completes, use the check's email-in address instead. Each check has its own.

Can I see how long backups take?

Yes, on Pro and Business. Ping /start at the beginning and the normal URL at the end.

Create a backup check