Cron job monitoring, done properly
Free forever, no card.
Cron is reliable at starting things. It has no idea whether they worked. Monitoring a cron job means answering three questions: did it start, did it finish, and did it succeed? Here's how the common approaches stack up.
Approach 1: Cron's own email
If MAILTO is set and the server can send mail, cron emails any output the job produces.
- Catches: jobs that print errors.
- Misses: jobs that never run, jobs that fail silently, servers that can't send mail (most cloud VMs). Also trains people to ignore a daily email full of normal output.
Approach 2: Log files and log alerts
Redirect output to a log and alert on error lines with your logging tool.
- Catches: errors the job writes down.
- Misses: jobs that never start, so they never log anything. A crashed cron daemon, a deleted crontab or a server that's off produces no lines to alert on.
Approach 3: Checking the result
A separate script checks for the job's output: does today's backup file exist, is the table updated?
- Catches: the thing you actually care about.
- Misses: nothing in theory. In practice, you need to write and maintain a checker for every job, and the checker is itself a scheduled job that needs monitoring.
Approach 4: Heartbeat pings
The job sends a ping to a monitoring service when it succeeds. The service knows the schedule and alerts when a ping is late.
- Catches: jobs that don't run, jobs that fail before the ping, jobs that hang, dead cron daemons, servers that are off, deleted crontabs.
- Misses: jobs that exit 0 but did the wrong thing. Combine with a sanity check inside the script for that.
Heartbeats catch the widest range of failures for the least effort, which is why they've become the standard.
Setting up heartbeat monitoring
- Create a check for each job with its schedule. Use the same cron expression as the crontab and the server's time zone.
- Set a grace time longer than the job's normal run time plus a margin. A backup that usually takes 20 minutes might get 45.
- Add the ping after the command, joined with
&&so it only fires on success. - Optionally ping /start at the beginning, so you can tell "never started" from "started and hung", and get run durations.
- Optionally send the exit code with
/$?so failures alert right away instead of after the grace time. - Route alerts to wherever someone will see them: a team Slack channel beats one person's inbox.
What to monitor first
Prioritize jobs where silence is expensive:
- Backups and database dumps
- Certificate renewals
- Billing, invoicing and payment syncs
- Data imports and exports with partners
- Cleanup jobs that stop disks filling up
- Queue workers or schedulers that should run all the time
Keeping it maintainable
Name checks after the job and server ("db-backup @ prod-db-1"). Store ping URLs as environment variables, not hard-coded in scripts. When you delete a cron job, delete its check, or you'll get an alert you don't need.
How Ping Clock does it
Period or cron schedules with time zones, grace times, start/success/fail/exit-code pings, run durations and log tails on Pro, email-in pings for tools that can't make HTTP requests, and alerts to email, Slack, Teams, Discord or webhooks.
Questions
How many checks do I need?
One per scheduled job you care about. Most small teams have 10 to 50.
Can one check cover several servers running the same job?
Better to use one check per server, so you know which one went quiet.