Monitoring Kubernetes CronJobs
20 checks free, no card.
Kubernetes CronJobs create Jobs on a schedule, and Jobs create Pods. There are several layers where a run can quietly not happen. Cluster metrics will tell you a Job failed if you look. They won't tell you a Job was never created.
How CronJobs miss runs
startingDeadlineSeconds. If the controller can't start a Job within this window, that run is skipped. Without it set, a CronJob that misses too many schedules in a row (100, per the Kubernetes docs) stops scheduling and logs an error.
concurrencyPolicy: Forbid. If the previous run is still going, the new one is skipped. A hung Job can block every future run.
suspend: true. Someone suspended it during an incident and forgot to unsuspend it.
Image pull failures. A deleted tag or expired registry credential leaves Pods in ImagePullBackOff.
Resource limits. Pods stay Pending because the cluster has no room, or get OOMKilled partway through.
Time zones. Before the timeZone field existed, schedules followed the controller manager's zone, usually UTC. The field is now stable in current Kubernetes versions; older clusters may still use UTC regardless.
Adding a heartbeat
The simplest pattern: run your command, then ping on success.
apiVersion: batch/v1
kind: CronJob
metadata:
name: nightly-report
spec:
schedule: "0 3 * * *"
timeZone: "Europe/London"
concurrencyPolicy: Forbid
startingDeadlineSeconds: 600
jobTemplate:
spec:
backoffLimit: 2
template:
spec:
restartPolicy: Never
containers:
- name: report
image: registry.example.com/report:1.4.2
env:
- name: PING_URL
valueFrom:
secretKeyRef:
name: pingclock
key: nightly-report
command: ["/bin/sh", "-c"]
args:
- ./run-report && wget -qO- "$PING_URL"Store the ping URL in a Secret, so it isn't sitting in your manifests in plain text.
Better signals
Start and finish. Ping $PING_URL/start before the command to measure duration and catch hangs.
Exit codes. Replace && with ; wget -qO- "$PING_URL/$?" to report failures straight away.
Retries. With backoffLimit above zero, Kubernetes retries failed Pods. Only the final success pings, which is usually what you want.
Choosing the grace time
Account for image pulls, scheduling delay and retries. If a Job usually takes 5 minutes but can retry twice, a grace time of 20 to 30 minutes avoids false alarms.
What heartbeats catch that cluster alerts don't
- A suspended CronJob
- A CronJob deleted during a cleanup
- A namespace or cluster that's down entirely
- Runs skipped because of concurrency or deadline settings
- A schedule typo that runs at the wrong time
A quick audit for existing CronJobs
Run kubectl get cronjobs -A and look at the LAST SCHEDULE column. Anything older than its schedule implies is worth a closer look. Then check SUSPEND for any True values nobody remembers setting. Finally, list failed Jobs with kubectl get jobs -A and look for repeated failures from the same CronJob. Add a heartbeat to each one that matters before you move on.
Questions
Should I use curl or wget?
Whichever your image has. Minimal images often include neither; add one, or use your language's HTTP client in the job itself.
Can I manage checks from code alongside manifests?
On Pro and Business, use the API to create checks as part of your deployment.