Kubernetes Probes: Liveness, Readiness, and the Restarts They Cause

A pod restarts every few minutes. The team digs through application logs looking for a crash that never happened, because the app did not crash. A probe killed it, on schedule, exactly as configured.

Probes are three small HTTP checks with outsized power. One of them can restart your containers, one controls whether traffic reaches them, and one decides how long they get to start up. Wire them up carelessly and Kubernetes will faithfully convert a small dependency hiccup into a fleet-wide restart storm. Wire them up well and most users never notice a bad deploy or a deadlocked process.

Three probes, three different jobs

Startup probe answers one question: has this container finished starting? While it is still failing, Kubernetes leaves the other probes alone and simply waits. Once it succeeds, it never runs again for that container. Its job is to protect slow starters, apps that load caches, run migrations, or warm up a runtime before they can honestly answer anything.

Readiness probe answers: should traffic be sent to this pod right now? A failing readiness probe removes the pod from the Service endpoints. The container keeps running. Nobody restarts anything. When the probe passes again, the pod quietly rejoins the rotation. Readiness is a traffic valve, nothing more.

Liveness probe answers: is this process alive, or is it stuck in a state where it will never recover on its own? A failing liveness probe makes Kubernetes kill the container and start a new one. It is the only probe with a body count, and it should be treated with that level of respect.

Liveness: the probe that kills

The classic mistake is pointing liveness at an endpoint that checks dependencies:

livenessProbe:
  httpGet:
    path: /health
    port: 8080
  periodSeconds: 5
  failureThreshold: 3

That config looks harmless until you read what /health actually does. If the handler pings the database, then a 20 second database blip fails the probe three times and Kubernetes restarts the pod. Every pod. At the same time, because they all share the same dependency. The restart does nothing to fix the database, the fresh pods take time to warm up, and they land right back on the same struggling database. A probe designed to heal the system just amplified the outage.

The rule for liveness is strict and simple. It may only check the process itself: is the event loop turning, is the server accepting connections, is the one deadlock you actually fear being detected. It must never check a database, a cache, a queue, or another service. If a dependency is down, restarting your process cannot help, and liveness will happily restart it anyway.

Keep the liveness handler cheap and separate:

mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
})

An empty 200 looks too dumb to be right. It is right. The kubelet is asking whether this process deserves to die, and the honest answer requires no I/O at all.

Readiness: where dependency checks belong

Dependencies belong in readiness, because the consequence of failing readiness is gentle. A pod whose database connection is broken should stop receiving traffic, and it should also be allowed to sit there and recover when the database does:

mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
    ctx, cancel := context.WithTimeout(r.Context(), 500*time.Millisecond)
    defer cancel()
    if err := db.PingContext(ctx); err != nil {
        http.Error(w, "not ready", http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
})

Two details matter here. The check has its own short timeout, because a readiness handler that hangs is worse than one that fails. And the pod stays alive through the whole episode. When the database comes back, the next probe passes and traffic returns, with no restarts, no cold caches, no warm-up penalty.

Readiness is also the right tool during shutdown. A pod that is going away should fail readiness first, drain out of the endpoints, and only then exit. Without that, every deploy sends a slice of live traffic to containers that are already halfway out the door.

Startup: the forgotten probe

Slow-starting apps used to have two bad options. Set a generous initialDelaySeconds on liveness and accept that a genuinely deadlocked container takes forever to get restarted. Or set a tight one and watch Kubernetes kill pods that were merely still booting, which looks exactly like a crash loop and pages exactly like one too.

The startup probe exists to end that trade-off:

startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 5
  failureThreshold: 30

That gives the container 150 seconds of grace during which liveness and readiness stay out of the way. After the first success, the startup probe retires and the other two take over with their normal, tighter settings. If your app takes a minute to load reference data or warm a cache, this single block is the difference between a smooth rollout and a CrashLoopBackOff mystery.

The settings are math, not vibes

Every probe is three numbers, and the numbers multiply into behaviour you can predict:

periodSeconds times failureThreshold is your reaction time. Liveness at 5 seconds and 3 failures means 15 seconds of unresponsiveness gets a container killed. A long GC pause, a CPU throttle under a tight limit, or one slow synchronous operation can eat 15 seconds without the process being dead at all. Slow the probe down or raise the threshold until the budget covers your worst honest stall. Ten seconds and three failures, a 30 second budget, is a saner default for most services.

timeoutSeconds is part of the budget. A probe that allows a 10 second response on a 5 second period just stacks overlapping checks against a struggling pod. Keep probe timeouts short, one or two seconds, and make the handlers fast enough to deserve that.

Probe traffic is real traffic. A thousand pods probed every five seconds is 200 requests per second of synthetic load, forever, aimed at your own health endpoints. If those endpoints touch a database, you have built a small permanent load test against it. Cheap handlers stop being a style preference at that scale.

When every pod goes unready at once

Readiness has one failure mode worth planning for. If the dependency every pod checks goes down, every pod fails readiness together, and the Service ends up with zero endpoints. Traffic has nowhere to go at all, which can be worse than sending it to struggling pods that might still answer the cheap requests. Some teams accept this on purpose: failing fast at the load balancer is cleaner than a slow brownout. Others make the readiness check shallower, testing only the dependencies without which the pod is truly useless, so a partial failure degrades the fleet instead of emptying it.

Deploys create the same shape on a smaller scale. Readiness gates a rolling update: new pods receive no traffic until they pass, which is what you want, but it also means a bad readiness config can stall a rollout indefinitely while the old pods keep serving. When a deploy hangs with new pods running but never ready, read the readiness handler before you blame the image. Half the time the pod is healthy and the probe is checking something the new version was never going to have, like a route that moved or a port that changed.

The takeaway

Each probe has exactly one job. Startup buys slow apps time to boot. Readiness decides who gets traffic and is where dependency checks belong. Liveness decides who gets killed, and it should ask nothing more than whether the process itself is still alive.

If you take one thing from this: go read your liveness probe handlers today. If any of them opens a connection to something outside the process, move that check to readiness before it turns your next dependency blip into a restart storm.