Zero-Downtime Deploys Without the Prayer

Most of the “zero-downtime” outages I’ve seen had nothing to do with the deploy strategy. They came from a container that said it was ready before it actually was.

The three gates that matter

A pod behind a rolling update goes through three checks. Get all three right and nobody notices the rollout happened.

The startup probe buys slow-booting apps some time before the liveness check starts judging them.

The readiness probe is the honest one. It should not return 200 until the app can actually serve traffic, which means warm caches and open DB pools, not just “the process is up.”

The preStop hook and terminationGracePeriodSeconds are the part everyone forgets. The pod needs to finish the requests it’s already handling before the kubelet sends SIGTERM.

lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 5"]
terminationGracePeriodSeconds: 30
readinessProbe:
  httpGet:
    path: /healthz/ready
    port: 8080
  periodSeconds: 3
  failureThreshold: 2

That sleep 5 looks pointless. It isn’t. It gives the endpoints controller time to pull the pod out of rotation before the process starts turning connections away. Without it you get a short window where traffic is still being routed to a pod that’s already shutting down, and that window is where your dropped requests come from.

The one thing to watch

Not CPU. Not memory. Watch the request error rate during the rollout window. If it stays flat, the deploy really was invisible. If it spikes for even ten seconds, one of those three gates is lying to you, and now you know which rollout to go look at.

If you find yourself watching a deploy nervously, refreshing a dashboard and hoping, that’s the signal that something in this list still isn’t wired up. A deploy you trust is one you don’t have to babysit.