Building Resilient Services: Timeouts, Retries, and Graceful Shutdown
Keep a Node service healthy when its dependencies, network, and orchestrator are all misbehaving at once.
- drain in-flight work and shut down cleanly when the orchestrator sends
SIGTERM; - separate a liveness check from a readiness check and know which one gates traffic;
- put a hard timeout on every outbound dependency call using
AbortController; - retry only idempotent work with exponential backoff and jitter -- and know when NOT to retry;
- wrap an unreliable dependency in a circuit breaker so one sick backend cannot take you down.
This builds on Microservices Architecture (how services are decomposed) and Logging and Observability (how you see what is happening). Here we focus purely on staying alive under partial failure. You are expected to be comfortable with promises, async/await, and the Node event loop already.
1. Graceful shutdown: SIGTERM is a request, not a kill
In a container platform (Kubernetes, ECS, Nomad, or a bare docker stop), a restart is routine: a new deploy, a scale-down, a node drain, or a failed health check all cause your process to be stopped. The platform does this in two steps. First it sends SIGTERM -- a polite "please finish up and exit". Then, after a grace period (30 seconds by default in Kubernetes), if you are still running it sends SIGKILL, which cannot be caught, blocked, or handled and terminates you instantly.
The naive service ignores SIGTERM. Node's default action for that signal is to exit immediately, so every request currently being served is cut off mid-response, open database transactions are abandoned, and the client sees a connection reset. Graceful shutdown means catching SIGTERM and, in order:
- Stop accepting new work -- call
server.close(), which stops listening for new connections but lets in-flight requests finish. - Fail readiness -- start returning "not ready" so the load balancer stops routing new traffic to this instance.
- Drain in-flight requests -- wait for the currently running handlers to complete, bounded by a timeout.
- Close dependencies -- end database pools, flush logs, disconnect from the message broker.
- Exit with code 0.
Analogy. SIGTERM is a shop being asked to close. A graceful shop locks the front door so no new customers enter, serves the people already inside, cashes out the till, then turns off the lights. An ungraceful shop simply cuts the power with customers mid-transaction. SIGKILL is cutting the power -- there is nothing you can do about it, which is exactly why you want to be finished before it arrives.
SIGKILL eventually kills you mid-work anyway -- the worst of both worlds. Give the drain, say, 25 seconds (comfortably under the platform's 30s grace) and process.exit(1) if it is exceeded, having logged which requests were still open.
One more subtlety in Kubernetes: there is a race between the platform removing your pod from the Service endpoints and sending SIGTERM. For a brief window the load balancer may still send you traffic after you started shutting down. The usual fix is to sleep a couple of seconds after receiving SIGTERM (or fail readiness first and pause) before calling server.close(), so the endpoint removal has propagated.