Zero-downtime ECS deploys and the health check settings that make or break them
A deploy goes out. The new task registers healthy, the old task deregisters, and for about thirty seconds a slice of your traffic hits a container that's already shutting down. Users see 502s. The deploy was "successful" — ECS swapped the tasks, nothing rolled back — but the logs tell a different story.
The problem is almost never the deploy itself. It's the gap between what the load balancer believes is healthy and what the container is actually ready to handle.
ECS rolling deploys work by starting new tasks before stopping old ones, controlled by `minimumHealthyPercent` and `maximumPercent` on the service. If you set `minimumHealthyPercent` to 100 and `maximumPercent` to 200, ECS brings up new tasks alongside the old ones, waits for them to pass health checks, then drains and kills the old ones. That's the mechanics. Whether it actually prevents dropped requests depends on three settings most teams copy from a tutorial and never revisit.
The first is the ALB target group health check. The defaults are `interval: 30`, `timeout: 5`, `healthyThreshold: 3`, `unhealthyThreshold: 3`. That means a new task has to respond to three consecutive checks, thirty seconds apart, before the load balancer sends it traffic. Your deploy takes at least ninety seconds before the new task receives a single request — and if your health check path is a slow endpoint, it's longer. We point the health check at a dedicated `/health` route that hits the database with a `SELECT 1` and returns 200. Fast, meaningful, and it doesn't compete with real traffic.
The second setting is `deregistrationDelay`, also called connection draining on the target group. When ECS tells the ALB to deregister an old target, the ALB stops sending new requests but holds existing connections open for this delay period. The default is 300 seconds, which is fine — but some teams set it to 5 or 10 to "speed up deploys," and that's where the 502s come from. If your app has long-running requests (file uploads, report generation, slow third-party API calls) and the deregistration delay is shorter than your slowest request, the ALB closes the connection mid-flight.
The third setting is what your app does when it receives `SIGTERM`. ECS sends a termination signal to the container, and the default behavior of most web servers is to stop accepting new connections and exit immediately. But the ALB might still have connections it considers healthy. Your app needs to stop listening, let in-flight requests finish, then exit. In Node, that looks like refusing new connections and waiting for existing ones to drain:
process.on('SIGTERM', () => {
console.log('SIGTERM received, draining');
server.close(() => {
console.log('All connections closed');
process.exit(0);
});
setTimeout(() => {
console.error('Forcing exit after 30s');
process.exit(1);
}, 30000);
});The `setTimeout` is the failsafe — if something hangs, the container still exits and ECS recycles it. Without that, a stuck connection keeps the old task alive indefinitely and your deploy stalls.
The order matters. The sequence is: new task starts, passes health check, ALB sends it traffic, ECS deregisters old target, ALB stops new connections to old target, old target finishes in-flight requests, old task exits. If the health check is too lenient, the ALB sends traffic to a container that isn't ready. If the deregistration delay is too short, the ALB closes connections before they finish. If the app doesn't handle SIGTERM, in-flight requests die on exit.
We set `deregistrationDelay` to 60 seconds for most apps, which covers the vast majority of request lifetimes without dragging out deploys. For apps with known long-running requests, we bump it to 120 or 300. The health check interval we set to 10 seconds with a healthy threshold of 2 — a new task becomes eligible for traffic in about twenty seconds, which is fast enough for most teams and slow enough to catch a container that starts up but isn't actually ready.
The way to verify this is working is to watch the ALB's `RequestCount` and `HTTPCode_ELB_5XX` metrics during a deploy. If 5XX stays flat and old-target request count drops to zero gradually, the drain is working. If there's a spike at the moment the old task deregisters, something in the chain is wrong.