CloudWatch alarms worth a 2am phone call
The third night after we handed off a new deployment, the client's on-call developer muted the entire CloudWatch alarm channel. He had been woken twice for an ALB TargetResponseTime spike that resolved itself in under a minute — a Java service doing its first GC pass after a cold deployment. By the time the RDS storage alarm fired at 4am, nobody saw it for six hours.
We have a rule for the infrastructure we maintain: an alarm that pages a human must represent a condition that human can actually fix, and the fix must be urgent enough to justify interrupting sleep. Everything else is a dashboard problem.
**What earns a phone call**
RDS database-connections hitting the instance limit. If new connections are being denied, the application is either leaking pools or traffic has crossed a threshold the instance can absorb. Both require intervention — either a failover, a scaling action, or a code rollback. Waiting until morning means the app stays down.
ALB 5xx rate above 5% sustained for two minutes. A brief spike during a deploy is normal. A sustained rate means the service is rejecting requests, and users are seeing errors. Two minutes filters out deploy noise without letting a real outage simmer.
Lambda throttles on a critical path. If the function handling payment webhooks starts throttling, retries pile up, and the downstream system gets a burst later that causes its own problems. Throttles on background jobs are less urgent — they just run slower — so we split these into two alarms with different severities.
RDS free storage under 10GB. Storage exhaustion is one of the few conditions that can corrupt a running database. The threshold depends on instance size, but the point is to page early enough that someone can extend storage before it hits zero, not after.
**What does not earn a phone call**
CPU utilization. A service running at 95% CPU for ten minutes is often just handling a traffic burst. If it is also serving requests within latency targets, high CPU is not an incident — it is capacity data. We put this on a dashboard and set a non-paging alarm that feeds into a weekly review.
TargetResponseTime under one second. This is the GC-pass problem. Transient latency spikes happen during deploys, autoscaling events, and connection pool warmup. If you page on every one, you train the on-call person to assume every alert is noise. If latency stays above your SLO for five minutes, that is different — but the threshold has to reflect actual user impact, not a generic 500ms target.
Lambda errors on non-critical paths. A report-generation function failing at 2am can retry during business hours. A payment-authorization function failing at 2am cannot. We tag functions by criticality and route alarms accordingly.
**The routing split**
We use two SNS topics. Pages go to a topic wired to the on-call rotation through PagerDuty or Opsgenie. Non-paging alarms go to a Slack channel that the team sees in the morning. The Slack channel gets noisy on purpose — it builds a record of what the system does under load, which informs capacity decisions later.
The cost of a false positive is not just lost sleep. It is the slow erosion of trust in the alarm system. Once the on-call person starts treating pages as probably-noise, the real incident at 3am gets acknowledged at 7am. We would rather have five alarms that always mean something than fifty that sometimes do.