Log retention settings that quietly become one of your largest bills

Log retention settings that quietly become one of your largest bills

Here is a pattern we have seen more than once. An AWS bill climbs from, say, Rs 18,000 to Rs 41,000 per month over three quarters. No new environments, no traffic spike, no additional EC2 instances. When someone finally opens Cost Explorer, the increase is CloudWatch Logs, and it is almost entirely from log groups set to never expire.

The default for a new CloudWatch log group is "Never expire." If you create log groups through the console and click through, that is what you get. If you create them via Terraform and omit `retention_in_days`, same thing. AWS will store those logs indefinitely, and they charge Rs 0.034 per GB per month for the storage. That sounds small. It is not, once you have a few high-throughput API endpoints logging full request and response bodies.

In an account like that, expect a couple of dozen log groups with a handful of them accounting for nearly all the cost. Each of those is typically a Lambda function behind API Gateway that logs the full event object, the full response, and a structured JSON block with user context. Work through one example: a webhook receiver hit 40,000 times a day, each log entry around 12 KB. That single function generates roughly 14 GB of logs per month, and with no retention set it stores all of it going back to the day it was deployed.

The fix has two parts, and only one of them is retention.

First, set retention. We apply 14-day retention to webhook-style log groups and 30-day retention to API log groups. The older data is almost never used — nobody reads logs from August in February. If you need longer retention for compliance or debugging, export to S3 first, where storage is Rs 0.024 per GB per month for Standard and drops to Rs 0.005 for Infrequent Access. CloudWatch Logs is for recent operational data, not an archive.

Second, stop logging so much. A webhook function does not need the full event body in every log entry. The request headers, the payload, and the response are useful during the first week of integration. A year later, they are noise. Trim the logging to the correlation ID, the status code, and the processing time, and the volume drops from 12 KB per invocation to under 1 KB.

Here is the Terraform block we use now for any new log group:

resource "aws_cloudwatch_log_group" "api_logs" {
  name              = "/aws/lambda/webhook-receiver"
  retention_in_days = 14

  tags = {
    ManagedBy = "terraform"
  }
}

If `retention_in_days` is missing, Terraform will not error. AWS will accept the create request and set retention to Never. This is the part that catches teams without a dedicated ops person — there is no signal that something is wrong until the bill arrives.

We also set up a billing alarm at Rs 500 incremental spend on CloudWatch, routed to a Slack channel. The alarm is coarse, but it catches the pattern where a new function starts logging heavily and nobody notices for two billing cycles.

With both changes in place, a CloudWatch Logs line in the low tens of thousands of rupees per month should drop to a small fraction of that, because you are now paying for two weeks of small entries instead of a year and a half of large ones. A 14-day window covers nearly every debugging scenario a small team actually hits. When someone does ask about a webhook failure from two months ago, the answer comes from the S3 export, not from CloudWatch, and that is fine as long as the export was set up before the retention change deleted the history.

If you are paying more than Rs 5,000 per month for CloudWatch Logs and you are not a high-frequency trading platform, check your retention settings first. Then check what you are logging. The retention is usually the larger number, but the logging volume is the root cause.