Log retention settings that quietly become one of your largest bills

Log retention settings that quietly become one of your largest bills

A logistics client in Ghaziabad asked us to look at their AWS bill, which had climbed from Rs 18,000 to Rs 41,000 per month over three quarters. No new environments, no traffic spike, no additional EC2 instances. The increase was CloudWatch Logs — Rs 14,200 per month, almost entirely from log groups set to never expire.

The default for a new CloudWatch log group is "Never expire." If you create log groups through the console and click through, that is what you get. If you create them via Terraform and omit `retention_in_days`, same thing. AWS will store those logs indefinitely, and they charge Rs 0.034 per GB per month for the storage. That sounds small. It is not, once you have a few high-throughput API endpoints logging full request and response bodies.

The client had 23 log groups. Four of them accounted for 91% of the cost. Each was a Lambda function behind API Gateway that logged the full event object, the full response, and a structured JSON block with user context. One of these functions was a webhook receiver hit 40,000 times a day, each log entry around 12 KB. That single function was generating roughly 14 GB of logs per month and storing all of it going back 18 months.

The fix has two parts, and only one of them is retention.

First, set retention. We applied 14-day retention to the webhook log group and 30-day retention to the three API log groups. The older data was not being used — nobody was reading logs from August in February. If you need longer retention for compliance or debugging, export to S3 first, where storage is Rs 0.024 per GB per month for Standard and drops to Rs 0.005 for Infrequent Access. CloudWatch Logs is for recent operational data, not an archive.

Second, stop logging so much. The webhook function did not need the full event body in every log entry. The request headers, the payload, and the response were useful during the first week of integration. Eighteen months later, they were noise. We trimmed the logging to the correlation ID, the status code, and the processing time. Log volume dropped from 12 KB per invocation to under 1 KB.

Here is the Terraform block we use now for any new log group:

resource "aws_cloudwatch_log_group" "api_logs" {
  name              = "/aws/lambda/webhook-receiver"
  retention_in_days = 14

  tags = {
    ManagedBy = "terraform"
  }
}

If `retention_in_days` is missing, Terraform will not error. AWS will accept the create request and set retention to Never. This is the part that catches teams without a dedicated ops person — there is no signal that something is wrong until the bill arrives.

We also set up a billing alarm at Rs 500 incremental spend on CloudWatch, routed to a Slack channel. The alarm is coarse, but it catches the pattern where a new function starts logging heavily and nobody notices for two billing cycles.

After the changes, the client's CloudWatch Logs cost dropped to Rs 1,800 per month. The 14-day window covers every debugging scenario they have actually needed. When they asked about a webhook failure from two months ago, we pulled it from the S3 export we had set up as part of the migration, not from CloudWatch.

If you are paying more than Rs 5,000 per month for CloudWatch Logs and you are not a high-frequency trading platform, check your retention settings first. Then check what you are logging. The retention is usually the larger number, but the logging volume is the root cause.