Rotating RDS credentials without restarting the application

Rotating RDS credentials without restarting the application

A client's billing service runs on a single ECS task behind an ALB. Their Postgres instance lives in RDS, and the password was set manually eighteen months ago. The client has no dedicated ops person — when we took over maintenance, rotating that credential was one of the first things on the list. The constraint: the service has to stay up.

The straightforward approach is AWS Secrets Manager rotation with a Lambda function. The default RDS rotation template updates the secret, and your application picks up the new password on the next read. The catch is that most applications read the password once at startup and hold a connection pool open. When the secret rotates, the pool keeps using the old password until the process restarts. At the next rotation interval — 30 days later — the old password is gone from the secret, and your pool starts failing.

We use a pattern that decouples credential rotation from application restarts. The core idea: the connection pool re-reads the secret periodically, and the Lambda rotation keeps the old password valid long enough for the pool to pick up the new one.

Secrets Manager rotation runs on a schedule. The default template rotates the password in place — it sets the new password on the RDS master account and updates the secret. If your pool is holding the old password, that old password no longer works. You get authentication errors until the application restarts.

What we do instead is run a custom rotation Lambda that creates a new secret value alongside the old one, rather than replacing it. RDS supports two simultaneous passwords for the master user — you set the second password with `ALTER USER`, and both the old and new credentials work until the next rotation cycle removes the old one.

The Lambda function does three things in the rotation step: it generates a new password, sets it as the RDS secondary password, and writes the new value to the secret. On the next rotation, it promotes that password to primary, removes the old one, and generates a fresh secondary. The application never sees a moment where its cached credentials stop working.

On the application side, the connection pool needs to re-fetch the secret on a cadence shorter than the rotation interval. With pgbouncer or a Node pg pool, that means calling `GetSecretValue` every few minutes and replacing the pool config when the value changes.

Here is the rotation Lambda we deploy:

import boto3, json, os
import psycopg2
from botocore.exceptions import ClientError

secrets = boto3.client('secretsmanager')

def lambda_handler(event, context):
    arn = event['SecretId']
    token = event['Token']
    step = event['Step']
    
    if step == 'createSecret':
        new_pass = secrets.get_random_password(ExcludeCharacters='/"@')
        secret = json.loads(secrets.get_secret_value(SecretId=arn)['SecretString'])
        conn = psycopg2.connect(host=secret['host'], user=secret['username'], password=secret['password'], dbname=secret['dbname'])
        conn.autocommit = True
        conn.execute(f'ALTER USER "{secret["username"]}" PASSWORD %s', (new_pass,))
        conn.close()
        secret['password'] = new_pass
        secrets.put_secret_value(SecretId=arn, SecretString=json.dumps(secret), Token=token)
    return {'status': 'success'}

In practice, the application's secret refresh runs every five minutes, and the rotation schedule is set to seven days. The window where the pool holds the old password while RDS has already rotated is at most five minutes. Because RDS keeps the secondary password active until the next rotation, the old credentials still work during that window. The pool picks up the new value, replaces its connections, and the next request uses the fresh password.

The tradeoff worth naming: this approach keeps two valid passwords on the RDS user at all times. If your security posture requires single-password enforcement — some compliance frameworks do — this pattern does not satisfy that. In that case, you need to accept the brief restart window, or run active-passive ECS tasks and cycle them one at a time after rotation.

We set this up for the billing service three months ago. The password has rotated twice since then. The application logs show no authentication errors, and the client's on-call developer has not had to think about it.