Home / AWS / AWS cloud troubleshooting: AWS Secrets Manager Rotation Lambda Timed Out Befo

AWS

AWS cloud troubleshooting: AWS Secrets Manager Rotation Lambda Timed Out Befo

Fix Secrets Manager rotation failures when the rotation Lambda times out during create, set, test, or finish steps because of low timeout values, network issues, or slow database operations.

Published: Jan 23, 20269 min readBy FixWikiHub Editorial Team

Abstract illustration for a troubleshooting knowledge base category.

Introduction

AWS Secrets Manager provides automatic secret rotation capabilities through Lambda functions that implement a four-step rotation process: create a new secret version, set the new secret in the target system, test that the new secret works, and finalize the rotation by updating version labels. This multi-step workflow ensures that secrets remain valid throughout the rotation process and that the target system is always accessible.

When the rotation Lambda function times out during any of these steps, the secret can be left in an inconsistent state. The rotation process may have partially completed, leaving the secret with pending changes or version labels that don't reflect the actual state of the target system. Subsequent rotation attempts can fail or behave unexpectedly until the rotation state is cleaned up.

Understanding the rotation workflow and timing requirements is essential for configuring Lambda functions appropriately. Different target systems (databases, APIs, services) have different performance characteristics, and the Lambda timeout must accommodate the slowest expected operation plus a safety margin.

Symptoms

When Secrets Manager rotation Lambda times out, you will observe these symptoms:

Rotation fails with a Lambda timeout error in CloudWatch Logs or Secrets Manager events
The secret shows a failed or stuck rotation state in the AWS Console
CloudWatch Logs for the rotation Lambda stop during setSecret, testSecret, or finishSecret steps
Retrying rotation immediately causes conflict errors or continues partial progress
The secret version labels may show unexpected states like AWSPENDING stuck in place
Target system credentials may be in an inconsistent state (old password in secret, new in database, or vice versa)

Common error messages in Secrets Manager events:

bash

Secrets Manager cannot rotate this secret. The Lambda function timed out during the rotation process.
Error: Task timed out after 30.00 seconds

Lambda CloudWatch Logs showing timeout:

bash

INIT_START Runtime Version: python:3.9...
START RequestId: abc-123
[rotation steps executing...]
END RequestId: abc-123
REPORT RequestId: abc-123 Duration: 30000.00 ms Billed Duration: 30000 ms Memory Size: 128 MB Max Memory Used: 78 MB
2024-01-15T10:30:45.123Z Task timed out after 30.00 seconds

Common Causes

Several factors cause Secrets Manager rotation Lambda timeouts:

1.Lambda timeout too low: The default 30-second timeout is often insufficient for real rotation logic, especially when rotation involves database connections, authentication, and credential changes. Complex rotations involving multiple systems or slow networks need significantly more time.
2.VPC connectivity issues: If the Lambda function is in a VPC but cannot reach the target database or service due to security group rules, route table issues, or VPC endpoint problems, connection attempts consume timeout budget before failing.
3.Slow database credential changes: Some databases have slow credential update operations, especially with password complexity requirements, replication lag, or when multiple instances must be updated sequentially.
4.Cold start overhead: Lambda cold starts can consume several seconds, eating into the timeout budget before rotation logic even begins. This is especially problematic for functions with large deployment packages or VPC configurations.
5.Memory underprovisioning: Lambda functions with low memory allocation (128MB) have slower CPU performance, which can slow down network operations, encryption/decryption, and database drivers.
6.Rotation step hanging: A specific rotation step (typically setSecret when updating credentials in the database) may hang due to database locks, connection pool exhaustion, or network timeouts.
7.Partial rotation state: If a previous rotation attempt failed partway through, the current attempt may encounter unexpected state, leading to longer execution or errors.

Step-by-Step Fix

Follow these steps to diagnose and resolve rotation Lambda timeout issues:

Step 1: Check the Lambda function configuration

Examine the current timeout and memory settings:

```bash # Get current Lambda configuration aws lambda get-function-configuration \ --function-name SecretsManagerRotation-my-secret

# Look for: # Timeout: Current timeout in seconds # MemorySize: Current memory allocation # VpcConfig: Whether function runs in VPC ```

Output example:

json

{
  "FunctionName": "SecretsManagerRotation-my-secret",
  "Timeout": 30,
  "MemorySize": 128,
  "VpcConfig": {
    "SubnetIds": ["subnet-12345", "subnet-67890"],
    "SecurityGroupIds": ["sg-abcdef"]
  }
}

Step 2: Increase Lambda timeout and memory

Increase timeout to accommodate the full rotation workflow:

```bash # Increase timeout to 2 minutes and memory to 256MB aws lambda update-function-configuration \ --function-name SecretsManagerRotation-my-secret \ --timeout 120 \ --memory-size 256

# For complex rotations (multi-region, slow databases), use longer timeouts aws lambda update-function-configuration \ --function-name SecretsManagerRotation-my-secret \ --timeout 300 \ --memory-size 512 ```

Recommended timeout values based on scenario: - Simple database rotation: 60-120 seconds - Multi-AZ database rotation: 120-180 seconds - API/service rotation: 60-90 seconds - Complex multi-step rotation: 180-300 seconds

Step 3: Verify VPC connectivity

Ensure the Lambda can reach the target system:

```bash # Check Lambda's VPC configuration aws lambda get-function-configuration \ --function-name SecretsManagerRotation-my-secret \ --query 'VpcConfig.{Subnets:SubnetIds,SecurityGroups:SecurityGroupIds}'

# Verify security group allows outbound to database port aws ec2 describe-security-groups \ --group-ids sg-abcdef \ --query 'SecurityGroups[0].IpPermissionsEgress'

# Test connectivity by invoking Lambda with test event aws lambda invoke \ --function-name SecretsManagerRotation-my-secret \ --payload '{"test": "connectivity"}' \ response.json ```

For RDS targets, verify connectivity:

```bash # Check RDS is in correct VPC and subnet aws rds describe-db-instances \ --db-instance-identifier my-database \ --query 'DBInstances[0].{VpcId:DBSubnetGroup.VpcId,SubnetGroup:DBSubnetGroup.DBSubnetGroupName}'

# Verify security group allows Lambda to connect aws ec2 describe-security-groups \ --group-ids sg-rds-security-group \ --query 'SecurityGroups[0].IpPermissions[?FromPort==3306]' ```

Step 4: Inspect CloudWatch Logs for the failing step

Identify which rotation step times out:

```bash # Tail Lambda logs aws logs tail /aws/lambda/SecretsManagerRotation-my-secret --follow

# Or get recent logs aws logs filter-log-events \ --log-group-name /aws/lambda/SecretsManagerRotation-my-secret \ --start-time $(date -u -d '1 hour ago' +%s)000

# Search for specific step names in logs aws logs filter-log-events \ --log-group-name /aws/lambda/SecretsManagerRotation-my-secret \ --filter-pattern "setSecret" ```

1.The four rotation steps to look for:
2.createSecret - Creates new secret version
3.setSecret - Updates target system with new credential
4.testSecret - Validates new credential works
5.finishSecret - Updates version labels

Step 5: Check for stuck rotation state

Examine secret version labels and rotation state:

```bash # Get secret details including version labels aws secretsmanager describe-secret \ --secret-id my-secret \ --query '{VersionIdsToStages:VersionIdsToStages,RotationEnabled:RotationEnabled,LastRotatedDate:LastRotatedDate}'

# Check for AWSPENDING label stuck on a version aws secretsmanager get-secret-value \ --secret-id my-secret \ --version-stage AWSPENDING ```

Clean up stuck rotation state if needed:

```bash # Cancel any in-progress rotation aws secretsmanager cancel-rotate-secret \ --secret-id my-secret

# Manually remove AWSPENDING label if stuck aws secretsmanager update-secret-version-stage \ --secret-id my-secret \ --version-stage AWSPENDING \ --remove-from-version-id <version-id> ```

Step 6: Test rotation manually

After configuration changes, test rotation:

```bash # Trigger immediate rotation aws secretsmanager rotate-secret \ --secret-id my-secret \ --rotation-lambda-arn arn:aws:lambda:us-east-1:123456789012:function:SecretsManagerRotation-my-secret

# Monitor the rotation aws secretsmanager describe-secret \ --secret-id my-secret \ --query 'LastRotatedDate' ```

Verification

After applying fixes, verify rotation works correctly:

```bash # Check secret status aws secretsmanager describe-secret \ --secret-id my-secret \ --query '{RotationEnabled:RotationEnabled,LastRotatedDate:LastRotatedDate,NextRotationDate:NextRotationDate}'

# Verify Lambda executed successfully aws logs filter-log-events \ --log-group-name /aws/lambda/SecretsManagerRotation-my-secret \ --filter-pattern "finishSecret" \ --start-time $(date -u -d '10 minutes ago' +%s)000

# Check that target system accepts the new credential # Use application-specific verification ```

Prevention

To prevent rotation Lambda timeout issues:

1.Size timeout for end-to-end rotation: Consider all steps including network latency, database operations, and cold starts. Add a 50% safety margin to measured execution time.
2.Keep rotation Lambda close to targets: Deploy Lambda in the same VPC and Availability Zones as the target systems to minimize network latency.
3.Monitor rotation as secret hygiene: Set up CloudWatch alarms for rotation failures and track rotation duration trends.

bash

# Create CloudWatch alarm for rotation failures
aws cloudwatch put-metric-alarm \
  --alarm-name secrets-manager-rotation-failures \
  --metric-name RotationFailed \
  --namespace AWS/SecretsManager \
  --statistic Sum \
  --period 300 \
  --threshold 1 \
  --comparison-operator GreaterThanOrEqualToThreshold

1.Test rotation workflows manually: Before enabling scheduled rotation, manually trigger rotation and verify the complete workflow executes successfully.
2.Use provisioned concurrency for critical secrets: For secrets that cannot tolerate rotation delays, consider Lambda provisioned concurrency to eliminate cold starts.
3.Implement rotation step logging: Ensure the rotation Lambda logs each step with timestamps to quickly identify which step is slow or hanging.

[AWS troubleshooting: Fix IAM Permission Denied - Complete Tro](fix-iam-permission-denied)
[AWS cloud troubleshooting: AWS ACM Certificate Pending Validation Because the](aws-acm-certificate-pending-validation-wrong-route53-zone)
[AWS cloud troubleshooting: AWS ALB Returns 502 Because the Target Closed the ](aws-alb-502-target-closed-connection-keepalive-timeout-mismatch)
[AWS cloud troubleshooting: Fix AWS ALB CreateListener TargetGroupNotFound Err](aws-alb-createlistener-targetgroupnotfound)
[AWS cloud troubleshooting: Fix Aws Alb Lambda 502 Bad Gateway Issue in AWS](aws-alb-lambda-502-bad-gateway)

Was this guide helpful?

Related search paths

People also search for

If the symptom is close but not identical, these search paths usually surface the right neighboring fixes faster than scrolling the full archive.

AWS cloud troubleshooting: AWS Secrets Manager Rotation Lambda Timed Out Befo AWS cloud troubleshooting: AWS Secrets Manager Rotation Lambda Timed Out Befo AWS AWS cloud troubleshooting: AWS Secrets Manager Rotation Lambda Timed Out Befo troubleshooting AWS cloud troubleshooting: AWS Secrets Manager Rotation Lambda Timed Out Befo fix Fix Secrets Manager rotation failures when the rotation Lambda times out during create, set, test, or finish steps because of low timeout values, network issues, or slow database operations AWS Fix Secrets Manager rotation failures when the rotation Lambda times out during create, set, test, or finish steps because of low timeout values, network issues, or slow database operations

Explore Related Topics

Browse Guides from Other Categories

Discover troubleshooting guides from related categories to expand your knowledge.

FAQ

AWS Troubleshooting FAQs

Common questions about troubleshooting and preventing similar issues

How do I know if this aws-errors troubleshooting guide applies to my situation?

This guide is designed for aws-errors issues. If you're experiencing similar symptoms described in the article, follow the step-by-step instructions. Start with the most common causes and work through the diagnostic process.

Is it safe to follow these aws-errors troubleshooting steps?

Yes, all steps are designed to be safe and non-destructive. We recommend creating backups before making significant changes and testing each step before proceeding to the next.

How long does it typically take to resolve this type of aws-errors issue?

Most aws-errors issues can be resolved within 30 minutes to 2 hours, depending on the complexity and root cause. Follow the troubleshooting flow to identify and fix the problem efficiently.

How can I prevent this aws-errors issue from happening again?

Regular maintenance, monitoring, and following best practices for aws-errors configuration can help prevent recurrence. Consider implementing automated checks and alerts for early detection.

Written by

FixWikiHub Editorial Team

Our editorial team consists of experienced DevOps engineers, systems administrators, and cloud architects with hands-on experience in production environments across AWS, Azure, GCP, and on-premises infrastructure.

Every guide undergoes technical review for accuracy and is updated when software versions, commands, or best practices change.

Last updated: Jan 23, 2026

About our team

Important Notice

Disclaimer & Safety Guidelines

The troubleshooting steps in this guide are provided for educational and informational purposes. Before applying any changes to production systems:

Test in a staging environment first — Always verify commands and configurations in a non-production environment before deploying to live systems.
Create backups — Ensure you have current backups of databases, configurations, and critical files before making changes.
Understand the impact — Review how each step may affect your specific environment, dependencies, and users.
Consult official documentation — This guide supplements, but does not replace, official vendor documentation and best practices.

FixWikiHub is not responsible for any damages arising from the use of this content. See our Terms of Use for more information.

Resources

Official Documentation & Further Reading

For authoritative information, consult the official documentation for the technologies discussed in this guide. Our troubleshooting content supplements, but does not replace, vendor documentation.

AWS Documentation — Official Amazon Web Services guides and API references
Kubernetes Documentation — Official Kubernetes documentation
Nginx Documentation — Official Nginx web server documentation
Apache Documentation — Official Apache HTTP Server documentation
Docker Documentation — Official Docker container documentation

AWS cloud troubleshooting: AWS Secrets Manager Rotation Lambda Timed Out Befo

Introduction

Symptoms

Common Causes

Step-by-Step Fix

Step 1: Check the Lambda function configuration

Step 2: Increase Lambda timeout and memory

Step 3: Verify VPC connectivity

Step 4: Inspect CloudWatch Logs for the failing step

Step 5: Check for stuck rotation state

Step 6: Test rotation manually

Verification

Prevention

People also search for

Browse Guides from Other Categories

WordPress

SSL

DNS

AWS Troubleshooting FAQs

FixWikiHub Editorial Team

Disclaimer & Safety Guidelines

Official Documentation & Further Reading

AWS cloud troubleshooting: AWS Secrets Manager Rotation Lambda Timed Out Befo

Introduction

Symptoms

Common Causes

Step-by-Step Fix

Step 1: Check the Lambda function configuration

Step 2: Increase Lambda timeout and memory

Step 3: Verify VPC connectivity

Step 4: Inspect CloudWatch Logs for the failing step

Step 5: Check for stuck rotation state

Step 6: Test rotation manually

Verification

Prevention

Related Articles

People also search for

Share this guide

More AWS Troubleshooting Guides

Browse Guides from Other Categories

AWS Troubleshooting FAQs

FixWikiHub Editorial Team

Disclaimer & Safety Guidelines

Official Documentation & Further Reading