Introduction
Nginx is successfully forwarding requests to the upstream backend server, but the backend is not responding within the configured timeout period. The request times out and Nginx returns an error to the client. This typically happens with slow backends, long-running processes, or overloaded servers.
Symptoms
```bash $ tail -f /var/log/nginx/error.log 2024/01/01 12:00:00 [error] upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.1.1, server: example.com, request: "GET /api/data HTTP/1.1", upstream: "http://127.0.0.1:3000/api/data"
$ curl http://localhost/api/data <html> <head><title>504 Gateway Time-out</title></head> <body> <center><h1>504 Gateway Time-out</h1></center> </body> </html>
# Sometimes 502 instead HTTP/1.1 502 Bad Gateway # With log: upstream prematurely closed connection
# Browser shows: # 504 Gateway Time-out # nginx ```
Common Causes
- 1.Slow backend processing: Backend takes longer than timeout to respond
- 2.Backend overloaded: Server under heavy load, queuing requests
- 3.Long-running operations: Database queries, file processing, API calls
- 4.Default timeout too short: 60 seconds may not be enough
- 5.Backend hanging: Application stuck or deadlocked
- 6.Network latency: Slow connection between Nginx and backend
- 7.Connection pool exhausted: No available backend connections
- 8.Backend crashed during request: Process died mid-response
Step-by-Step Fix
- 1.Check logs for specific error messages
- 2.Verify configuration settings
- 3.Test network connectivity
- 4.Review recent changes
- 5.Apply corrective action
- 6.Verify the fix
Step 1: Identify the Timeout Type
Different timeout errors indicate different issues:
```bash # Check error logs grep "timed out" /var/log/nginx/error.log
# Types of timeout errors: # (110: Connection timed out) - Cannot connect to upstream # (110: Connection timed out) while reading - Upstream not responding # (110: Connection timed out) while sending - Upstream not receiving
# Check current timeout settings grep -r "timeout" /etc/nginx/ | grep proxy nginx -T 2>/dev/null | grep timeout ```
Step 2: Check Backend Performance
Verify backend response times:
```bash # Test backend directly time curl http://127.0.0.1:3000/api/data
# Check backend logs tail -f /var/log/app/error.log
# Check backend process status ps aux | grep node ps aux | grep python systemctl status php-fpm
# Monitor backend resources top -p $(pgrep -f "node server") htop
# Check database connections # For PostgreSQL psql -c "SELECT count(*) FROM pg_stat_activity;" # For MySQL mysql -e "SHOW PROCESSLIST;" ```
Step 3: Increase Proxy Timeouts
Adjust timeout settings in Nginx:
```nginx location /api { proxy_pass http://backend;
# Connection timeout (time to establish connection) proxy_connect_timeout 300s;
# Read timeout (time to wait for response) proxy_read_timeout 300s;
# Send timeout (time to send request to upstream) proxy_send_timeout 300s;
# For very long operations proxy_read_timeout 600s; # 10 minutes } ```
Timeout recommendations: ```nginx # Fast APIs (< 5 seconds response) proxy_read_timeout 30s; proxy_connect_timeout 10s;
# Normal APIs (5-30 seconds) proxy_read_timeout 60s; proxy_connect_timeout 30s;
# Long operations (database queries, file processing) proxy_read_timeout 300s; proxy_connect_timeout 60s;
# Very long operations (reports, exports) proxy_read_timeout 600s; ```
Step 4: Configure Upstream Timeouts
Set timeouts in upstream block:
```nginx upstream backend { server 127.0.0.1:3000;
# Keepalive connections keepalive 32; keepalive_timeout 60s; keepalive_requests 100; }
server { location / { proxy_pass http://backend; proxy_http_version 1.1; proxy_set_header Connection "";
proxy_connect_timeout 60s; proxy_read_timeout 300s; } } ```
Step 5: Fix Slow Backend Operations
Optimize backend performance:
```bash # Identify slow operations # Check application logs for slow queries tail -f /var/log/app/slow.log
# Profile backend # Node.js node --prof server.js # Python python -m cProfile app.py
# Database optimization # Add indexes CREATE INDEX idx_column ON table(column);
# Use connection pooling # In app configuration pool_size: 10 max_overflow: 20
# Cache expensive operations # Implement Redis caching ```
Backend timeout configuration: ```javascript // Node.js Express app.use('/api/data', (req, res) => { req.setTimeout(300000); // 5 minutes res.setTimeout(300000); });
// Set server timeout server.timeout = 300000; ```
# Python Flask
@app.route('/api/data')
def data():
# Increase timeout for this route
return Response(generate_data(), direct_passthrough=True)Step 6: Configure Nginx Error Handling
Handle timeouts gracefully:
```nginx location /api { proxy_pass http://backend; proxy_read_timeout 300s;
# What to do on upstream failure proxy_next_upstream error timeout http_502 http_503 http_504; proxy_next_upstream_tries 3; proxy_next_upstream_timeout 60s;
# Custom error page error_page 504 = @timeout; }
location @timeout { default_type application/json; return 504 '{"error": "Request timed out", "retry": true}'; } ```
Step 7: Add Upstream Health Checks
Monitor backend health:
```nginx upstream backend { server 127.0.0.1:3000 max_fails=5 fail_timeout=30s; server 127.0.0.1:3001 max_fails=5 fail_timeout=30s backup;
# Fail tracking # max_fails: Number of failed requests before marking down # fail_timeout: Time to wait before retrying }
location / { proxy_pass http://backend;
# Passive health check proxy_next_upstream error timeout; } ```
For active health checks (Nginx Plus or with module): ```nginx upstream backend { zone backend 64k;
server 127.0.0.1:3000;
health_check interval=5s fails=3 passes=2; health_check_timeout 5s; } ```
Step 8: Implement Request Caching
Cache responses to reduce timeout risk:
```nginx # Cache configuration proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=api_cache:10m max_size=100m inactive=60m;
location /api { proxy_pass http://backend; proxy_read_timeout 300s;
# Enable caching proxy_cache api_cache; proxy_cache_valid 200 302 10m; proxy_cache_valid 404 1m; proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
# Serve stale cache on timeout proxy_cache_use_stale timeout;
add_header X-Cache-Status $upstream_cache_status; } ```
Step 9: Use Streaming for Long Responses
Stream large responses:
```nginx location /api/export { proxy_pass http://backend;
# Disable buffering for streaming proxy_buffering off;
# Or minimal buffering proxy_buffer_size 4k; proxy_buffers 4 4k;
# Longer timeout for exports proxy_read_timeout 600s; } ```
Step 10: Monitor Timeout Metrics
Track timeout occurrences:
```nginx # Add custom log format log_format timed '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent" ' 'rt=$request_time uct="$upstream_connect_time" ' 'uht="$upstream_header_time" urt="$upstream_response_time"';
access_log /var/log/nginx/access.log timed; ```
```bash # Analyze slow requests awk '$14 > 30' /var/log/nginx/access.log # Requests > 30s
# Check upstream response times cat /var/log/nginx/access.log | grep -E "urt=[0-9]{3,}"
# Monitor in real-time tail -f /var/log/nginx/access.log | awk '{print $14}' ```
Verification
Confirm timeouts are resolved:
```bash # Test slow endpoint time curl http://localhost/api/data # Should complete within timeout
# Check error logs for timeouts grep "timed out" /var/log/nginx/error.log | tail -20 # Should see no new timeout errors
# Monitor response times tail -f /var/log/nginx/access.log
# Test multiple requests for i in {1..5}; do echo "Request $i:" time curl -s http://localhost/api/data > /dev/null done
# Check upstream status curl http://localhost/nginx_status
# Verify configuration sudo nginx -t ```
Requests should now complete successfully without 504 timeout errors.
Related Articles
- [Nginx troubleshooting: Fix Lambda Permission Denied - Complete ](fix-lambda-permission-denied)
- [Nginx web server troubleshooting: Fix Client Max Body Size Large Upload Nginx Issue ](client-max-body-size-large-upload-nginx)
- [Fix Apache 502 Proxy Error](fix-apache-502-proxy-error)
- [Fix Apache LogLevel Core Debug Configuration](fix-apache-loglevel-core-debug)
- [Fix Cloudflare 502 Bad Gateway Error](fix-cloudflare-502-bad-gateway)
<script type="application/ld+json"> { "@context": "https://schema.org", "@type": "TechArticle", "headline": "Nginx Upstream Timed Out", "description": "Resolve Nginx upstream timeout errors with proxy_read_timeout, proxy_connect_timeout, and backend optimization", "url": "https://www.fixwikihub.com/fix-nginx-proxy-upstream-timeout", "publisher": { "@type": "Organization", "name": "FixWikiHub", "url": "https://www.fixwikihub.com" }, "author": { "@type": "Person", "name": "FixWikiHub Editorial Team" }, "datePublished": "2025-11-19T12:08:44.568Z", "dateModified": "2025-11-19T12:08:44.568Z" } </script>