Nginx Reverse Proxy and 502/504 Troubleshooting¶

A 502 Bad Gateway or 504 Gateway Timeout from nginx means nginx itself is working fine — it successfully received the request and is telling you the backend it's proxying to failed to respond correctly (502) or didn't respond in time (504). This is one of the most common real-world production incidents, and almost always diagnosable in under five minutes if you check things in the right order.

How it works¶

A minimal reverse proxy config¶

# /etc/nginx/sites-available/app.conf
upstream app_backend {
    server 127.0.0.1:8000;
}

server {
    listen 80;
    server_name app.example.com;

    location / {
        proxy_pass http://app_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_read_timeout 60s;
    }
}

sudo ln -s /etc/nginx/sites-available/app.conf /etc/nginx/sites-enabled/
sudo nginx -t                          # syntax-check BEFORE reloading — same principle as sshd -t
sudo systemctl reload nginx            # reload, not restart — zero-downtime config apply

proxy_set_header X-Forwarded-For matters for anything downstream that needs the real client IP (access logs, rate limiting, geo-IP) — without it, the backend only ever sees nginx's own IP as the source of every request.

Diagnosing 502 Bad Gateway¶

502 means nginx could not get a valid response at all from the upstream — usually because the upstream process isn't running, isn't listening on the expected port, or closed the connection abruptly.

# 1. Is anything actually listening where nginx expects it?
ss -tlnp | grep 8000

# 2. Check nginx's own error log for the specific reason
sudo tail -50 /var/log/nginx/error.log

A connect() failed (111: Connection refused) in the error log means nothing is listening on that port at all — the backend process is down, crashed, or listening on a different port/interface than configured. (104: Connection reset by peer) means something was listening but closed the connection abruptly mid-request — often the application itself crashing on that specific request (check the application's own logs next, not nginx's).

Diagnosing 504 Gateway Timeout¶

504 means nginx did connect to the upstream, but didn't get a response within proxy_read_timeout (default 60s). This points at the backend being alive but too slow, not absent.

# Is the backend slow, or hung entirely?
time curl -sS http://127.0.0.1:8000/health    # call it directly, bypassing nginx

If the direct curl also hangs past the timeout, the problem is squarely in the backend application (check its own logs, thread pool / worker exhaustion, a slow downstream database query it's waiting on). If the direct call returns quickly but nginx still times out, suspect the proxy_read_timeout value itself being too low for a legitimately slow endpoint, or a networking issue specifically on the nginx → backend hop (e.g., a firewall rule rate-limiting that specific path).

Load balancing basics¶

upstream app_backend {
    least_conn;
    server 10.0.0.11:8000;
    server 10.0.0.12:8000;
    server 10.0.0.13:8000 backup;
}

least_conn sends new connections to whichever backend currently has the fewest active connections (vs. the default round-robin, which ignores backend load entirely). backup marks a server as only used when all non-backup servers are unavailable — useful for a standby instance.

Ubuntu/Debian vs. RHEL-family differences¶

Practical exercise¶

1. Start a trivial backend: python3 -m http.server 8000.
2. Configure nginx to reverse-proxy to it (as in the example above) and confirm it works via curl http://localhost/.
3. Stop the Python server (Ctrl+C) and confirm nginx now returns 502; check error.log for the exact reason logged.
4. Restart the backend but make it artificially slow to respond (e.g. python3 -c "import time; time.sleep(120)" behind a tiny custom handler, or simply set proxy_read_timeout 2s; against the normal server to force a timeout) and confirm 504.

How to verify you did it correctly¶

• Step 3: the HTTP response should be exactly 502, and error.log's newest line should mention Connection refused referencing the backend's address and port.
• Step 4: the response should be 504 after waiting the configured proxy_read_timeout, not before.

Common mistakes¶

• Restarting nginx instead of reloading for a config change. reload re-reads config with zero dropped connections; restart briefly drops the listening socket entirely.
• Increasing proxy_read_timeout as a first response to 504s, without checking whether the backend is actually just hung (infinite loop, deadlocked worker) rather than legitimately slow — a longer timeout on a genuinely hung backend just makes clients wait longer before the same failure.
• Not checking the application's own logs, only nginx's — nginx's error log tells you that the backend failed, rarely why the backend itself failed.

⚠ Dangerous commands¶

• systemctl restart nginx on a production host with active long-lived connections (WebSockets, long-polling) drops all of them immediately — reload is almost always the correct choice for config changes; reserve restart for when nginx itself is actually stuck.
• Testing a new proxy_pass upstream in production without nginx -t first can take the entire site down on a typo, not just the one route being changed (a syntax error in one server block can prevent nginx from reloading at all, leaving the previous config running until fixed — this is actually a safety feature of reload, but only if you never bypass -t and just force-kill and restart instead).

Related articles¶

• DNS Resolution and Troubleshooting
• Diagnosing High CPU and Memory Usage
• Docker Networking Fundamentals

Review questions¶

1. What's the practical difference between what a 502 and a 504 error each tell you about where the failure is?
2. Why should nginx -t always be run before reload, even though reload itself is usually described as "safe"?
3. Why doesn't increasing proxy_read_timeout fix a backend that's genuinely deadlocked, even though it "fixes" the symptom of seeing 504s?
4. What's the operational difference between systemctl reload nginx and systemctl restart nginx for currently-connected clients?