The obvious conclusion was that the API was slow. It wasn't.

This post walks through how we narrowed it down, step by step, and the measurement habit that would have saved us time from the start.

TL;DR: The database and the API were fast. The time was being lost after the API had its response ready, somewhere between the server and the client. Always compare backend time with total time, and test from more than one place.

The setup

The load test ran from outside the server, over a VPN. Every request travelled the same route:

Load test client → VPN → Nginx (reverse proxy) → API → Database

And the response came back the same way, in reverse.

That gives us five suspects: the database, the API, Nginx, the VPN and network, and the load test client itself.

Step 1: Check the database

The first suspect in any slow API is usually the database, so that's where we started.

Queries were completing in milliseconds. Whatever was taking two minutes, it wasn't the database.

Step 2: Run the same test from the server

Next, I replayed the same workload from the API server itself, still going through Nginx and the API, so only the outside part of the journey was removed.

Same API, same proxy, same database. The only big difference was where the requests came from.

That one comparison narrowed things down enormously. If the API or database were the problem, the test from the server would have been slow too.

Step 3: Read the Nginx logs

Nginx can log two different timings for every request, and comparing them is the key to this whole story:

• $upstream_response_time is how long the backend, our API, took to send its response to Nginx.
• $request_time is the total time Nginx spent on the request, from the first bytes it read from the client until it finished sending the response back. That includes reading the request, any queueing, and delivering the response.

For one slow request, the logs showed:

The API had its answer ready in under a third of a second. Getting that answer back to the client took almost 80 seconds.

This can happen because Nginx buffers responses by default. The API hands the full response to Nginx quickly and moves on, and Nginx then sends it on to the client at whatever speed the connection allows. If the path to the client is slow, the total time grows while the backend time stays small.

Note: if response buffering is turned off, for example for streaming responses, a slow client can hold the backend connection open too. In that case the upstream time grows as well, which is a useful clue in itself.

Caution: the gap between the two numbers isn't always response delivery. It can also be a slow request upload or queueing in Nginx. Log request_length and body_bytes_sent to tell them apart.

So where did the time go?

The evidence pointed to the delivery path, the part of the journey between the server and the client:

• The database was fast.
• The API was fast.
• The test from the server, through Nginx, was fast.
• Only requests travelling from outside were slow.

At this point, the remaining suspects are all on that path:

• VPN bandwidth. Every concurrent response shares one tunnel. If a load test pushes more data than the tunnel can carry, responses queue up, waits stretch into minutes, and the longest ones time out, which shows up as failures.
• Packet size problems (MTU). VPNs add extra headers to every packet. If full-size packets no longer fit and the network can't signal that properly, large responses stall while small ones look fine.
• The wider network. Packet loss, a distant gateway, or traffic inspection along the route.
• The load test client itself. A load generator that's short on CPU, memory or bandwidth can't read responses fast enough, which makes the server look slow.

How to check this yourself

1. Split a single request's time with curl

Run this from where your users or test client are:

curl -o /dev/null -s -w "first byte: %{time_starttransfer}s | total: %{time_total}s | size: %{size_download} bytes | speed: %{speed_download} bytes/s\n" https://your-api-url

A fast first byte with a slow total means the server responded quickly and the transfer is the slow part.

2. Run the same test from more than one place

Compare results from the server itself, from inside the same network, and from where your real users are. The point where things get slow is where to look.

3. Compare data needed with capacity available

Work out how much data your test moves per second: response size × requests ÷ test duration. Then measure what the path can actually carry, for example with iperf3 across the VPN. If the test needs more than the path can deliver, you've found your bottleneck.

4. Watch the load generator

Check CPU, memory and network usage on the machine running the test while it runs. If it's maxed out, your results describe the test machine, not your API.

Lessons

1. "The API is slow" is a guess, not a diagnosis. Measure before you optimize.
2. Backend time and total time are different numbers. If they're far apart, the problem is outside your code.
3. Test from multiple places. One comparison, from the server versus from outside, ruled out most of the stack in minutes.
4. Load tests test the whole path, including the tester. Make sure the client, the network and any VPN can handle the load you're generating.

A healthy API doesn't always mean a healthy application. Your users don't experience your API on its own. They experience the whole journey.

Because code only makes sense in context.

Update (Nov 9): @lycore pointed out that request_time minus upstream_response_time can also include request upload and queueing, not just response delivery. Fixed the wording above and added request_length and body_bytes_sent to the log format.

Have you ever chased a "slow API" that turned out not to be slow at all? I'd love to hear your story in the comments.

If you prefer short videos, this story is also a reel on Instagram at @codeincontext.ag.