Capacity Planning for LLM Apps: Rate Limits, Concurrency, and the Queue You Already Have

Most LLM apps are load tested once, by accident, on the day they get traffic. The demo handled five concurrent users fine. Then a customer rolls it out to a team of three hundred, an agent starts making eight tool calls per turn, and the error rate goes to twelve percent. The model did not get worse. The system never had a capacity plan.

This is a tutorial on building one. It is mostly arithmetic and a few bounded queues, and it takes about a day.

Your unit of capacity is tokens per minute, not requests

Provider quotas are usually expressed two ways: requests per minute (RPM) and tokens per minute (TPM), per model, per account or project. Almost every team we review plans against RPM because requests are what they count in their dashboards. TPM is the one that bites, because token volume per request is not constant — it grows with conversation history, retrieved context, and agent step count.

Start by measuring, per endpoint and per model:

• Prompt tokens per call, p50 and p95. The p95 matters more; that is the long chat with eight retrieved chunks.
• Completion tokens per call, p50 and p95.
• Model calls per user request. For a RAG chain this may be 1-2. For an agent it is a distribution with a long tail, and you need the tail: the 95th-percentile trajectory, not the average one.

Multiply: tokens per user request at p95 = (prompt + completion) x calls. Then your ceiling is TPM quota / tokens-per-request, in requests per minute. Compare that number to your expected peak. Teams are routinely surprised to find their headroom is 3x, not 30x, and that most of it is prompt tokens they are resending every turn.

Do the same arithmetic for embeddings if you ingest documents on user action. A single uploaded PDF can be ten thousand embedding calls, and it will happily consume the quota your chat traffic needs.

Bound concurrency somewhere you control

The failure mode that produces the worst user experience is unbounded fan-out. An async handler accepts every request, each one fans out to parallel retrieval and tool calls, and the provider sees a spike it answers with 429s. Your retry logic then re-sends everything at once, which is a stampede, which produces more 429s. Latency goes up for everyone, including the requests that would have succeeded.

The fix is a single place in the process that caps in-flight model calls. A semaphore around the client, a worker pool, or a small queue in front of the LLM-calling layer — the mechanism matters less than the fact that there is exactly one and every path goes through it. Set the cap from the arithmetic above, not from a round number.

Two practical notes:

• Cap per model, not globally. Your cheap classifier model and your large synthesis model have separate quotas and separate latency profiles. One shared semaphore sized for the big model starves the small one.
• Give background work a smaller cap than interactive work. Ingestion, batch re-embedding, and nightly eval runs should never be able to consume the quota that serves live users. Separate pools, or separate API keys and projects if your provider supports per-project quotas.

Retry like you expect to be rate limited

You will be rate limited. Treat 429 as a normal control signal, not an exception to log.

• Respect Retry-After when the provider sends it. It is better information than your backoff formula.
• Exponential backoff with full jitter. Without jitter, every queued request retries at the same instant and you have rebuilt the stampede.
• Cap total retry budget per user request, in wall-clock seconds, not in attempts. A request that retries four times over ninety seconds has already failed from the user's perspective.
• Distinguish retryable from terminal. 429 and 5xx are retryable. A 400 from a malformed tool schema is not, and retrying it burns budget and money.
• Do not retry inside an agent loop and in the HTTP client and in the orchestration layer. Layered retries multiply. Pick one layer, usually the client, and make the others propagate.

Decide your degradation before you need it

When you hit the ceiling, something gives. Choose what, in advance, and write it down:

• Shed load at the edge. Return a clear "at capacity, try again shortly" to new requests rather than accepting them into a queue that is already longer than the user's patience.
• Fall back to a smaller or secondary model for lower-value paths. Fallback chains help, but only if you have evaluated the fallback model on your own suite. An untested fallback converts an outage into silent quality loss, which is worse because nobody pages for it.
• Drop optional work first. Reranking, query rewriting, and summary generation are usually degradable. Retrieval and the final answer are not.
• Queue asynchronous work honestly. If a job will take four minutes under load, show a job status, not a spinner that times out at thirty seconds.

The one thing not to do is let the degradation be implicit. "It gets slow" is not a plan; it is the absence of one.

Load test the trajectory, not the endpoint

A conventional load test that hammers one endpoint with one fixed payload will tell you very little about an agent. Token volume per request is the variable that matters, and a fixed payload holds it constant.

Build the test from real traffic instead. Take fifty to two hundred traces from production or staging, keep their actual inputs and conversation lengths, and replay them at increasing arrival rates. Record four things per run:

1. End-to-end latency, p50 and p95 — and time-to-first-token separately if you stream.
2. Error rate, split by cause: 429, timeout, tool failure, validation failure.
3. Tokens per minute consumed, against quota.
4. Cost per request, which usually rises under load because retries are not free.

Run it against a staging deployment with its own API key so you are not rate limiting your own production traffic to learn how rate limiting works. If you cannot get a separate key, run it off-hours and cap it.

What you are looking for is the knee: the arrival rate at which p95 latency bends upward. That is your real capacity. Set alerts at seventy percent of it and make the autoscaling, if any, trigger below it.

Watch the three numbers that predict the incident

In production, most capacity incidents are visible fifteen minutes early in telemetry nobody charted. Chart these:

• Queue depth / in-flight calls against your semaphore cap. Sustained saturation is the leading indicator.
• 429 rate per model. A slow rise means you are near quota; a step change means someone changed the prompt or shipped a feature that added a model call.
• Tokens per request, p95. This creeps. A new retrieval setting that returns ten chunks instead of five cuts your request ceiling roughly in half, and nothing in the deploy will mention capacity.

Pair the last one with your eval suite. A change that improves answer quality by adding context is a legitimate trade, but it is a trade: you spent capacity and latency for it. Make both sides of that visible in the same review.

The short version

Measure tokens per request at p95, divide your TPM quota by it, and you have a request ceiling. Put one bounded queue in front of the model so that ceiling is enforced somewhere you control. Retry with jitter and a wall-clock budget. Pick your degradation path on purpose. Then replay real traces at rising load until p95 bends, and alert below that point.

None of this is exotic infrastructure work. It is the same capacity discipline you would apply to any service that calls a slow, metered dependency — applied to one that happens to be a model.

If you want a second set of eyes on where your current system breaks under load, that profile is part of our LLM app review, and the fixes are the bread and butter of production hardening.