Most LLM apps are measured on quality and cost. Latency gets attention only after a user complains, and then it gets attention in the least useful way: someone swaps in a smaller model, quality drops, and the complaint changes shape.
Latency is an engineering budget like any other. You can measure it per stage, spend it where users notice, and cut it where they do not. This post is the sequence we use on production hardening engagements: pick the metric that matches the UX, instrument the stages, then attack the two or three stages that actually own the seconds.
Pick the right metric before you optimize
"The app is slow" is three different problems depending on the surface.
- Time to first token (TTFT) — how long the user stares at nothing. This is the number that governs perceived speed in any streaming chat UI.
- Total wall time — the number that matters for non-streaming work: extraction jobs, batch classification, an agent that must finish before the next step runs.
- Time to first useful token — TTFT's honest cousin. If your stream opens with 40 tokens of preamble ("Certainly! Let me look that up for you...") before any content, your measured TTFT is good and your UX is not.
Pick one primary metric per surface and track p50, p95, and p99. Averages hide the tail, and the tail is what people remember. A p50 of 1.2s with a p99 of 28s is not a fast app; it is a fast app with an incident nobody has filed yet.
Instrument stages, not the endpoint
An end-to-end timer tells you that a request took 6 seconds. It does not tell you where to work. Break the request into the stages you can actually change, and emit a span for each one. If you run LangSmith or any OTel-compatible tracing, you already have the plumbing — most teams simply have not made the stage boundaries explicit.
A typical RAG turn breaks down like this:
- Input guardrail or moderation call (often a full model round trip nobody counted).
- Query rewriting or classification (another round trip).
- Embedding of the query.
- Vector search and keyword search.
- Reranking (a cross-encoder call, frequently the quiet leader).
- Prompt assembly.
- Generation — split into TTFT and token-generation time.
- Post-processing: validation, repair, citation resolution.
Write the median and p95 of each stage into a table. Nearly every team we do this with finds at least one surprise. The most common: a sequential chain of small model calls — moderation, then rewrite, then generate — where each call is "fast" at 400ms and the three together own more of the budget than the main generation. The second most common: a reranker running on 100 candidates when 30 would have scored the same recall.
Generation time is a token-count problem
Once you are inside the generation call, there are only a few levers, and two of them are about counting tokens.
Output tokens dominate. Decoding is serial. If the model emits 800 tokens at 60 tokens/sec, that is 13 seconds and no amount of infrastructure fixes it. Cutting a verbose answer format, dropping a chain-of-thought preamble you never show the user, or moving from prose to a compact structured payload often halves wall time with no quality change. Measure the change with your eval suite, not by reading three outputs.
Input tokens drive TTFT. Prefill is fast but not free, and a 30k-token context has a noticeably slower first token than a 4k one. This is also where prompt caching pays: if your system prompt, tool schemas, and few-shot examples are stable and sit at the front of the prompt, providers that support prefix caching will skip re-prefilling them. That requires discipline about ordering — one dynamic timestamp at the top of the system prompt invalidates the whole prefix. We have seen a single injected "Current date and time" line cost a team both their cache-hit rate and roughly 300ms of TTFT on every request.
Model choice is a real lever, but a measured one. Smaller and faster models are legitimate for classification, routing, query rewriting, and extraction with a tight schema. Route by task, keep the larger model for synthesis, and gate every route change behind the same evals you use for prompt changes. Routing without evals is just a quality cut with extra steps.
Overlap the work you cannot delete
After trimming, what remains is coordination. Two patterns recover the most time.
Parallelize independent stages. Dense search and BM25 have no dependency on each other; run them concurrently and fuse. Moderation of the user's input does not need to wait for query rewriting. In LangGraph, fan-out to parallel nodes that write to separate state keys and join at a reducer node. In plain Python, asyncio.gather is enough. The savings are exactly the smaller of the two branches, which is usually 200–600ms per pair — small individually, meaningful when you have three such pairs.
Start streaming before you are finished thinking. Streaming is a latency feature, not a cosmetic one. LangGraph streams token-level output (messages mode) and intermediate state (updates mode) separately, and you want both: tokens for the final answer, state updates for everything before it. A UI that shows "searching 4 sources → reranking → drafting" while the pipeline runs has a TTFT that users experience as under a second, even when the final answer starts at four. This is the single largest perceived-latency win available in agent UIs, and it costs frontend work rather than quality.
Be honest about what streaming does not fix. If your structured-output call must be fully parsed before anything renders, streaming buys you nothing; the fix there is a smaller schema or a two-phase response that streams prose while the JSON resolves behind it.
Agents: the budget is the step count
For agentic workflows, per-call tuning matters less than the number of sequential model calls. A five-step trajectory with a 3-second generation each is 15 seconds regardless of how good your prefix caching is.
Track steps-per-task as a first-class metric next to latency. Then reduce it the way you would reduce any loop: give tools that do more per call (one search_and_fetch instead of search then three fetches), tighten tool descriptions so the model stops taking exploratory detours, cap retries, and add a step budget with a graceful degradation path. Trajectory evals tell you whether a step cut changed the answer; latency dashboards tell you whether it was worth it.
Run the long tasks asynchronously. If a job genuinely takes 90 seconds, no optimization makes it feel interactive — accept it, checkpoint it, return a handle, and notify on completion. Users tolerate a background job far better than a spinner that looks broken.
Put the budget in CI
Latency regresses the same way quality does: one prompt edit at a time. Add timing assertions to the eval harness you already run — a fixed set of representative inputs, p95 per stage, and a failure when a stage exceeds its budget by more than some margin. Numbers from a shared CI runner are noisy, so compare against a rolling baseline rather than a hard constant, and alert on drift rather than on any single run.
The end state is unglamorous and checkable: every surface has a named metric, every stage has a measured share of the budget, and every change that moves either one shows up as a number before it ships.
If your app is slow and you cannot yet say which stage owns the seconds, that is the first thing to fix — and it is usually a day of instrumentation, not a rewrite. We do this as part of an LLM app review or a focused production hardening engagement. Tell us what you are seeing and we will reply with questions.