LLM cost conversations usually start in the wrong place: with the bill. The bill tells you spend went up; it does not tell you which feature, which prompt, or which retry loop did it. So the first move in any cost engagement is never "switch to a cheaper model" — it is accounting. Once you can see where tokens go, the savings are usually sitting in plain sight, and most of them do not touch output quality at all.
Step one: cost accounting from traces
Your tracing layer already records token counts per call. Roll them up into a simple ledger: cost per request, grouped by feature and by prompt version, with input and output tokens separated. Input tokens usually dominate in RAG systems (all that retrieved context), output tokens in generation-heavy features — and the fix differs accordingly.
The distribution matters more than the mean. In most systems we profile, spend is concentrated: a handful of request paths, and within them a tail of pathological requests — huge retrieved contexts, agent loops that ran long, retry storms — account for a disproportionate share. Finding that concentration turns "reduce costs" from a vague mandate into two or three specific tickets.
While you are in the ledger, look for the classic silent burners: a summarization step running on content nobody reads, verbose few-shot examples repeated in every call, retries that re-send the full context after a formatting failure, and agent loops without a step budget.
Caching: the cheapest tokens are the ones you never send
Three caching layers apply, in increasing order of care required:
Provider-side prompt caching. Most providers now discount input tokens that repeat a previous request's prefix. The optimization is purely structural: order your prompt so the stable parts — system message, instructions, few-shot examples, tool definitions — come first, and the volatile parts (the user's question, retrieved chunks) come last. A prompt that interleaves instructions with dynamic content defeats the cache; reordering it is often an afternoon's work for a durable double-digit percentage cut in input cost. No quality risk whatsoever.
Embedding caches. Re-embedding unchanged documents on every index rebuild is pure waste. Key embeddings by content hash and model version; skip what has not changed.
Response caching. Exact-match caching works for genuinely repeated requests (popular queries, deduplicated batch items). Semantic caching — serving a stored answer for a "similar enough" query — is where quality risk enters, because similarity thresholds are wrong at the margins. If you use it, log cache hits distinctly and include cached-answer cases in your eval sampling, so a bad threshold shows up as data rather than as user complaints.
Model routing: pay for reasoning only where reasoning happens
Most pipelines contain steps of very different difficulty, all running on the same large model out of convenience. Classification, routing, query rewriting, extraction into a rigid schema — these are small-model tasks in most systems. The frontier model earns its price on the synthesis step, not on deciding which of four categories a ticket belongs to.
The discipline is to route by step, then prove it: run your eval suite with the cheaper model on the candidate step, compare against baseline, and promote the swap only when the delta is noise. Sometimes the small model genuinely is not good enough — that is a fine outcome; the eval told you before your users did. Do the same exercise on model upgrades in the other direction: new cheaper generations frequently dominate older expensive ones, and an eval-gated swap makes catching that routine.
A fallback chain closes the loop: try the cheap path, escalate to the expensive model when validation fails or confidence is low. You pay the premium only on the requests that need it.
Context discipline: input tokens are a retrieval-quality dividend
In RAG systems, the biggest input-token lever is upstream of the prompt: retrieval quality. Teams stuff ten chunks into context because they do not trust their top three — the padding is compensation for weak ranking. Fix ranking (better chunking, reranking) and you can cut k with improved answer quality, because the model stops reading distractors. This is the happy case where the cost work and the quality work are the same work.
Related trims: cap per-chunk length at retrieval time, summarize long conversation histories instead of resending them, and set explicit output length expectations in the prompt — unbounded verbose answers are a cost and a UX problem wearing one hat.
Make it stay fixed
Cost regressions are as real as quality regressions and sneak in the same way — a new feature, a prompt edit, a bumped k. Two guardrails keep the ledger honest: per-request cost in your eval harness (every eval run reports tokens alongside pass rates, so a change that buys two points of quality for triple the cost is a visible trade, not a surprise), and budget alerts per feature, not just per account, so the on-call engineer learns about a retry storm from a dashboard instead of an invoice.
Sequence the work in that order — accounting, caching, routing, context — and gate each change on evals. Quality regressions from cost work are optional, and the teams that avoid them are simply the ones that measured.