Most RAG tuning effort goes into the index: chunk sizes, embedding models, rerankers. Meanwhile the query arrives untouched, exactly as the user typed it. That is a problem, because users do not write queries the way documents are written. They type "what about the EU one?" in the fourth turn of a conversation, or paste an error message, or ask two questions at once. The retriever sees a string and does its best.
Query transformation is the step between the user's message and the retriever. Done well, it recovers recall that no index change will. Done carelessly, it adds a model call, 400 ms, and a new way to be wrong. This post covers which transformations earn their cost, and how to decide with an eval rather than a hunch.
First, find out which queries are failing
Do not add a rewriter because a blog post said so. Take your labeled retrieval set (if you do not have one, see our post on bootstrapping an eval suite from production traces) and bucket the queries where the right document is missing from the top k. In our experience the misses cluster into a few shapes:
- Context-dependent follow-ups. "And for contractors?" has no retrievable meaning without the previous turn.
- Compound questions. One query embeds to the average of two topics and ranks neither well.
- Vocabulary mismatch. The user says "cancel", the policy says "terminate". Dense retrieval covers some of this, lexical search covers none of it.
- Underspecified or noisy input. Pasted logs, long preambles, a one-word query.
Each shape has a different fix. If most misses are in one bucket, you need one transformation, not a pipeline.
Transformation 1: standalone rewriting for follow-ups
For chat applications this is usually the highest-yield change. Take the conversation history and the latest message, and produce one self-contained query. Keep the prompt narrow:
REWRITE = """Rewrite the user's last message as a standalone search query.
Resolve pronouns and references using the conversation.
Do not answer the question. Do not add facts not present in the conversation.
If the message is already standalone, return it unchanged.
Return only the query."""
def standalone_query(history, message, model):
if not history: # first turn: skip the call entirely
return message
return model.invoke([("system", REWRITE), *history[-6:], ("human", message)]).content
Two details matter. Skip the call on the first turn, since there is nothing to resolve. And cap the history you pass in; the last few turns are enough, and a long history raises cost and tempts the model to editorialize.
Use a small, fast model. This is a constrained rewrite, not reasoning. Verify that claim on your own eval before trusting it.
Transformation 2: decomposition for compound questions
If your misses include multi-part questions, split them, retrieve per sub-query, and merge. The merge is where teams get it wrong. Retrieve a small k per sub-query, deduplicate by chunk ID, and rerank the union against the original question. Otherwise the final context is dominated by whichever sub-query matched noisily.
Cap the number of sub-queries (three is a sensible ceiling). An unbounded decomposer will turn a simple question into six retrievals and a larger bill.
Transformation 3: expansion, and why hybrid search often replaces it
Multi-query expansion generates several paraphrases and unions the results. It helps with vocabulary mismatch, but it costs a model call and multiplies retrieval load. Before adding it, check whether hybrid search (BM25 plus dense, fused) already closes the gap for your misses. For jargon and identifiers, a lexical leg often does what paraphrasing was meant to do, with no extra generation step.
HyDE-style approaches, which embed a hypothetical answer instead of the query, can help on sparse, short queries. They also embed the model's guesses, which can pull in plausible but wrong neighbors. Treat them as a candidate to test, not a default.
Routing: not every query needs retrieval
A router decides whether to retrieve at all, and from which source. Greetings, questions about the product's own behavior, and requests over a single attached document should skip the vector store. In LangGraph this is a conditional edge after the rewrite node:
def route(state) -> str:
return state["route"] # "retrieve_docs" | "retrieve_tickets" | "direct"
graph.add_conditional_edges("classify", route,
{"retrieve_docs": "docs", "retrieve_tickets": "tickets", "direct": "answer"})
Keep the classifier's label set small and make "direct" an explicit option. Log the chosen route in your trace so you can audit misroutes later. A misroute is silent: the answer just comes back confidently wrong with no sources.
Measure the transformation, not the final answer
Each transformation needs its own before and after on retrieval metrics, run on the same labeled set:
- Baseline: raw query, recall@k and MRR.
- With the transformation: same retriever, same k.
- Slice results by the failure buckets above. A rewriter should move the follow-up bucket and leave standalone queries flat. If standalone queries regress, the rewriter is changing queries it should have left alone.
- Record the added latency (p50 and p95) and tokens per request.
Ship the change only if the recall delta justifies the latency and spend. A 2-point recall gain on a slice that is 3% of traffic rarely does.
Failure modes to test for
- Answering instead of rewriting. The model produces a response in place of a query. Add an eval case and check query length against the input.
- Injected facts. The rewrite adds details from the model's prior knowledge, which then steer retrieval. Compare entities in the rewrite against entities in the conversation.
- Over-rewriting. Exact identifiers, error codes, and quoted strings get paraphrased away. Instruct the model to preserve them verbatim, and test on queries that contain them.
- Compounding latency. Rewrite, classify, and decompose as three sequential calls can cost more than the answer. Merge the rewrite and classify steps into one structured-output call where you can.
When to leave the query alone
If your traffic is single-turn, short, and vocabulary-aligned with your documents, a rewriter is overhead. Fix chunking or add a lexical leg first. The honest answer from the eval is sometimes that the query was fine and the index was the problem.
If you want a second pair of eyes on where your retrieval misses actually come from, that is a standard part of an LLM App Review.