A pattern shows up in almost every agent review we do. The agent is sharp for the first several turns. Somewhere past turn twenty it starts repeating tool calls it already made, forgetting a constraint the user gave at the start, or answering from a stale document it retrieved ten steps ago. The team's first instinct is to try a stronger model or a bigger window. Neither fixes it, because the failure is not reasoning capacity. It is what the loop keeps putting in the prompt.
Large windows made this worse, not better. When a million tokens fit, nothing forces a decision about what belongs in the prompt, so the message list grows by default and the model is asked to find one instruction among fifty pages of tool output. Retrieval quality inside the context window follows roughly the same curve as retrieval quality inside a vector index: precision matters more than volume.
Context engineering is the discipline of deciding, every turn, what the model sees. Here is how we approach it in client repositories.
Account for the context you are actually sending
Before changing anything, measure. For a representative long session, log per turn: total prompt tokens, and the split across system prompt, tool definitions, conversation messages, tool results, and retrieved documents.
That table is usually the whole diagnosis. The distributions we see most often:
- Tool results dominate. One
list_filesor a raw API response returns 8,000 tokens of JSON, most of it fields nobody reads, and it stays in the message list forever. - Retrieved documents accumulate. Each retrieval step appends its chunks. By turn fifteen the agent is carrying four generations of documents, three of them irrelevant to the current sub-goal, all of them competing with the live one.
- Tool definitions are a fixed tax. Thirty tools of schema on every request, including the requests that need none.
Until you have those numbers, any change you make to the loop is a guess.
Give each part of the prompt a budget
Treat the window as an allocation, not a ceiling to grow into. A workable starting split for a tool-using agent: system prompt and tool schemas fixed and small, recent conversation guaranteed, retrieved context capped, tool output capped hard, and meaningful headroom left for the response.
The important word is capped. A budget you only enforce when the request would overflow is not a budget; it means quality silently degrades right up to the limit, then something gets truncated at the least convenient moment. Enforce the caps at every turn, from the first one.
Prune tool output at the boundary
The highest-leverage change is usually the least clever: do not put raw tool responses into the message list.
Wrap each tool with a post-processor that returns only the fields the agent reasons over. A ticket lookup returns id, title, state, assignee, and a truncated body — not the full payload with fourteen URL fields and an audit trail. A search tool returns ranked titles with snippets and ids, and a separate fetch tool pulls the full text of exactly one result when the agent decides it needs it. That search-then-fetch shape keeps breadth cheap and depth deliberate.
Two rules we apply consistently:
- Cap every tool's output length in code, with an explicit truncation marker so the model knows content was cut rather than assuming it saw everything.
- Write large artifacts to state or to disk, not into the transcript. Put the file path or a state key in the message and let a later step read it back if needed. LangGraph state is the right home for a 30,000-token document; the message list is not.
Compact deliberately, not reactively
Long sessions eventually need the history to shrink. Two mechanisms, and they are not interchangeable.
Trimming drops old messages by count or token budget. Cheap, lossless for what remains, and fine for chat-shaped assistants where old turns genuinely stop mattering. The trap is tool-call pairing: dropping a tool call while keeping its result — or the reverse — produces malformed histories that some providers reject and others quietly mishandle. Trim on message-pair boundaries, and always keep the system message.
Compaction replaces a span of old turns with a structured summary. Use it when early context carries commitments the agent must honor for the whole session. Do not ask for a prose summary; ask for the fields you actually need — the user's goal, constraints and preferences stated so far, decisions already made, actions already taken with their outcomes, and open questions. A schema-shaped summary survives repeated re-summarization far better than free text, which drifts a little more each pass until the original constraint is gone.
Trigger compaction on a token threshold — say, when history crosses a set fraction of the budget — not on turn count, since turns vary wildly in size. And keep the last few turns verbatim after the summary. Agents lose the thread when the immediately preceding step is only paraphrased.
Move durable facts out of the transcript
Some things should not be re-derived, re-summarized, or trusted to survive compaction: the user's stable preferences, project-level facts, credentials scoping, decisions with downstream consequences. Those belong in an external store keyed by user or thread, written explicitly by the agent or by your code, and injected into the system prompt as a short block at the start of each turn.
This is a small amount of machinery — a store, a write path, a read that renders a bounded block — and it changes the failure mode from "the agent forgot" to "the agent read a stale fact," which is at least debuggable. Keep the injected block bounded and reviewable. Memory that grows without a cap turns into the same context bloat you just removed, only harder to see.
One caution: write to memory sparingly and specifically. Agents given a free-form remember_this tool will store conversational noise, and every future turn pays for it.
Reset context at task boundaries
If your workflow has phases — research, then plan, then execute — do not carry the research transcript into execution. Hand the next phase a structured artifact: the plan, the chosen sources, the constraints. Sub-agents are the clean version of this. A sub-agent runs its own loop in a fresh window and returns a result, not its scratch work. The parent stays small, and the noisy exploration is thrown away by construction rather than by summarization.
This is also the honest answer to some multi-agent enthusiasm: the value is usually context isolation, not the collaboration metaphor.
Measure it like any other change
Context changes are exactly as testable as prompt changes, and they need to be, because pruning too aggressively causes a distinct failure — the agent re-fetches things it already had, or asks the user a question that was answered on turn three.
Run a set of recorded long sessions through the loop before and after, and report:
- Task success on the final output, scored against your existing rubric.
- Constraint retention: for sessions where an instruction was given early, did the final output still honor it? This is the metric compaction breaks first.
- Redundant tool calls: repeated calls with identical arguments. A rise means you pruned something the agent needed.
- Tokens per session and time to completion, so the savings are stated, not assumed.
We generally want the first two flat or better and the last two down. When constraint retention drops, the fix is almost always the summary schema, not the trigger threshold.
The short version
Measure what you send. Give each region a budget and enforce it every turn. Prune tool output at the wrapper. Compact into a schema, keep recent turns verbatim, and store durable facts outside the transcript. Reset at phase boundaries. Then prove it with recorded sessions.
None of this is exotic, and most of it is a day or two of work in an existing LangGraph app. It is also, in our experience, the difference between an agent that demos well and one that survives a thirty-turn session with a real user.