Most teams evaluate an agent the way they evaluate a chatbot: run the input, grade the final string, count passes. That works until it doesn't, and with agents it stops working early. An agent can produce the correct final answer after calling the wrong tool three times, retrying a failed search, and spending forty seconds and 60,000 tokens getting there. Final-answer grading calls that a pass. Your users and your invoice disagree.
An agent's output is not a string. It is a trajectory: an ordered sequence of model turns, tool calls, tool results, and state transitions that ends in a response. If you want to know why an agent regressed after a model upgrade or a prompt edit, the trajectory is where the evidence lives.
Three failure modes final-answer evals miss
Right answer, wrong path. The agent guessed from parametric knowledge instead of calling the lookup tool. It passes today on the questions the base model happens to know, and fails silently the moment a customer asks about data only your tools can see.
Right path, unacceptable cost. Loops. Re-reading the same document. A planner that decomposes a two-step task into nine. Nothing in the final answer reveals it; your latency p95 and token spend do, a month later.
Correct output, dangerous side effects. The agent issued a write, sent a message, or refunded an order on the way to a correct summary. Read-only grading of the last turn cannot see this at all. For any agent with tools that mutate state, side-effect assertions are not a nice-to-have.
What to record
Before you can score trajectories you have to capture them. With LangGraph this is mostly free: stream the graph with a thread ID and persist the checkpoint history, or read the run tree from LangSmith. For each example, store the full turn sequence, every tool call with its arguments, every tool result, the final state, and the totals — steps, tokens, wall-clock, cost.
One discipline matters more than the tooling choice: make tool execution swappable. Evals that hit live APIs are slow, flaky, and occasionally destructive. Run your eval trajectories against recorded or stubbed tools, with the real implementations reserved for a small smoke set. Deterministic tools also mean a trajectory diff between two runs reflects the model and the prompt, not yesterday's search index.
Score the trajectory in layers
Resist the urge to invent a single agent quality number. Different properties want different graders, and mixing them produces a metric nobody can act on.
Tool-call correctness (exact, cheap). Did the agent call the tools the task requires, with arguments that parse and satisfy the constraints? Most of this is assertable in plain code: the search_orders call must include a customer ID; the refund call must never appear in a read-only scenario. Code assertions are fast, free, and never hallucinate.
Order and dependency (exact, where order matters). Some tasks have a required partial order — authenticate before fetching, fetch before summarizing. Assert the constraint, not a rigid golden sequence. Pinning the exact step list makes every harmless refactor look like a regression and trains the team to ignore the suite.
Efficiency (numeric, budgeted). Set a step ceiling, a token budget, and a latency budget per task class, derived from what a competent trajectory actually costs — measure your current p50, allow headroom, and gate on the ceiling. Efficiency regressions are the most common and least noticed agent regressions after a model swap.
Reasoning quality (LLM judge, narrow). Keep the judge for things code cannot check: was the plan sensible, did the agent recover appropriately from a tool error, did it ask the user instead of guessing when the request was ambiguous? One criterion per judge prompt, and calibrate each against hand-labeled examples before you let it gate anything. A vague "rate this trajectory 1-10" judge is noise with a decimal point.
Final answer (as one signal among several). Still worth grading. Just stop treating it as the whole evaluation.
Build the set from real runs
The same rule as any eval suite: sample from production, not imagination. Pull recent agent runs and label roughly half failures — loops, wrong tool, wrong recovery, refusals — and half representative traffic. For each, write down the assertions above: required calls, forbidden calls, ordering constraints, budgets, and what the answer must contain.
Add the adversarial cases deliberately, because they rarely occur in sampled traffic but dominate incidents: a tool that returns an error, a tool that returns an empty result, a tool that returns plausible garbage, an ambiguous request, and a request the agent should refuse. How an agent behaves when a tool fails is a design property you should be testing on purpose, not discovering during an outage.
Thirty to fifty labeled trajectories is enough to start. It is small enough to run on every prompt change and large enough to catch the failures that matter.
Report diffs, not dashboards
A CI run should print what changed since the last accepted run, per example: which assertions flipped, which budgets moved, which trajectories got longer. "Pass rate fell three points" starts a meeting. "These four examples now call search twice before summarize, and mean steps went from 6 to 9" starts a fix, usually within the hour.
Keep the accepted baseline in the repository next to the graph definition. When a change is a deliberate trade — more steps for better recovery — a human accepts the new baseline in the pull request, and the reason lives in the commit message where the next engineer will find it.
The honest trade-off
Trajectory evals cost more to build than answer grading. You write assertions, you stub tools, you argue about budgets. The return shows up the first time someone proposes a model upgrade or a prompt rewrite: instead of a week of cautious canarying, you read a diff. That is the same argument as any test suite, and it holds here for the same reason — agents have too many degrees of freedom to be validated by looking at the last line of output.
If your agent is in production and your evals only grade final answers, start with the cheapest layer. Assert the required and forbidden tool calls on thirty real tasks and put a step ceiling on each. You will learn something uncomfortable in the first run.