End-to-End vs. Step-Level Agent Evaluation

Agents don't produce answers in one step. A single user request turns into a chain of model calls, tool invocations, retries, and sometimes handoffs to sub-agents, and that chain is where most of the actual work, and most of the actual failure, happens. Evaluating only the final answer tells you if the agent got there. It tells you nothing about the ten decisions it made on the way. A full evaluation strategy needs both the outcome layer and the step layer, because each one catches mistakes the other simply cannot see.

Three layers fall out of this structure once you take it seriously. End-to-end evaluation asks whether the task succeeded. Trajectory-level evaluation asks whether the path there was sound and efficient. Component-level evaluation asks which specific tool, retriever, or sub-agent broke down. None of the three substitutes for the others, and treating any single layer as sufficient is how teams end up shipping agents that pass every test and still fail in front of customers.

What end-to-end evaluation measures

End-to-end evaluation is the layer almost every benchmark and every tutorial covers: score the last message against an expected result, done. It's popular because it's cheap. There's no need to inspect intermediate states, no need to define what "good reasoning" looks like step by step, just a clean pass-or-fail signal that's easy to compare across model versions and easy to wire into a CI/CD pipeline as a regression gate. For any claim about whether an agent "works," this layer provides the accountability.

It's also where the story stops being useful. An agent can land on the right answer while the path it took was wrong, wasteful, or outright unsafe, and end-to-end scoring has no way to tell the difference. An agent that stumbles into the correct output by coincidence scores identically to one that reasoned through the problem correctly, both get marked as a pass, and both look the same in a dashboard. That's a problem, because a correct answer reached through broken reasoning is fragile. It worked this time because the inputs happened to line up. When the inputs change slightly, the same broken process produces a wrong answer, and there's no way to predict when.

An agent can look busy, call tools that sound right, reason in fluent, confident language, and still not finish the task. And even when it does finish, it may have called the wrong tool, passed a malformed argument that happened not to break anything, looped through the same retrieval three times before giving up, or burned far more tokens than the task required. End-to-end evaluation can't see any of that. It only sees the last message.

What step-level evaluation measures that end-to-end cannot

Step-level, or trajectory, evaluation scores the entire sequence, meaning the reasoning, the tool calls, the intermediate outputs, not just the last artifact. It answers a different set of questions entirely: did the agent pick the right tool and pass it the right arguments? Was the order of operations sensible, or did it try to write a file before it read the one it needed? Did it loop, retry the same failed call five times, or take a detour that added latency without adding value? Did it get a document retrieval that happened to be right for the wrong reason? And when something did go wrong, did the agent recover, or did it compound the mistake into something worse downstream?

That last question points to compounding error, probably the strongest argument for step-level evaluation. In a multi-step agent, a small mistake at step 3, a slightly wrong parameter, a misread piece of context, can cascade by step 7 into a completely broken plan. The real cause lies far upstream of the visible failure, which appears only downstream in the final output. By the time the agent produces a bad final answer, the real mistake is buried ten steps back, and end-to-end evaluation has no mechanism for finding it.

Most teams running agents in production could not tell you whether their agent called the right tool on step 14 of a 50-step chain. That's not a knock on those teams; it's just what output-only evaluation leaves you with: a pass/fail signal and no diagnostic trail behind it.

Process Reward Models versus Outcome Reward Models: the architectural choice behind each layer

This split between outcome and process isn't a new idea invented for agents, it's a distinction the reinforcement learning literature has worked through for years, under the names Outcome Reward Models and Process Reward Models.

One kind of reward model gives a reward signal based only on the final output, essentially estimating the probability that a given completion ends in a correct answer. It's the natural mathematical cousin of end-to-end evaluation: score the end state, ignore how you got there.

Another model scores each step in the reasoning chain individually. That granularity is what makes it possible to localize an error instead of just detecting that one occurred somewhere. PRMs also improve explainability, since a reviewer (human or automated) can point to the exact step where things went wrong, and that same signal can be used to give targeted feedback during generation, not just after the fact. Research using step-level PRMs, or step-guided preference optimization more broadly, has shown meaningfully faster convergence and better sample efficiency compared to training on outcome rewards alone, with some studies reporting gains of several times over. That's not a small margin. It explains why process supervision has become a serious area of research rather than a footnote to outcome-based methods.

How major benchmarks split or conflate the two evaluation layers

Looking at what the field's major benchmarks actually score reveals which layer the industry has prioritized and which one has been left mostly unaddressed.

GAIA (General AI Assistants) tests real-world questions that demand multi-step reasoning, web browsing, handling multiple modalities, and tool use. Scoring is exact-match after normalization, which is cheap to compute and hard to argue with. But it's purely end-to-end: there's no signal at all about the steps in between. GAIA accuracy is currently around 74.5%, against a human baseline that meaningfully exceeds agent performance, and a related benchmark, WebArena, shows agents at 74.3% success against a 78.24% human baseline. A chunk of what GAIA counts as "failures" are actually formatting mismatches: the agent had the right answer but presented it in a shape the exact-match scorer didn't recognize. That's a reasoning success mislabeled as a reasoning failure, and it's exactly the kind of nuance end-to-end scoring can't distinguish.

SWE-bench takes a different, and arguably sturdier, approach: it runs the actual test suite against the agent's code change and checks whether previously failing tests now pass. That's execution-based verification, not an LLM guessing at correctness, and it set a precedent for grounding agent evaluation in outcomes that can be checked mechanically rather than judged. The benchmark now ships in several forms: Verified (a 500-instance subset filtered by human reviewers), Lite, Full, Multilingual (300 tasks spanning 9 programming languages), and Multimodal. Even so, SWE-bench remains end-to-end by design, it doesn't score the intermediate state of the code or the reasoning that produced it. And it has its own documented issues: overly detailed issue descriptions can inflate resolution rates by giving the agent too much of the answer up front, differences in scaffolding between agents make it genuinely hard to say whether a performance gain came from a better model or a better harness. The current task set stands at 2,294 issues.

τ-bench, developed by Princeton and Sierra researchers, evaluates agents on general-purpose service tasks by checking the resulting database state rather than doing a text match against an expected string. Even so, τ-bench is still fundamentally judging outcomes rather than scoring the steps themselves.

The benchmark-to-production gap that both layers must close

Strong benchmark numbers do not reliably predict strong production performance, and the gap between the two is large enough to be its own problem. Enterprise agentic AI deployments show something like a 37% gap between lab benchmark scores and real-world performance once systems hit production traffic. That's a wide enough margin to make benchmark leaderboards a poor proxy for whether an agent is actually ready to ship.

The adoption numbers tell a related story. Around 85% of companies experiment with generative AI, yet a large share never get an agent past the proof-of-concept stage into production. Gartner has projected that more than 40% of agentic AI projects will be canceled by 2027, and a meaningful part of that is attributed not to the models being incapable, but to organizations lacking any systematic way to evaluate an agent once it's deployed and running against real, unpredictable input.

Offline evaluation, even good offline evaluation with full step-level coverage, cannot close that gap by itself. A static test set can tell you the agent didn't regress on yesterday's questions. It cannot tell you how the agent handles an input nobody wrote a test for. Multi-turn conversations drift in ways a fixed benchmark was never built to capture, and tool failures that occur only when live, real-world systems are involved (a flaky API, a rate limit, a schema that changed last week) do not occur in a static suite. Much of what passes for agent evaluation today boils down to outcome metrics that hide the intermediate failures, manual spot-checks that don't scale past a handful of examples, or benchmark scores that have nothing to do with latency, cost, or the constraints of a real CI pipeline.

LLM-as-judge: the default scoring method for both layers and its known failure modes

Since human review can't keep pace with production-level volume, LLM-as-judge has become the default scoring mechanism for both end-to-end and step-level evaluation alike. The case for it is stronger than it might sound at first: an LLM judge agrees with human reviewers on a given task roughly 85% of the time, which is actually higher than the agreement rate between two human reviewers grading the same task independently. That's a genuinely useful signal, and it's part of why LLM-as-judge scaled as fast as it did.

But it is not a plug-and-play solution, and treating it as one is where teams get burned. Surveys of LLM-as-judge reliability keep turning up the same recurring problems: judges are inconsistent across different prompt phrasings and even across repeated runs of the identical prompt, they carry systematic scoring biases (favoring longer answers, or answers written in a certain style, regardless of correctness), they calibrate poorly to specialized domains, and the field still lacks any real meta-evaluation standard for checking whether a judge is doing its job. In production settings, frontier models have been shown to exceed 50% error rates on bias tests in production, which is a striking number for a method this widely deployed. A judge that was validated on ordinary chat conversations is not automatically reliable for scoring a RAG pipeline, a code review, or an agent's multi-step task performance, not without domain-specific calibration first.

Using it well takes real engineering discipline: structured rubrics instead of open-ended "rate this response" prompts, multiple judge passes to average out run-to-run variance, and ongoing calibration against a set of human-labeled examples to catch bias and drift before they quietly corrupt the eval pipeline.

When trajectory evaluation is worth the investment

Trajectory evaluation isn't free, and pretending otherwise does teams no favors. Building reference trajectories takes real labeling work, maintaining the evaluator infrastructure is an ongoing cost, and running LLM-as-judge or human review against every intermediate step, rather than just the final answer, multiplies the cost of evaluation directly.

For simple, single-tool tasks, where an agent makes one call and returns a result, end-to-end coverage alone is probably enough. There's not much of a trajectory to inspect when the whole path is one step long, and the overhead of building step-level infrastructure won't pay for itself.

The calculus changes fast once an agent is chaining several tools together, handing work off to sub-agents, or operating in a domain like finance or healthcare, where a plausible-sounding wrong answer carries real cost. That's where errors compound and show up as the actual failure mode in production, and it's exactly the failure mode that only step-level evaluation is built to catch.