Eval Harness Design for Tool-Using Agents

Agent benchmark scores get reported as properties of the model. In practice, they are properties of the harness, the scaffolding of tools, context, and control logic surrounding the model, which has as much influence on measured performance as the weights underneath it. Understanding what a harness is, and why disclosing it rigorously has become a discipline of its own, is the starting point for interpreting any claim about what an agent can do.

The harness shapes what a tool-using agent can do

A harness is the scaffolding that specifies which tools an agent can see, how those tools are described to it, what auxiliary information rides along with each observation the agent receives at every step, and how all of that gets assembled into the prompt the model actually reads. A recent constitutive definition puts this on firmer footing: an agent harness is a runtime layer with four necessary elements, an agent loop, a tool interface, context management, and control mechanisms. None of these four elements is the model itself. All four are decisions made by the people building the system around the model, and all four are configurable independently of which model sits at the center.

The distinction that the rest of this piece turns on is the gap between tool capability and tool architecture. Tool capability describes what an agent can do, the set of actions and information available to it. Tool architecture describes how those capabilities get organized and exposed to the model, the interface layer that sits between capability and use. Two agents given identical capabilities can produce substantially different outcomes on the same tasks, and the difference traces entirely to architecture.

That claim is not speculative. It comes from controlled research isolating architecture as a variable while holding capability fixed, the subject of the next section. A team that reports a benchmark score for its model may, without realizing it, be reporting the performance of its harness. Two labs running the same underlying model through different scaffolding can publish different numbers, and neither number describes the model in isolation. Any serious evaluation of a tool-using agent has to treat the harness as part of what is being measured.

How tool architecture changes agent behavior in measurable ways

The cleanest evidence for this claim comes from a study conducted by researchers at Microsoft Research and Purdue, published in August 2026, which ran six distinct tool architectures across three different models on repository-level issue-fixing tasks, generating a large set of trajectories for comparison. The design held the underlying information and available actions similar across conditions, varying only how those capabilities were organized and exposed to the model. That is the experimental structure that makes the results credible: capability stayed constant, architecture moved, and the outcomes moved with it.

The paper frames the distinction with an analogy. Tool capability functions like software functionality, the question of what a system can do. Tool architecture functions like software architecture, the question of how a system is organized to do it. Just as software architecture shapes non-functional properties like robustness and adaptability rather than raw feature count, harness architecture shapes non-functional properties of agent behavior: consistency across repeated attempts, breadth of exploration, and efficiency in steps and tokens spent.

The numbers are specific. More structured low-level interfaces improved consistency across repeated attempts compared to a basic bash-only architecture, by giving the agent finer-grained, more structured tool interfaces rather than raw shell access, which changed how reliably it reached the same outcome across independent runs. Python CodeAct-style interfaces, which let the model express actions as executable code rather than as discrete tool calls, achieved comparable task performance while using 41.6 percent fewer steps and substantially less token volume. Both findings describe the same model capability routed through different architecture, producing outcomes that a benchmark focused only on pass rates would never distinguish.

Not every architectural addition earns its complexity. Lightweight, text-based scaffolding tools that let an agent record intermediate reasoning showed limited effect on behavior in the same study, a reminder that architectural change does not automatically translate into performance change. The direction of the effect has to be measured, not assumed.

Production teams are discovering the same principle independently of the research literature. Vercel's engineering team removed the majority of the tools registered to their agent, a live example of a broader failure mode: a bloated tool registry is itself an architectural choice, and it carries a measurable cost even when every individual tool works correctly. Trimming the registry is not a matter of tidiness. It changes what the model has to reason about at every step, and that reasoning burden is exactly the kind of non-functional cost the Microsoft Research findings describe.

These findings sit inside a broader taxonomy. Published agent projects recur across five architectural dimensions, subagent structure, context management, tool systems, safety mechanisms, and orchestration, each of which offers its own spectrum of design choices. The next sections build toward a fuller treatment of those five, and the through-line is already visible: architecture shapes capability rather than footnoting it. It is a separate variable with its own measurable effects.

Why benchmark scores for tool-using agents mislead

If architecture produces effects this large, the obvious next question is why so few published benchmark scores account for it. The answer is structural. Existing benchmarks typically abstract away execution detail, compare complete end-to-end agent systems against each other, or hold the harness fixed across every model being compared, which makes configuration-level variation nearly impossible to isolate or study within the benchmark itself. A score produced under these conditions describes a model-harness pairing. Reporting it as a property of the model alone strips out the variable most likely to explain why a different lab, running the same base model through different scaffolding, gets a different number.

This is a measurement integrity problem rather than a complaint about any specific benchmark. The convention of scoring only the final answer, whether the agent's last message matches a target output, hides an entire category of failure. An agent can reason its way to a plausible-looking answer while its tool calls, its reading of workspace state, or its handling of intermediate evidence have quietly decoupled from what actually happened during execution. Final-answer scoring cannot see this. It checks the destination, not the route, and a wrong route that happens to arrive at a right-looking answer scores identically to a correct one.

Trajectory-level and per-turn scoring catch a distinct class of failures that final-answer checks miss entirely: looping behavior, failed recovery after an error, incorrect tool-call formation, and policy violations such as over-promising to a user, skipping an identity check, or taking an unauthorized action. The DeepEval framing draws a further distinction that matters for anyone designing an evaluation pipeline: offline development-time evals, which measure behavior against a known ground truth after the fact, are a different kind of instrument than runtime guardrails, which intercept a live response before it reaches a user. Treating these two as interchangeable, or worse, using the same metric for both without acknowledging the difference in role, is itself a measurement error, not a shortcut.

There is a further complication specific to harness-model interaction. Models trained with specific harnesses in mind can become fitted to those particular designs, so a harness swap after training is not a neutral operation. Because the best-performing harness configuration is specific to the model it wraps and has to be re-tuned whenever the base model changes, a gap opens continuously between what a model is capable of and what its current harness lets it realize. Closing that gap falls to the people building the system, by hand, on an ongoing basis. There is no evaluation convention yet that automatically flags when this gap has widened.

The three evaluation layers a rigorous harness eval must cover

Measuring whether a tool-using agent does its job correctly requires three distinct evaluation layers, final-answer, trajectory, and per-turn, each catching failures the others cannot.

Final-answer scoring checks the agent's last message against a target. It is necessary, in that a system producing wrong answers consistently has failed regardless of how elegantly it got there, but it is not sufficient, because it is blind to whether the path taken was safe, efficient, or even coherent. Trajectory scoring looks at the full sequence of steps and tool calls an agent made en route to that answer, and it catches looping behavior, redundant tool calls, failed error recovery, and whether the agent actually explored the context relevant to the task or missed it entirely. Per-turn scoring evaluates the meaning of each individual turn as it would occur in production, and this is the layer where policy compliance lives, whether a specific action at a specific moment violated a rule the system was supposed to follow.

Tau-Bench illustrates what the per-turn layer adds that simpler tool-use evaluations miss. Most tool-use benchmarks check whether an agent called the right tool with the right parameters. Tau-Bench goes further, checking whether the agent's actions violated any documented policy along the way, which catches over-promising to a customer, skipping a required identity check, or taking an action the agent was never authorized to take, failures that a parameter-matching check would score as a success. Its headline metric, pass^k across k independent rollouts, measures consistency directly rather than treating a single successful run as proof of reliability.

BFCL, the Berkeley Function Calling Leaderboard, illustrates the trajectory and correctness layer from a different angle. It evaluates function calling across an AST track, an executable track, and an irrelevance bucket, the last of which tests whether a model correctly declines to call any tool when none of the available tools actually apply to the task.

Deterministic scoring is achievable for a meaningful slice of these evaluations, which matters because LLM-as-judge scoring introduces its own noise into a measurement process that is already trying to isolate noise elsewhere. The agent-eval harness demonstrates this with three deterministic oracle booleans, refusal detection, user-goal compliance, and attack-success detection, that score authorized versus unauthorized red-team tasks without a judge model touching the primary metrics. Where a task's success condition can be defined in advance as a checkable boolean, that check should replace a judge model's opinion, not supplement it.

Harness configuration as the independent variable in diagnostic benchmarks

A handful of benchmarks published across 2026 were built specifically to treat harness configuration as the variable under study, rather than as background noise to be controlled away. What they found, when they looked, is that configuration-level variation produces outcome differences large enough to reshape how a model's capability should be reported.

Harness-Bench, published by researchers at Peking University and Qiyuan Tech in May 2026, evaluates 106 realistic end-to-end agent tasks drawn from practical, real-world agent-use patterns, each run inside its own sandboxed offline environment with task-specific configuration and evaluation criteria. Every task was manually reviewed for realism, difficulty, solvability, and reliability of its evaluation criteria, and the tasks themselves require completing concrete workflows rather than executing isolated tool calls in a vacuum. Across 5,194 execution trajectories, substantial variation emerged across different model-harness pairings, and making that variation visible and attributable to specific configuration choices is the benchmark's explicit design purpose.

MCP-Atlas takes a different route to a related goal. It runs a set of natural-language tasks against 36 live production MCP servers exposing 220 distinct tools, and the deliberate choice to use live servers rather than mocked APIs is what gives the benchmark its value. Live servers introduce rate limiting, pagination behavior, schema versioning, and transient network errors, conditions that mocked environments systematically fail to reproduce. The benchmark builds on the Model Context Protocol as a standardized interface for tool discovery and invocation across a range of providers, which lets the same evaluation methodology apply across a heterogeneous set of real tools rather than a curated, artificial set.

AgentCompass takes a third approach, integrating benchmarks that span five capability dimensions, tool use, web and research tasks, scientific reasoning, agentic coding, and productivity tasks, within interactive and long-horizon environments that include repository-level code modification and open-ended web navigation. Its purpose is to give a cross-environment view of agent performance rather than a snapshot from a single task type, which matters because a harness tuned for one capability dimension may perform very differently on another.

The design principle shared across these systems is sandboxed, reproducible environments that avoid live-service dependencies while preserving task realism, the opposite of the mocked-API approach MCP-Atlas explicitly criticizes. Taken together, these benchmarks make something visible that prior evaluation conventions could not: outcome differences attributable to specific, nameable configuration choices, rather than differences written off as model variance or evaluation noise.

The five architectural decisions every harness builder must make explicitly

A harness does not arrive pre-configured. Five architectural decisions, made explicitly or by default, determine most of how it behaves once it is put under evaluation. A study surveying 70 publicly available agent projects identified these five recurring dimensions: subagent architecture, context management, tool systems, safety mechanisms, and orchestration, each with its own spectrum of design options that shapes what the resulting harness can and cannot do.

Subagent architecture concerns whether the system runs as a single agent or coordinates multiple agents, and if multiple, whether coordination happens through a central manager or through decentralized handoffs between agents. AutoHarness, from Google DeepMind, distinguishes two orchestration templates along this axis, harness-as-action-verifier and harness-as-policy, which carry meaningfully different performance profiles depending on the task structure they are applied to.

Context management concerns what information gets assembled into each step of the prompt and what gets compacted or discarded as a task runs longer than a single context window can hold. Meta-Harness, developed by researchers at Stanford, KRAFTON, and MIT in March 2026, improves on a prior state-of-the-art context management system by 7.7 points on online text classification while using 4× fewer context tokens, showing that context efficiency is a measurable, optimizable property of harness design.

Tool systems concern which tools get registered to an agent, how richly each one is described, and how many are exposed at once. POSTECH's h-low, h-mid, and h-high harness variants isolate this single dimension, tool description richness, holding everything else fixed, and the resulting behavioral differences show that description quality alone changes harness performance. The Vercel decision to strip down a bloated tool registry belongs to this same dimension: fewer, better-described tools outperform a larger, thinly described set.

Safety mechanisms concern the layered guardrails a harness applies, input validation, output filtering, risk ratings assigned to individual tools, and triggers for human intervention when a task crosses a defined threshold. Constraints enforced at the tool schema level differ from permission checks enforced in the live execution path. The former is structurally more robust because it cannot be reasoned around by the model mid-task, and it belongs in harness design rather than being left to a system prompt's instructions.

Orchestration concerns how the agent loop manages planning, task decomposition, error recovery, and continuity across a long-horizon task. Meta's production harness for automating machine learning pipelines demonstrates a concrete pattern here: hibernate-and-wake checkpointing that lets a multi-day task resume after interruption without losing accumulated context, a design suited to scientific workflows where a single turn can exceed what a model's context window holds.

A harness that never makes them consciously has still made them by default, typically to the lowest-effort setting available, and that default carries its own measurable performance cost.

Harness design and model post-training must be co-optimized

The evidence assembled across the preceding sections points toward a conclusion that the field has been slow to formalize: harness design and model post-training are not sequential stages of a pipeline where one team finishes and hands off to the other. They are coupled variables that determine a joint outcome, and optimizing one while treating the other as fixed produces a system tuned for a configuration that will not hold.

The co-evolution risk runs in both directions. A model trained or fine-tuned against a specific harness configuration can become fitted to the particular tool descriptions, context assembly patterns, and control logic it was trained under, so that swapping in a different harness after training degrades performance even though the model's underlying capability has not changed. Conversely, a harness tuned against one model's quirks, its tendency to over-explore, its particular failure modes under long-context tasks, its habits around tool-call formatting, will not transfer cleanly to a different base model without re-tuning. This is the practical meaning behind the claim that the best harness is model-specific: it has to be revisited every time the underlying model changes, not configured once and left alone.

This has direct consequences for how organizations should structure the work of building and evaluating tool-using agents. Treating harness engineering as a downstream integration task, handled after model selection and separate from the evaluation process, guarantees that published capability numbers will understate or overstate what the model can actually do depending on how well the harness happens to fit. The harness is half of the system, not scaffolding assembled once a model is chosen. It is half of the system being evaluated, and it earns exactly as much scrutiny as the model it surrounds.