Human Evaluation Protocols for Open-Ended Agent Tasks

Human evaluation of open-ended agent tasks cannot be skipped, and it cannot be scaled the way standard evaluation scales. The reason sits in what an agent actually produces: not a single output to grade, but a trajectory, a sequence of plans, tool calls, intermediate decisions, and handoffs, where the final answer is just the last artifact left behind. Standard evaluation, built for single-turn language models, was never designed to look at the chain. It was designed to look at the endpoint.

That mismatch creates a specific blind spot. A correct final answer can sit on top of a trajectory riddled with problems: wrong tools called and then quietly recovered from, loops that burned time and compute without adding value, intermediate steps that violated policy, tool invocations the agent claims to have made but never did. None of that is visible when the only thing graded is whether the final answer matches an expected one. The agent looks like it succeeded, and in one narrow sense, it did, but the path it took to get there may contain exactly the failures a production deployment needs caught.

Errors in agent systems compound. A weak assumption formed early in a trajectory propagates through every downstream decision, so by the time a failure becomes visible, it's often far removed from where it actually originated. Diagnosing the real problem after the fact, without a recorded trace to inspect, becomes close to impossible. The mistake happened at step four, and the damage showed up at step nineteen, and no amount of staring at the final output recovers that connection.

Agents also behave stochastically. Run the same task against the same agent twice: the trajectories will likely differ, sometimes meaningfully, even when both runs land on an acceptable final answer. That means a single pass or fail score carries weak signal on its own, and any protocol that wants real confidence has to account for run-to-run variation, adding cost to evaluation. None of this is a tooling gap that better dashboards or faster graders will close. It's a structural mismatch between what agents do, which is act across time under uncertainty, and what standard evaluation assumes, which is that one output can represent the whole of a system's behavior.

The three failure modes that benchmark-style eval cannot see

Benchmark-style evaluation misses three distinct kinds of failure, and each one slips through for a different structural reason.

The first is behavioral. Agents built on frontier backbones have been documented claiming to have logged or updated system state without ever invoking the tool that would have done so. A study covering 23,500 agent turns records exactly this pattern: phantom tool-call claims that produce a plausible, confident final response while the tool invocation that would have produced it never actually occurred. A trace makes this directly verifiable. Final-answer scoring has no way to see it at all, because the text the agent returns reads no differently from a run where the tool call genuinely succeeded.

The second is procedural. An agent can reach the correct end state by moving through illegitimate or unsafe intermediate steps, a pattern a 2026 preprint names "corrupt success." The final answer checks out, the task appears complete, and the metric that only looks at outcomes has nothing to flag. A trajectory that violates policy in the middle but lands correctly at the end is still a failing trajectory. That distinction only becomes visible once evaluation looks at the steps, not just the destination, and the fuller mechanics of catching it belong to the discussion of HOB's audit rules further on.

The third, and the sharpest evidence that these failure modes are real rather than theoretical, is open-ended reasoning failure. Kirgis, Kapoor, Narayanan, and collaborators from Princeton, Stanford, the UK AI Security Institute, Johns Hopkins, and Georgetown's CSET ran what they call shadow evaluations in August 2026, giving frontier agents the central research questions behind two unpublished NeurIPS 2026 submissions³⁵⁸⁹¹⁰¹²⁴⁶⁷¹¹. The agents completed all the engineering work without human help. Neither paper made substantial progress on the actual research question, and the original authors rejected both submissions without ambiguity. The agents hadn't failed at coding or execution. They failed at judgment: poor calibration about what clears the bar for publishable research, uncreative responses when the research design hit shortcomings, ineffective backtracking once an approach proved unworkable, weak awareness of resource constraints, and drift away from the original instructions as the work progressed.

What ties these three failure modes together is where they hide. Each one stays invisible as long as evaluation stops at the final answer, and each one becomes visible once evaluation reaches down to the trajectory and the behavior that produced it.

Why scaling human review per task does not solve the problem

The obvious response to failures that only appear in the trajectory is to put a human on more trajectories. That response runs into limits fast, and the limits aren't about willingness, they're structural.

Human review at scale typically works by sampling: quality analysts check a subset of agent outputs against detailed rubrics, and that sample stands in as the quality signal for an entire high-stakes workflow. The arithmetic doesn't close. Sampling means most trajectories go unexamined by design, and the ones that do get reviewed are no guarantee of catching the specific run where a phantom tool call or a policy-violating detour occurred.

Adding reviewers doesn't fix this cleanly either. Inter-annotator agreement among human reviewers is imperfect even when everyone is working from the same rubric, so putting more people on the problem introduces more disagreement that then has to be reconciled, rather than simply producing a larger, more confident sample. The bottleneck is reviewer expertise: knowing which tool calls in a given domain deserve scrutiny, which policy deviations actually matter, and which refusals represent a safe stopping point versus a failure to resolve the task requires domain knowledge that is expensive to apply one run at a time.

The way out of this is relocating human involvement, not reducing it. Human judgment creates the most leverage when it's applied upstream, in the design of the evaluation protocol itself, rather than downstream, repeated turn by turn across every individual run.

Placing human expertise upstream: the Human-on-the-Bridge approach

Human-on-the-Bridge builds the expertise into the protocol once, and lets that single investment get reused across however many runs follow. Bousetouane's HOB paper frames this as a shift in where experts spend their time. Instead of reviewing outputs one at a time, experts contribute domain context, Red-Team Traps, Juror Personas, scoring guidelines, audit rules, and fallback policies, curated once and then executed repeatedly by what the paper calls the ProofAgent Harness, through multi-turn adversarial evaluations, trace capture, multi-juror scoring, and evidence-linked reporting.

Red-Team Traps turn red-teaming from a one-off event into infrastructure. Instead of a single adversarial exercise that produces findings for one moment in time, the traps are pre-designed adversarial scenarios that encode domain failure knowledge into reusable test cases, challenging the agent the same way across repeated runs and across different versions of the agent itself. Juror Personas solve a related problem from a different angle: rather than funneling every evaluation through one evaluator's perspective, the same run gets scored from multiple domain-appropriate viewpoints at once, which removes the single-evaluator bottleneck without requiring proportionally more human hours for every run.

Audit rules are what make HOB's findings usable. They connect trace evidence directly to scores, so a phantom tool call or a skipped mandatory step produces an evidence-linked finding instead of an unexplained drop in a number somewhere. That link is what makes root cause analysis tractable once evaluation runs at scale: a reviewer, or a team building on top of HOB's findings, can trace a low score back to the specific action in the trace that caused it.

The study behind HOB covered 23,500 agent turns across finance, healthcare, and code generation, and it produced a finding worth taking seriously on its own: smaller Harness LLMs were able to challenge frontier-backbone agents effectively once armed with expert-curated evaluation intelligence. Evaluation quality is a function of how well the evaluation it's running has been designed, not strictly of how large the evaluating model is. HOB also catches an asymmetric failure mode that outcome scoring misses entirely: a safe but non-resolving refusal, where an agent preserves the appearance of safety by declining to act, while still failing to complete the task it was given. HOB flags that case alongside outright policy violations, because both are behavioral failures hiding behind a passable-looking final response.

The strongest objection to this approach is that pre-curated traps can miss failure modes that only emerge at runtime, since encoding domain knowledge upfront requires experts to anticipate problems before they've happened. HOB's answer is that the trap library isn't static. Evaluation intelligence gets updated as new failure modes appear in production, making the protocol a living document. The alternative, reviewing every run from scratch with an expert in the loop, surfaces the same failure modes at a far higher cost each time. It just makes discovering the same failure modes far more expensive each time.

This is also the layer where infrastructure built for trace-level evaluation earns its place. Platforms like Confident AI's DeepEval are built around the operational work HOB's paradigm demands: capturing traces, running multi-juror scoring, and mapping specific metrics to specific levels of an agent's behavior. The paradigm and the platform play complementary roles. HOB defines what expert-curated evaluation intelligence should contain. Trace-level infrastructure is what makes that intelligence executable across thousands of agent turns without an expert needing to sit in front of every single one.

Shadow evaluations as a protocol for open-ended research and judgment tasks

Some tasks resist pre-curated traps entirely, because the standard for success isn't known in advance. Open-ended AI research is the clearest example: nobody can write a Red-Team Trap for "did this agent make a genuinely good research decision" when the answer depends on a research question that hasn't been settled yet. The fix here is recruiting the evaluator who already holds the standard, namely the domain expert who defined the research question in the first place.

That's the method Kirgis, Kapoor, Narayanan, and their collaborators call shadow evaluations, and the paper positions it explicitly against two weaker alternatives³⁵⁸⁹¹⁰¹²⁴⁶⁷¹¹. Narrow, verifiable benchmarks exclude open-ended research by construction, since they only work where correctness can be checked mechanically. Blind peer review, the traditional fallback, is overstretched, produces stochastic outcomes, and suffers from review quality that's inconsistent at best. Shadow evaluation offers a third path: give an agent the central research question behind a real, high-quality unpublished paper, and have the paper's own original authors grade what the agent produces.

The method is necessarily small-sample and qualitatively rich. Each evaluation unit is a full research trajectory, judged by the people who defined the question the trajectory was meant to answer, not an aggregate score across many short, interchangeable tasks. The two NeurIPS 2026 case studies show what that kind of evaluation reveals that automated grading never could. The agents completed the engineering work in both cases. They failed on the judgment-intensive steps: choosing which hypotheses were worth pursuing, recognizing when a line of inquiry had hit a dead end, adapting once the findings diverged from expectations. Both sets of original authors rejected the resulting papers, with unambiguous and well-reasoned rejections.

Across both cases, and reproduced in a robustness check using a second model and a second scaffold, five failure modes recurred: poor judgment about what clears the bar for publishable research, uncreative responses when the research design ran into shortcomings, ineffective backtracking once an approach had failed, poor awareness of resource constraints, and drift away from the original instructions over the course of the work. The CRUX initiative, led out of Princeton with collaborators including the UK AI Security Institute, is built to run shadow evaluations on a regular cadence, functioning as a longitudinal tracking mechanism. That repeatability is what turns shadow evaluation into a protocol rather than a single research exercise redone from scratch each time.

Shadow evaluations don't replace HOB-style protocols, and the two solve different halves of the same problem. HOB encodes failure modes that are already known into reusable traps, so a protocol doesn't need to rediscover them every run. Shadow evaluations exist for the failures nobody could have anticipated and written into a trap in advance, surfacing them by handing judgment to the person who already holds the standard the agent is being measured against.

Granularity decisions: matching evaluation depth to what can go wrong at each level

A protocol can adopt every principle described so far and still fail in practice if it checks the wrong things at the wrong level. Getting granularity right means matching what gets measured to what can actually go wrong at that layer of the system.

Confident AI's guide to agent evaluation metrics lays out a layered structure that puts tool-call accuracy and argument correctness at the component level, since that's where a single tool invocation either matches what the task required or doesn't. Plan adherence and step efficiency belong at the trajectory level, because those properties only make sense across a sequence of actions, not a single one. Task completion belongs at the end-to-end level, where the question is whether the overall job got done. Collapsing these into one undifferentiated score produces numbers that look clean and mean very little, because a strong trajectory-level pattern can be buried inside a weak end-to-end score, or the reverse.

Task completion measured only against the final text an agent returns will pass an agent that reports completing an action it never actually completed. The correct check is state verification: inspecting the database, the file system, or whatever system state the action was supposed to change, rather than trusting the string the agent returned about what it did. Loop count has a similar blind spot. A trajectory that took twenty steps and one that took three can return an identical final answer, so excessive looping stays invisible in a trace total unless cost is attributed per step.

For agents that operate tools or interact with live systems, a workable structure replays realistic workflows inside isolated sandboxes, verifies the resulting application state, deliberately injects recoverable failures to see how the agent responds, inspects the trace for any action that caused real damage, and includes tasks where the right behavior is to stop and ask for clarification.

Deterministic checks, schema validation, confirming the right tool got selected, belong at the turn level, where a clear right and wrong answer exists and a rule-based check can catch it cheaply and reliably. LLM-as-judge scoring belongs where outputs are genuinely open-ended and real judgment is required. Mixing the two up wastes compute on cases where a deterministic check would have worked, and it introduces noise into judgments where precision was available for free.

LLM-as-judge in the protocol: what it can and cannot replace

LLM-as-judge has a legitimate place in this architecture, bounded by the upstream design work that HOB and rubric-based protocols supply. Used outside those bounds, it introduces exactly the kind of unreliable signal the rest of the protocol was built to avoid.

Agreement between LLM judges and human reviewers degrades in specialized domains and for the most open-ended outputs. Creative and open-ended reasoning tasks show substantially lower human-LLM agreement than general tasks do. The scope given to a judge has to be calibrated to the domain it's operating in.

A separate concern affects raw agreement numbers directly: high agreement between judges can mask low validity. Judges may line up consistently with each other while being wrong in the same direction, missing the thing that actually matters in a given domain while still producing scores that look stable and trustworthy on paper.

The quality of an LLM judge's output tracks the quality of the rubric it was given. A judge working from a vague criterion returns vague scores. A judge working from HOB-curated scoring guidelines, with explicit audit rules attached, returns scores that are auditable and tied to specific evidence in the trace. The upstream design work described earlier is what makes the judge's output worth trusting at all.

LLM judges also carry known failure modes on coding and other structured tasks: sensitivity to the order in which options are presented, and inconsistency across repeated runs of the same evaluation. The right response is to route exact-verifiable criteria to deterministic checks and reserve LLM judgment for criteria that are genuinely open, where no deterministic check could substitute.

The field treats a specific threshold as the practical floor for production use: kappa above 0.8 per criterion, with anything above 0.6 considered acceptable for development but not production-ready, judge-human correlation above 0.8, and no catastrophic failure modes turning up in qualitative audit. Below that threshold, an LLM judge is not a reliable stand-in for human review on that criterion, and the protocol should treat it accordingly rather than defer to a number that hasn't earned the confidence placed in it.