Trajectory Annotation for Agentic Task Datasets

Trajectory annotation is the practice of attaching structured labels, at the step level, the trajectory level, or both, to the multi-step records an AI agent produces while working through a task. It produces the training data, the benchmarks, and the failure post-mortems that shape nearly everything else in agent development. Understanding how it's done, and where each method breaks down, matters more now than it did two years ago, because agents have moved from demo to deployment and the annotation methods built for research settings are being asked to do industrial-scale work they weren't designed for.

What a trajectory is and what annotation means here

A trajectory is the full record an agent generates while attempting a task from start to finish: not just the answer it lands on, but every step along the way. Every annotated trajectory shares a small set of structural elements. There's a high-level goal or user request that kicks things off. There's a stream of observations and environment feedback the agent receives as it acts. There's the natural language reasoning, the chain-of-thought the agent produces to decide what to do next. And there are the grounded actions themselves, the tool calls and commands that actually touch the environment.

Annotation is what turns this raw record into something useful for training or evaluation. It means attaching labels to the sequence, sometimes to each step, sometimes to the trajectory as a whole. Those labels can be a simple binary marker of success or failure, a classification of what kind of error occurred, or a human rating or LLM-generated critique that says something more qualitative about how the agent performed. From these labels, teams compute global metrics: Task Success Rate, Trajectory Efficiency, Tool Call Accuracy. And what those metrics unlock downstream is substantial: supervised fine-tuning, reinforcement learning, evaluation benchmarking, and failure analysis all depend on annotated trajectories as their raw material.

Annotation depends on this, and that dependency shapes how reliable everything built on top of it can be. Annotation is the layer everything else stands on. It's the layer everything else stands on, which also makes it the layer most likely to buckle under pressure.

Annotation as the central bottleneck for agentic systems at scale

LangChain's State of AI Agents report finds that a majority of organizations now have agents in production, with quality having become the number-one barrier to further deployment. That's a striking inversion. Agents are no longer stuck in the research phase; they're live, and the thing holding teams back isn't compute or model access, it's confidence in whether the agent is actually doing the job correctly.

Evaluating an agent is a categorically different problem from evaluating a model. A model benchmark tests capability and understanding in a fairly static sense: does this answer look right? Agent evaluation has to test sequential decision-making across an environment that changes as the agent acts on it. This requires judging a whole chain of decisions rather than one output.

Complicating things further, trajectory data doesn't exist in any usable form out in the wild. It can't be scraped the way text corpora can, because producing it requires situational reasoning and multimodal interactivity, an agent actually operating inside an environment, not just narrating what it might do. So every trajectory that gets labeled has to be generated and then annotated, and that annotation work is punishingly slow. Human annotation of a single trajectory on popular agentic benchmarks can take hours.

Constructing HLE required 1,000 experts to label only 2,500 examples, which makes large-scale expansion by that method alone essentially impractical. A thousand experts for twenty-five hundred data points is not a pipeline, for it's a research project.

Step-level annotation makes the burden worse, not better. Verifiable, programmatic criteria (does this test pass, did this API call return the right value) are rare once you drop below the level of the final outcome. Most of what needs judging at the step level is fuzzy: was this reasoning sound, was this tool choice sensible given the context. That kind of qualitative judgment is exactly the kind that resists scaling through human labor alone. Teams get pushed, almost by necessity, toward automated approaches. But automating judgment introduces a new question that doesn't go away just because it's convenient: how much do you trust the automation?

The four main annotation approaches and their tradeoffs

Four broad approaches have emerged, and each one cuts cost differently, at the cost of something else.

Fully manual human annotation remains the gold standard for accuracy. Expert annotators review screenshots, actions, and full reasoning chains before assigning a label, and nothing currently matches that level of judgment. It is also, unsurprisingly, prohibitively expensive at any real scale.

LLM-as-a-judge, or LLMJ, is the answer most teams reach for when manual annotation can't keep pace. It uses LLMs to generate natural language critiques at both the process level, judging every step, and the outcome level, judging the final result. It's now used everywhere from initial data curation through production deployment, because it's cheap and it scales. Responsible teams don't just trust it blindly, though: production teams compute Cohen's kappa between the judge and a labeled human sample before shipping any new rubric, then re-sample monthly to catch judge drift. The trade-off that gets skipped over more often than it should is that the soundness of the critiques themselves, not just the final verdict, frequently goes unmeasured.

Hybrid multi-model labeling tries to split the difference. It works in two stages: a quality-scoring pass first removes low-fidelity samples, then multiple heterogeneous verifiers, drawn from different model families, each produce a binary verdict and a taxonomy label. Verdicts get aggregated by majority vote, and ties get routed to a human for adjudication. Where models disagree, that disagreement itself becomes useful information, a signal that the case is genuinely ambiguous rather than something to average away. This approach reduces the single-model bias that plagues plain LLMJ, and it costs more than pure LLMJ, but still less than full human annotation.

Rule-based and programmatic annotation works from the opposite direction. Expert or "golden" trajectories from benchmark environments establish what the correct path looks like, the full trajectory gets divided into sub-trajectories by action sequence, an LLM generates initial labels, and a rule-based script filters and remaps those labels against the known-correct structure. It's reliable wherever tasks are verifiable and well-defined. It simply does not generalize to fuzzy, qualitative dimensions, which rules out a large share of what agent evaluation actually needs to capture.

Finally, there's the automated pipeline with quality scoring, which treats trajectory quality as a filtering problem rather than a labeling problem. Anything below threshold gets discarded early, which stabilizes the quality of the dataset as a whole even when the annotation underneath is largely automated. Best use: "golden" anchor sets of 50–100 hand-crafted cases covering the most critical user scenarios. Each candidate trajectory is assigned a quality score (e.g., 1–5) on structural validity and behavioral plausibility: complete turn structure, valid tool calls, step-to-step coherence, realistic risk-trigger chains.

How well automated judges agree with humans

It produces the training data, the benchmarks, and the failure post-mortems that shape nearly everything else in agent development. The evidence is mixed, and the shape of the disagreement turns out to be more informative than the headline numbers.

On WebArena tasks, evaluating four LLM judges against trajectories produced by a GPT-4 agent, the best judge reached 80.6% accuracy against WebArena's own rule-based evaluator. That's a respectable ceiling, but it's a ceiling, not a floor, and it means one in five judgments from even the strongest judge tested didn't match ground truth.

iOSWorld's 2026 results paint a more encouraging picture at the trajectory level. Judges also correctly classified the majority of individual rubric criteria, though agreement there ran somewhat lower than at the whole-task level.

The Counsel dataset complicates the picture in a useful way. It's a meta-evaluation, a dataset built specifically to check how well LLMJ critiques themselves hold up on tasks drawn from τ-bench and DA-Code. Human annotators reviewing those critiques reached a Krippendorff's alpha of roughly 0.78, a solid, reliable level of agreement among the humans themselves. The more capable judges tested reached strong agreement with humans on where in a trajectory an error occurred. But agreement on reasoning quality, on whether the judge's explanation for the error was actually sound, dropped considerably. Location and reasoning are not the same skill, and judges are noticeably better at the first than the second. The one consistent pattern: more capable models and more reasoning effort both pushed agreement with humans upward, suggesting the gap is closable, just not yet closed.

Taken together, this evidence suggests LLMJ is good enough for binary outcome judgments at real scale. Where it struggles, and struggles specifically, is step-level reasoning quality: not just flagging that something went wrong, but correctly explaining why. That gap is precisely what pushes the field toward asking a different question: not just how good is the judge, but at what resolution should judgment even be applied. Binary task-level success judgments agree 89% of the time (κ=0.77, F1=0.86).

Granularity choices: step-level, trajectory-level, and span-level annotation

Granularity is a decision, not a technical detail, and it should be driven by what the labels are actually for.

Step-level annotation labels each individual action: was the tool choice right, were the arguments correct, was the reasoning behind the step sound. It's the only granularity that shows where a failure actually originates within a sequence, rather than just confirming that a failure happened somewhere. It's also the most expensive to do with human labor, and the most naturally paired with LLMJ operating at the process level, since programmatic ground truth is rare at this resolution and qualitative judgment dominates.

Trajectory-level annotation steps back and asks one question: did the agent complete the task? It's the natural fit for chosen-versus-rejected trajectory pairs used to train reward models that operate over an entire multi-turn, multi-tool sequence rather than any single step, and it pairs well with preference-learning objectives such as DPO. It's also well-suited to training judges that need to reason over full tool-execution logs and long-horizon plans, since the object being judged is the whole arc rather than a fragment of it. The obvious cost: it says nothing about where inside that arc things went wrong.

Span-level annotation sits between the two, and it's the most fine-grained of the three. It identifies the specific sub-sequence within a trajectory where an error actually originates, which requires a real curation pipeline, from trajectory collection through span segmentation to expert-verified error labeling. Recent work on error localization in deep-research agents ran tasks across three frontier base models under two agent frameworks, producing 2,790 long-form agent trajectories for exactly this kind of fine-grained analysis.

Decisive-error, or counterfactual, annotation is a distinct variant. Annotators are given the task instruction, the full failed trajectory, and the final failure outcome, and asked one specific question: what is the earliest step that directly caused the failure, under the counterfactual test of whether the task would have succeeded had that step alone been correct. This formalizes causal attribution instead of settling for mere correlation between an error and a bad outcome, which is a meaningfully harder and more useful standard.

Choosing among these isn't a matter of preference so much as matching the label to the downstream job. Supervised fine-tuning on positive examples generally only needs trajectory-level labels. Reinforcement learning with step-level rewards needs step-level labels, full stop, there's no substitute. Failure analysis and self-correction work need span-level or decisive-error annotation, because without knowing exactly where the chain broke, there's nothing concrete to fix.

What the major annotated trajectory datasets contain

The datasets built so far show that the more carefully labeled a dataset is, the smaller it tends to be, and the largest datasets lean hard on automation.

HLE spans a large multimodal question set across more than 100 disciplines and required substantial expert involvement to construct. AgentBank, published at EMNLP 2024 Findings, takes a different approach: a large collection of high-quality interaction trajectories spanning five distinct agent skill dimensions, built with a pipeline specifically designed to minimize difficulty bias, and used to fine-tune the Samoyed family of agent models.

Tool-use datasets built for scale look different again. TOUCAN, released in 2025, contains 1.5 million tool-agent trajectories pulled from nearly 500 real-world MCP servers, covering a wide swath of tools. It's filtered with both rule-based and model-based validation, and its tasks, annotations, and trajectories are all generated through a combination of LLMs, agent frameworks, and remote MCP servers. Because the tool calls are real rather than simulated, hallucination-driven code errors drop, though hallucination as a general risk doesn't disappear entirely. Models fine-tuned on TOUCAN have outperformed larger closed-source models on BFCL V3, which is a notable result for a dataset built largely through automated generation.

AutoTool, also called Open-AgentRL and presented at ICML 2026, pairs its large tool-use trajectory set with explicit tool-selection rationales, covering tasks across mathematics, science, code generation, and multimodal reasoning. Trained only on a subset of tools it has seen, it generalizes to a much larger library that includes many tools it never encountered during training, at inference time, which says something encouraging about how well tool-selection reasoning transfers. TaskCraft, from ICLR 2026, automates the generation of tool-intensive tasks across a range of difficulty levels, including multihop decompositions, and supports both SFT and RL trajectory sampling; on GAIA, a multi-hop QA baseline improved substantially once TaskCraft tasks were added for RL, with further gains as more tasks were layered in.

On the safety side, R-Judge offers 569 annotated multi-turn interaction cases spanning 5 application categories, 27 scenarios, and 10 distinct risk types, built specifically to evaluate risk awareness in tool-using agents.

Then there are the failure and trace-debugging datasets, which connect most directly to the next section. TRAIL, from 2025, contains 148 human-annotated traces drawn from single- and multi-agent systems, focused on software engineering and open-world information retrieval, built around a formal error-type taxonomy; tellingly, the best-performing model tested, Gemini-2.5-pro, scored only 11% on TRAIL, showing that current LLMs are still poor at debugging their own traces. MAST offers 200 manually annotated multi-agent failure trajectories, and Who&When offers 127, both useful but both limited in scale for systematic evaluation.

The pattern across all of this is hard to miss. Scale and annotation depth pull against each other: the largest datasets rely on automated annotation, and the most carefully labeled ones, TRAIL, MAST, Who&When, are small by comparison. No dataset currently on offer combines both. On the retrieval side, the Unified Agent Trajectory Dataset and GAE-Bench draw on annotated demonstrations across varied real-world GUI scenarios and introduce multimodal trajectory retrieval as a new modeling task, with GAE-Retriever outperforming strong baselines on retrieval recall. GAIA: 466 real-world questions evaluating reasoning, tool use, and web browsing. Counsel (2026): the first public dataset of meta-evaluations for agentic tasks; process-level critiques from open-weight LLMJs on τ-bench and DA-Code; human annotators label each flagged error as "spot on," "correct location but poor reasoning," or "should not have flagged"; Krippendorff's α ≈ 0.78; permissively licensed for broad community use.

Annotating for failure attribution specifically

Failure attribution is the sharpest test of everything discussed so far, because a wrong answer alone doesn't say much. Knowing an agent failed is trivial; knowing which step caused the failure is the actual diagnostic work, and current tooling handles that worst. TRAIL's 11% score for even a top-tier model is a blunt reminder that trace debugging remains a largely unsolved, widespread problem.

That's why decisive-error, counterfactual-style annotation matters more than its niche framing suggests. Asking whether a task would have succeeded had one specific step been correct forces a causal claim instead of a vague gesture at "something went wrong around here." Span-level pipelines, like the one that produced 2,790 long-form deep-research trajectories across three frontier models and two frameworks, exist precisely because coarser labels can't support this kind of question.

The smaller failure-specific datasets, TRAIL, MAST, Who&When, all share the same shape: rigorously labeled, genuinely useful, and too small on their own to train a general-purpose failure-attribution system. Building that system, at the scale TOUCAN or TaskCraft have reached for tool use, remains open work. Until it lands, failure attribution will stay closest to solved in principle and furthest from solved in practice.