Counterfactual Testing for Agent Decision Points

An agent completes a twelve-step task and the dashboard says pass. Another agent runs the same task, fails on the final step, and the dashboard says fail. Neither result tells anyone which of those twelve steps actually mattered. A final-answer pass/fail score answers one question, whether the task succeeded, and leaves a much more important question untouched, which decision inside the run caused that result. Real agent evaluation needs to cover three layers, the final answer, the full trajectory, and the individual turns within it, but most teams building and shipping agents still treat the first layer as the whole job. Observability tools can tell you what happened, step by step, and scoring tools can tell you whether it passed. Neither one tells you which step caused the failure, and that gap doesn't close just because you're logging more data.

The instinct to blame the step where the bad thing actually happened, the tool call that leaked the data or the message that issued the wrong refund, is usually wrong. Pointing at the visible failure and calling it the cause mistakes the symptom for the disease.

Attempts to patch this with an LLM acting as judge, reading the trace after the fact and assigning blame, run into a separate limit. On the Who&When benchmark, built specifically to test step-level attribution in multi-agent failures, state-of-the-art methods score very low at identifying the actual responsible step. That result is the clearest evidence available that correlation-based attribution, however sophisticated the model doing the judging, isn't a substitute for establishing cause.

How pass-rate metrics hide behavioral change

Pass rate can stay almost flat while the underlying behavior of an agent changes substantially, and that is precisely the blind spot that makes pass rate dangerous as a primary metric. None of these shifts is visible in a pass rate, because pass rate only asks whether the final answer was correct.

That finding isn't an artifact of one dataset or one set of tasks. It follows directly from how pass rate is built: it saturates on tasks the agent was already succeeding at, and those high-baseline tasks are exactly where skill-driven behavioral effects concentrate. A metric that already tops out near its ceiling on a task simply has no room left to register a change in how the agent got there.

A related distortion appears in how reliability gets reported. The gap between those two numbers is a function of which metric someone chose to quote. Pass@k flatters peak performance; pass^k is the number that reflects what a production system actually needs, consistency across repeated runs, not a lucky best case.

Both distortions, the skills effect hidden inside a saturated pass rate and the reliability gap between pass@k and pass^k, trace back to the same root cause. Everything about which steps drove that outcome is lost in the compression.

What causal intervention does that correlation-based evaluation cannot

Counterfactual testing solves this by doing something categorically different from reading a completed log. It applies a formal do-operation to one specific decision point in an agent's trajectory, then re-runs the trajectory from that point forward to see how the outcome changes, with a causal inference framework built on structural causal models, directed acyclic graphs, and the backdoor criterion giving this operation its mathematical footing. Applied to an agent, the planning process itself gets formalized as a structural causal model: one that maps how the state of the environment shapes what action the agent generates, and how that action in turn shapes the next state.

The operation at the center of this is intervention: take one decision point, change what the agent chose to do there, hold the rest of the agent's policy fixed, and let the trajectory play out again from that point. The outcome that results gets compared against the original outcome, and the shift between the two is the causal effect of that single decision.

This is a fundamentally different kind of evidence than anything available from watching a completed run. A correlation observed in a finished log, two things happening together or one following another, cannot distinguish between a step that caused an outcome and a step that merely happened alongside it because something further upstream caused both. Acting on the naive correlation, by pouring more budget into marketing, would waste the money. Intervention breaks a confound like that: by changing one variable directly and holding everything else fixed, you find out what that variable actually does.

Pearl's Abduct-Act-Predict procedure gives this a concrete shape for agent traces. First, abduct the hidden state of the system from what was actually observed. Then act, applying the intervention, the counterfactual decision, at the chosen point. Then predict what outcome follows under that hypothetical, holding the rest of the policy constant. For an agent trace specifically, altering a step means replacing the action the agent actually selected at step t with a different, counterfactual action, keeping everything before t exactly as it happened, and letting the trajectory propagate forward from there under the same underlying policy.

This procedure rigorously defines what researchers call the decisive error in a multi-agent failure: the earliest mistake in the trajectory whose correction would have reversed the final outcome. That's not something you can read off a trace by eye. It can only be recovered by actually intervening on candidate steps and checking which one, when corrected, flips the result.

The specific challenge counterfactual attribution faces in multi-step and multi-agent systems

What is already a hard problem in a single agent running a single trajectory turns into a combinatorial one once trajectories get long and once multiple agents start interacting. Failures cascade through a system, and the real cause of a bad outcome can sit many steps, or many agents, upstream of where the failure actually becomes visible.

Consider a single agent working through a twelve-step task. A bad decision made at step 3 might not break anything visibly right away. The failure only surfaces at step 12, and the log, read in the ordinary way, points straight at step 12 as the problem. The actual cause, step 3, leaves no obvious trace by the time anyone is looking at the output.

Multi-agent systems compound this further, because now the error can cross from one agent to another. The agent that makes the original bad decision and the agent whose output ultimately fails can be entirely different components of the system, with several intermediate handoffs between them. DEMM-Bench gives this a formal name: the Decision Evidence Gap. Baselines that treat trace presence as evidence of accountability systematically overclaim what they can actually show.

Answering that governance question requires reconstructing the decision boundary itself. Credit assignment methods based on Shapley values, which distribute responsibility across a set of interacting steps, typically need Monte Carlo re-sampling, and the number of samples required grows rapidly as the task gets longer and as more agents get added to the system.

There is a further structural pressure working against any team hoping the problem will resolve itself as systems mature. More logging, more provenance, more policy records, none of it closes the gap on its own.

Five frameworks that apply counterfactual intervention to agent decision points

A handful of frameworks published through 2026 now put this causal-intervention approach into working systems, and each one tackles a different part of attributing which decision caused a failure.

Causal Agent Replay, built by Jaineet Shah (arXiv:2606.08275, submitted June 6, 2026, with open-source code available), models an entire agent run as a structural causal model and applies a do-operation to a chosen step, then replays the trajectory forward under the same stochastic policy. It defines a formal intervention algebra over agent steps, a single-step contrastive estimator with a rule for identifying the exact point of commitment, and a budget-bounded Monte Carlo Shapley estimator that splits causal credit across multiple interacting steps rather than assigning it all to one. That Shapley estimator is the framework's most important contribution for multi-step credit assignment, since it produces a formal distribution of responsibility instead of a single accused step.

CausalFlow, from Akash Bonagiri, Devang Borkar, Gerard Janno Anderias, Setareh Rafatirad, and Houman Homayoun (arXiv:2605.25338, submitted May 25, 2026), takes a different approach: targeted counterfactual repair. Rather than just identifying where a failure originated, it searches for the smallest possible intervention that would have turned a failed run into a successful one. The team evaluated it across four benchmarks, GSM8K for mathematical reasoning, MBPP for code generation, SealQA Hard for question answering, and MedBrowseComp for medical web browsing, covering more than 3,000 problems in total, and the framework converts a substantial share of failed executions into validated minimal repairs.

DCFA, built by Wang and colleagues (arXiv:2609.04749, September 2026), was designed specifically for multi-agent failure attribution. Tested on the Who&When benchmark across six different LLMs, DCFA improves step-level accuracy meaningfully over prior baselines, a meaningful gain given how low the baseline accuracy on that benchmark has been.

AgenTracer, presented at ICLR 2026, addresses the shortage of training data for failure attribution directly. Instead of relying on human annotators to label naturally occurring failures, which are rare and expensive to find, it builds its TracerTraj-2.5K dataset, more than 2,000 high-fidelity failure trajectories, by deliberately injecting known faults into multi-agent runs and replaying them counterfactually. That fault-injection method produces ground-truth attribution labels directly, since the researchers already know what error they introduced and where.

CausalLoss-Fin (arXiv:2609.25960, 2026) narrows the lens to financial agents and decomposes a realized loss into three separate pieces: an infrastructure effect, a policy differential measured against the best policy that could actually have been implemented, and a residual tied to the reference policy itself. That decomposition demonstrates something aggregate metrics are structurally unable to do: separating a loss caused by the agent's own decisions from a loss caused by the environment it was operating in.

Alongside these research frameworks, tooling for running and scoring agent evaluations has had to catch up to the same problem. Inspect AI, in its 0.3.12 release, fixed a bug where results went undefined when no scorer was supplied and fixed its viewer to properly handle samples that carried no scores. Changes like these matter because a scoring pipeline that silently drops unscored samples or crashes without a scorer produces exactly the kind of gap this entire discussion is about: a missing piece of evidence about what actually happened in a run. Morph has approached this same gap from the infrastructure side, building the kind of fast, resettable sandboxed environments that counterfactual replay depends on, since re-running a trajectory from an intervened step requires being able to reset and re-execute an agent's environment cheaply and repeatedly, not just log what happened once. For a team deciding where in its pipeline to add this kind of evaluation, the environment layer that supports repeated, isolated replay is as much a part of the answer as the attribution algorithm itself.

Whether LLMs can perform the counterfactual reasoning these frameworks require

Every one of these frameworks eventually runs into a question that causes the whole approach to strain: when an LLM is doing the counterfactual reasoning, or sitting as the judge inside the auditing loop, is it actually performing causal inference, or is it pattern-matching on language that sounds causal? The answer matters directly for anyone trusting an LLM-based component inside one of these systems, because a judge that only sounds causal will fail in exactly the cases where the real cause is non-obvious, which are the cases that matter most.

Zečević and colleagues raised this concern in 2023 with what they called the "causal parrot" argument: an LLM trained on text containing causal claims can recite the correlations between those claims and their causal-sounding language without ever performing actual causal inference over them. To explain why an LLM can sometimes appear to reason causally without doing so, they introduced the idea of meta structural causal models, a framework for understanding how a model trained on data describing causal relationships can produce causally-phrased output while never running an intervention of its own.

More recent work sharpens that diagnosis rather than settling it. Thinking Fast, Thinking Wrong (arXiv:2604.10511, 2026) finds that LLMs often know the right causal facts but struggle specifically when applying that knowledge requires overriding an intuitive, pattern-based prior. In other words, the failure isn't a blank spot in what the model knows. The failure occurs when the correct causal answer conflicts with what the model would otherwise be inclined to say by default, which is exactly the situation a counterfactual auditing loop is likely to create when the true cause of a failure is buried several steps upstream of where it became visible.

That distinction has a direct, practical consequence for anyone building or evaluating these frameworks. An LLM-based component can be a reasonable tool for proposing candidate interventions or summarizing a trace, but the actual causal claim, that intervening on step 3 rather than step 7 would have changed the outcome at step 12, needs to be settled by running the intervention and observing the result, not by asking a model to reason its way there from the log alone. The frameworks surveyed above that build in an explicit do-operation and re-execution step, rather than relying on an LLM's judgment of cause, are the ones built to survive this objection rather than be undone by it.