PaperlayerView exact revision on arXiv

Source-linked AI summary

What Process Evaluation of Coding Agents Actually Measures: Action, Task, and Step Are Three Different Levels

Jiawei He, Mengyu Shi, Jie jia, Xikai Yang, Dong Sun

arXiv:2608.22960v1cs.AI

TL;DR

Existing process evaluations often conflate action prediction, task uncertainty, and step attribution, leaving their measured target unclear. The paper introduces a structural-causal, replay-based framework with SCAE and finds that action and uncertainty signals operate at different levels, while full-trace attribution is biased and does not certify causal contribution.

  • Problem

    Existing process-level evaluations often treat action prediction, task uncertainty, and step attribution as the same problem, making their measurement target unclear.

  • Method

    The paper introduces a measurement framework using prefix-conditioned identification, replay-based SCAE estimation, and controlled manipulation of judges’ information.

  • Results

    Across 499 file-localization episodes from 12 repositories, next actions followed execution provenance, uncertainty was mainly task-level, and full-trace judges exhibited collider bias.

  • Takeaways & Limitations

    Process evaluation should be interpreted according to the level it measures rather than treated as a single benchmark target.

  • Takeaways & Limitations

    Of 190 intervention-estimated effects, 0 survived BH-FDR correction at q = 0.10, so individual step contributions could not be certified.

Abstract

from arXiv · show

Coding agents are increasingly evaluated not only by whether they solve a task, but also by how they execute it. However, existing process-level evaluations often treat action prediction, task uncertainty, and step attribution as if they were the same problem, which makes it unclear what such evaluations actually measure. In this paper, we introduce a measurement framework for process evaluation in coding agents and instantiate step-level causal attribution with SCAE, a replay-based estimator derived from a structural causal model of agent execution. Our framework combines prefix-conditioned identification, replay/intervention-based estimation, and controlled judge-information manipulation to study process evaluation at the action, task, and step levels. Experiments on 499 file-localization episodes from 12 repositories show that next actions are driven primarily by execution provenance rather than code-graph transitions, execution uncertainty is structured at the task rather than step level, and full-trace judges exhibit systematic collider bias, suggesting that current process evaluation often measures semantic relevance rather than certified causal contribution.

1 INTRODUCTION

The paper frames coding-agent process evaluation as three distinct measurement problems: predicting actions, characterizing task uncertainty, and attributing step-level causal effects. Using SCAE and controlled experiments, it finds that action and task signals are informative in different ways, while feasible step-level causal localization remains weak and full-trace judging is biased.

  • File-localization episodes provide replayable environments, logged prefixes, and verifiable terminal outcomes for process analysis.The experiments reconstruct prefixes, replay continuations, and estimate step-local effects under explicit assumptions.
  • Next actions are governed primarily by execution provenance from recent tool outputs rather than code-graph transitions.The strongest signal is what paths the agent has just seen, not the repository dependency graph.
  • Execution uncertainty is structured mainly at the task level, whereas the identifiable step-level causal quantity is too weak to localize at feasible replay cost.The paper therefore separates useful task-level uncertainty signals from certified causal attribution.
  • Process evaluation should distinguish action prediction, task uncertainty, and step attribution rather than treat process quality as one construct.
  • SCAE is a replay-based measurement framework derived from a structural causal model of agent execution.The framework makes step-local causal effects explicit and estimable in principle.
  • Full-trace judges shift blame when later steps become visible, indicating that current process attribution often reflects semantic relevance rather than certified causal contribution.The paper isolates this as collider bias in the evaluation mechanism.

2 RELATED WORK

The related work spans counterfactual reasoning, process supervision, coding-agent evaluation, uncertainty quantification, and structural navigation priors. The paper distinguishes these lines from its focus on identifying causal effects of substituting a step on the final outcome.

  • Counterfactual Reasoning in Discrete Sequential Systems: SCAE builds on Gumbel–max structural causal models and perturb-and-argmax representations for counterfactual reasoning over discrete action sequences.The framework uses replay rather than assuming a known transition mechanism.
  • Step-Level Credit Assignment and Process Supervision: Process reward models supervise intermediate steps primarily to improve policy learning, not to explain a particular step’s role in one trajectory.Their signals are often derived from Monte Carlo estimates over completed rollouts.
  • Process-Oriented Evaluation of Coding Agents: Annotation-based process evaluation estimates whether annotators mark steps defective, whereas this paper targets the causal effect of substituting a step on the final outcome.These are different estimands, so annotation agreement alone cannot establish causal validity.
  • Uncertainty Quantification for Language Models: Prior uncertainty research emphasizes single-turn confidence, Bayesian uncertainty decompositions, and conformal coverage rather than execution uncertainty in coding-agent trajectories.
  • Structural Priors for Agent Navigation: Coding-agent designs commonly embed repository similarity, dependency information, and repository-shaped tools, motivating the assumption that code structure guides next actions.

3 METHOD

The framework separates action prediction, task-level uncertainty, and step-level causal attribution, using SCAE to measure all three from replayable coding-agent executions. Its causal claims depend on prefix conditioning and explicit assumptions, while practical estimation trades attribution granularity against replay cost.

  • Framework overview: SCAE models coding-agent execution and produces action-, task-, and step-level measurements from trajectories, repository snapshots, and verifiable outcomes.The pipeline formalizes execution with a structural causal model, reconstructs prefixes, replays continuations, and estimates prefix-conditioned effects when assumptions hold.
  • Three readings: The fitted anytime value object supports attribution increments, pre-execution forecasting, and replay-based execution-uncertainty analysis, but these readings are not interchangeable.The uncertainty decomposition organizes task, policy, and model uncertainty, yet the three components are not claimed to be fully identified on real traces.
  • Replay and identification: Prefix replay reconstructs logged prefixes without modifying the agent, then resumes execution repeatedly to estimate continuation outcomes and local action variation.An entropy probe selects observational fork-resampling when actions retain randomness and intervention when they are effectively deterministic.
  • Step-level attribution: The step-level estimand asks whether replacing one action while holding the realized history fixed would change the terminal outcome.Prefix-only conditioning identifies this quantity under the stated observability, replayability, and decoding-exogeneity assumptions.
  • Step-level attribution: Full-trace judging and restart-based ablations target different conditioning sets from prefix-conditioned attribution, with full-trace judging introducing post-treatment bias.Conditioning on downstream actions or terminal outcomes changes the estimand and can shift attribution toward later steps.
  • Estimation scope: Replay estimates use sampled prefixes and finite budgets, so contributions are segment-level approximations whose granularity is traded against replay cost.Shared reconstructed prefixes and pinned environment noise form a common-random-numbers design that makes small replay budgets usable.

4 EXPERIMENTS

Across action, graph, reuse, uncertainty, and attribution analyses, execution provenance predicts local behavior better than code structure, while uncertainty and causal credit do not localize reliably at the step level.

  • Action prediction: On 67.0% first-visit steps, last-output provenance reaches top-3 = 0.326 versus 0.058 for the strengthened graph predictor and 0.072 for repository frequency.The paired improvement over the graph predictor is +0.136 at top-3 and +0.259 at top-10.
  • Action prediction: First-visit predictability is heterogeneous: handler, path, and novel-query steps reach top-3 values of 0.582, 0.185, and 0.029, respectively.The results identify novel free-text query generation as the hardest object type to predict.
  • Action prediction: Routing candidate construction by predicted tool improves top-1 from 0.109 to 0.150 and exploits tool–object dependence measured at I = 0.159 nats.The routed predictor also improves top-10 by +0.052 and reduces the effective candidate set.
  • Graph and reuse: The dependency graph does not explain local step order: one-hop, three-hop, and same-community lifts are 0.94, 1.02, and 1.01 against a within-episode permutation null.Immediate self-repetition is the exception, with lift 1.47, while the graph separates visited sets from random sets by 30.9× in internal edge density.
  • Graph and reuse: Reuse rates alone are uninformative, whereas temporal reuse patterns reveal behavior: adjacent object repetition is 1.83× chance and tool self-repetition is only 1.02 with p = 0.51.Tool transitions instead show mutual-information lift 1.13× chance, while object revisit patterns are temporally clustered.
  • Uncertainty and attribution: Instance identity explains replay-variance structure with ICC = 0.640 versus a null of 0.196, while only 5.3% of 190 intervention-estimated step effects reach uncorrected p < 0.05 and none survive BH-FDR.Step position and tool type contribute little, and the replay budget was 1901 continuations over 12.7 hours.

5 LIMITATIONS

The paper’s conclusions are constrained by task scope, replay cost, partially testable assumptions, judge-model validation, uncertainty validation, and graph coverage. These limitations narrow the claims without eliminating the central distinction among action prediction, task uncertainty, and step attribution.

  • The analysis covers file localization, so process structure in code editing, patch synthesis, and open-ended tool use may differ substantially.
  • 0 of 190 intervention-estimated effects survive BH-FDR correction at q = 0.10, preventing certification of which step mattered in an individual episode.Only 5.3% reach uncorrected p < 0.05, consistent with the nominal chance rate.
  • Selected-prefix estimates backed by finite replays provide segment-level approximations rather than dense per-step measurements.Increasing replay density is unlikely to remove the core limitation because outcome-relevant uncertainty is mainly task-level.
  • Decoding-noise exogeneity and intervention-branch transferability remain structural assumptions that cannot be fully verified from traces.The paper calls for testing sensitivity to departures from these assumptions.
  • Judge evaluation is limited by a modest, low-powered cross-model sample, and content-level scoring is less stable than the displacement mechanism.The strongest claim concerns information-set effects in full-trace judging, not every LLM-based process evaluator.
  • The full uncertainty decomposition is validated on a synthetic known-generating process, while real-trace evidence robustly supports primarily task-level outcome uncertainty.
  • Only 46.2% of agent objects resolve cleanly to dependency-graph nodes, limiting generalization of graph-based conclusions beyond the resolvable subset.
  • The central claim is scoped to replayable benchmarks where causal targets can be precisely defined, not to all process evaluation or all settings.

6 CONCLUSION

Existing process evaluation is too coarse when it treats action prediction, task uncertainty, and step attribution as one object. Using SCAE and controlled information-set comparisons, the paper finds these levels behave differently and should be evaluated according to the evidence they actually provide.

  • Action prediction, task uncertainty, and step attribution are distinct analytical levels whose conclusions do not automatically transfer.
  • The SCAE-centered framework uses prefix-conditioned identification, replay and intervention, and controlled judge-information changes to study the three levels.
  • Next actions are predicted primarily by execution provenance rather than code-graph transitions.Agents move mainly according to what they have just observed rather than repository structure alone.
  • Execution uncertainty has meaningful instance-level structure but little step-level structure, making outcome-relevant uncertainty primarily task-level.
  • Full-trace judges shift selected blame later when downstream steps become visible, indicating collider bias and downstream-informed semantic relevance rather than certified causal contribution.
  • Process-level signals can remain useful for prediction, uncertainty characterization, or semantic selection while step-level causal localization remains infeasible at practical replay budgets.
  • Process evaluators should state explicit targets, test assumptions, and interpret outputs according to the evidence they genuinely provide.

A NOTATION AND FORMAL MATERIAL

The appendix formalizes the notation, conditioning requirements, causal-ladder position, and failure modes underlying SCAE. It distinguishes identifiable replay-based quantities from cases where positivity, compaction, counterfactual divergence, or data-flow proxies limit causal interpretation.

  • A NOTATION AND FORMAL MATERIAL: The appendix supplies notation, conditioning-set consequences, causal-ladder placement, identification failures, and proofs supporting the formal framework.
  • A.2 CONSEQUENCES OF THE CONDITIONING SET FOR CURRENT PRACTICE: Proposition 1 licenses conditioning on the realized prefix; conditioning on later actions or outcomes opens biasing paths through downstream variables.
  • A.4 POSITION ON THE CAUSAL LADDER: SCAE provides action-level counterfactuals with prefix-conditioned continuation, not exact token-level counterfactuals, spanning Level III intervention and Level II.5 continuation.
  • A.5 WHERE IDENTIFICATION BREAKS: Positivity is the binding practical constraint: deterministic policies lack the comparison arm needed to recover necessity-type quantities, motivating measured overlap and soft intervention.
  • A.5 WHERE IDENTIFICATION BREAKS: Context compaction can discard treatment-relevant history, making the estimand unidentified rather than merely harder to estimate; randomized compaction could test this, but was not run.
  • A.5 WHERE IDENTIFICATION BREAKS: Counterfactual influence decays as factual and counterfactual paths diverge, so long-horizon Level III quantities can degrade toward Level II and should be reported accordingly.
  • A.6 PROOFS: The proof relies on joint exogeneity of decoding noises, conditional ignorability given the prefix, consistency, and a known assignment mechanism.
  • A.7 DATA-FLOW REACHABILITY IS NOT CAUSAL EFFECT: Data-flow inertness is not causal irrelevance: an empty grep can redirect the agent, while a large file read can leave later actions unchanged.

A.6.1 PROOF OF PROPOSITION 1

The proof establishes identification by conditioning on the realized prefix and separating current decoding noise from continuation noise. Under positivity, consistency, and the stated structural assumptions, the resulting quantity follows the one-step g-formula.

  • Given a realized prefix, the current action is a deterministic function of the current decoding noise, with the structural state fixed.
  • Joint exogeneity makes current decoding noise independent of both the prefix and continuation noise.
  • Under an intervention fixing action a, recursively generated observations, states, and the terminal outcome depend on continuation noise and environment noise.
  • These independence and structural relations yield conditional ignorability of the potential outcome given the prefix.
  • Consistency and positive action probability identify the interventional expectation through the one-step g-formula with a known assignment mechanism.
  • Unlike dynamic-treatment settings dominated by time-varying confounding, this proof treats logged policy context as sufficient while positivity remains the binding obstacle.

A.6.2 PROOF OF THEOREM 1

The proof decomposes each step’s outcome change into action and observation contributions, while distinguishing observational identification from intervention when needed. It also establishes that the estimator’s baseline is the policy’s own action distribution and that finite replay estimates are approximate.

  • A.6.2 PROOF OF THEOREM 1: The telescoping argument expresses total outcome change as the sum of per-step increments from v_0 to v_T.The terminal outcome is deterministic given the full trajectory, while v_0 is the expected outcome before execution.
  • A.6.2 PROOF OF THEOREM 1: Inserting the action between prefix and full observation separates each step into an action effect and an observation contribution.The observation term holds the action fixed, while the action term averages over the policy’s action distribution.
  • A.6.2 PROOF OF THEOREM 1: Applying the identification proposition converts observational conditional expectations into interventional counterparts wherever the required conditions hold.This substitution yields the action effect relative to the policy’s own baseline.
  • A.6.2 PROOF OF THEOREM 1: The step effect compares the observed action with what the same policy would typically have done, not with an optimal action.Finite replay estimates carry sampling error, which the paper uses as an internal consistency check rather than treating them as exact.
  • A.6.2 PROOF OF THEOREM 1: SCAE estimates step contributions by truncating logged prefixes, replaying continuations, and branching between observational and intervention procedures.The algorithm pins the environment, uses byte-identical continuation prompts, and executes substituted actions to obtain genuine observations.
  • A.6.2 PROOF OF THEOREM 1: Byte-identical continuation prompts and genuine execution of substituted actions are necessary because changing either would alter the estimand.A fabricated observation would intervene on the observation rather than only on the action.

B.2 EXPERIMENTAL SETTINGS

The experiments evaluate action prediction, graph structure, execution uncertainty, and attribution with controlled reference sets and replay conditions. The design measures overlap explicitly, manipulates judge information, and acknowledges scope limits imposed by high task success and unimplemented alternatives.

  • B.2 EXPERIMENTAL SETTINGS: The agent uses Codex CLI with gpt-5.5-0424-global in a read-only sandbox, with no server-side approval requests permitted.The experimental setup exposes thread paths for forking and sets a 900-second per-turn timeout.
  • B.2 EXPERIMENTAL SETTINGS: Environment pinning achieved 1.00 byte-identical replay over 100 actions.Locale, timezone, terminal width, hash seeds, tool ordering, and truncation points are fixed.
  • B.2 EXPERIMENTAL SETTINGS: Attribution uses 8 replays per point and five equidistant points per episode, while entropy probes use 4–6 replays and τH = 0.The observational branch is taken exactly when replays realize at least two distinct action buckets.
  • B.2 EXPERIMENTAL SETTINGS: The judge comparison holds model and rubric fixed while varying visible trace information between full-trace and prefix-only conditions.The full-trace judge sees the entire trajectory and outcome, whereas the prefix-only judge sees a fixed 60% prefix without the outcome.
  • B.2 EXPERIMENTAL SETTINGS: The study compares predictors and nulls on identical points, including execution-provenance, graph, permutation, uncertainty, and attribution references.Attribution includes three judge models, a prefix/full-trace ablation, five structural rules, and an exact random-selector expectation.
  • B.2 EXPERIMENTAL SETTINGS: Back-filled process reward models and ablation-with-restart are not implemented, leaving those comparisons outside the experimental evidence.The paper treats both as gaps rather than claiming empirical results for them.
  • B.2 EXPERIMENTAL SETTINGS: Overlap is present at 50% of 10 probed decision points, with position correlation +0.072, requiring per-step branch selection.The paper reports three different within-episode overlap patterns rather than assuming a globally observational or interventional estimator.
  • B.2 EXPERIMENTAL SETTINGS: The corpus’s high aggregate success rate constrains binary attribution, so attribution is restricted to multi-target instances and selective-risk metrics accompany calibration.Single-target instances are often solved, leaving little headroom for the binary outcome.

C.2 ADDITIONAL MEASUREMENTS BEHIND THE ACTION-LEVEL RESULT

Additional measurements explain why execution provenance predicts next actions: the agent’s recently mentioned paths form a small action space. They also quantify the trade-off between provenance-history coverage and candidate-set size, while checking sample expansion and calibration.

  • C.2 ADDITIONAL MEASUREMENTS BEHIND THE ACTION-LEVEL RESULT: Agents mention 50.5 distinct files per episode but visit 6.5, yielding an 18.0% conversion rate.This makes the agent’s provenance set much smaller than the repository and explains why provenance prediction is sharper than graph ranking.
  • C.2 ADDITIONAL MEASUREMENTS BEHIND THE ACTION-LEVEL RESULT: The strengthened graph predictor has thousands of median effective candidates, compared with 3.46 for provenance prediction.The graph ranks the repository broadly, whereas provenance ranks files already surfaced by the agent.
  • C.2 ADDITIONAL MEASUREMENTS BEHIND THE ACTION-LEVEL RESULT: Using the full observation history adds +0.011 top-3 performance with CI [+0.005, +0.019] but increases effective candidates from 3.46 to 18.8.The last-output variant is reported as the headline because it is sharper and cheaper.
  • C.2 ADDITIONAL MEASUREMENTS BEHIND THE ACTION-LEVEL RESULT: Dropping the code graph expands evaluation from 509 decision points to 7616 points over 488 episodes.On the overlapping graph-built subset, the headline metric differs by under 20%, with the residual gap attributed to object-type composition.
  • C.2 ADDITIONAL MEASUREMENTS BEHIND THE ACTION-LEVEL RESULT: Tool identity and object type have mutual information 0.159 nats versus a permutation null of 0.006, with p = 5 × 10^-4.This dependence makes a factored P(tool)P(obj) model misspecified; routing retains ECE = 0.012.

C.3 ADDITIONAL MEASUREMENTS BEHIND THE GRAPH AND REUSE RESULTS

The graph and reuse analyses separate repository-region structure from movement dynamics and control for leakage in gold-file measurements. Their nulls show that some apparent sequential structure is explained by the visited multiset rather than ordering.

  • C.3 ADDITIONAL MEASUREMENTS BEHIND THE GRAPH AND REUSE RESULTS: The observed adjacent-step one-hop rate is 0.284 versus a within-episode permutation expectation of 0.301.The reported one-hop lift of 0.94 therefore indicates that the graph describes the working region rather than the agent’s movement.
  • C.3 ADDITIONAL MEASUREMENTS BEHIND THE GRAPH AND REUSE RESULTS: 65% of gold files were already visited before ranking, so leakage-controlled analyses use only unvisited gold files.This leaves n = 39 unseen and n = 54 prefix-seeded instances.
  • C.3 ADDITIONAL MEASUREMENTS BEHIND THE GRAPH AND REUSE RESULTS: Reuse statistics are computed over 487 episodes, and the permutation null for reuse rate has standard deviation exactly zero.The number of first occurrences is determined by the distinct-file multiset, making reuse rate invariant to order.

C.4 THE QUERY CHANNEL AND A MEASURED PARSER DEFECT

The query channel’s measured parser defect initially inflated apparent next-action predictability, but correction preserved the conclusion that novel search queries constrain forecasting. Uncertainty decomposition and forecasting remain weakly identified on real traces.

  • Measured parser defect: 26.6% of command segments fail shlex parsing, affecting 75.3% of query objects and causing some search patterns to collapse.The fallback splits on whitespace, retaining quote characters and truncating malformed commands.
  • Measured parser defect: −0.036 top-3 results from repairing parser-induced fragments, reducing the same model from 0.067 to 0.031.The fragments represented a spurious repetition mode, so removing them made queries less predictable.
  • Interpretation: The conclusion that queries bind next-action prediction survives because the parser defect biased predictability upward rather than creating the constraint.The authors contrast this query channel with execution provenance from paths recently seen in tool outputs.
  • Uncertainty measurements: 0 of 190 effects survive FDR correction, so the earlier claim that high-entropy steps carry more significant effects does not replicate.The stratified difference is +0.056 with CI [-0.008, +0.135], while continuous diagnostics show no relationship.
  • Uncertainty measurements: Only Upolicy is estimated directly on real traces, at 0.049 from 23 instances with K = 4 repeated rollouts.The three-way decomposition is validated on synthetic data, where parameter recovery correlation is 0.994 and mean absolute probability error is 0.022, not on these traces.
  • Forecasting endpoint: Leave-one-repository-out AUROC is 0.481 over 7 covariates, while ECE is 0.015 and the risk–coverage curve remains uninformative.The 93.99% base rate makes low ECE compatible with an almost constant predictor; evaluation covers 12 repositories.

C.9 ADDITIONAL MEASUREMENTS BEHIND THE ATTRIBUTION RESULTS

Additional attribution measurements show that full-trace judges select gold-provenance steps, but their verdicts are difficult to compare with causal estimates because the reference selector is noisy and selection spaces differ.

  • Judge displacement: The collider contrast shifts mean normalized position from 0.214 with prefix-only information to 0.751 with the full trace.The median displacement is +0.667, placing the prefix-only judge in the first quarter and the full-trace judge in the last quarter.
  • Provenance enrichment: 0.900 of full-trace top-1 verdicts fall on gold-provenance steps, but only 32.9% of verdicts lie inside the estimator’s window set.The directly comparable subset contains n = 23 episodes, so unrestricted and comparable-subset baselines are reported separately.
  • Cross-model replication: Positive displacement appears in all 3 of 3 judges, with enrichment of 0.826, 0.565, and 0.826 versus a common-episode baseline of 0.443.The replication uses one additional general-purpose model and one code-specialised model, with 48 API calls per judge and byte-identical prompts.
  • Attribution sample: 70 attribution episodes span 10 repositories, including 19 failing episodes, with mean replay variance of 0.034.High-entropy steps have mean normalized position 0.416, indicating that overlap is not concentrated early in the trajectory.
  • Reference reliability: The identified top-1 causal reference does not survive FDR correction, with 0 of 190 effects remaining significant.It therefore behaves like a draw over approximately 5 estimation points while judges select among approximately 20 steps.
  • Reference reliability: The statistics R, Z, and mean blamed contribution inherit noise from b∆ and cannot distinguish a defensible alternative criterion from absent causal signal.Only 54 episodes are non-degenerate and 54.7% of estimation points have b∆ exactly zero, so provenance enrichment and information-set isolation provide the adjudicating comparisons.
  • Estimands: Annotation-agreement evaluations and SCAE target different estimands: rater-marked defects versus the terminal-outcome effect of substituting one step.Their calibration targets also differ, and annotation agreement does not validate causal attribution because judged defects can have zero causal contribution.

D REPRODUCIBILITY, PRE-REGISTRATION, AND FALSIFIABILITY

The paper documents reproducibility procedures, preregistered outcomes, and design corrections that delimit how its attribution and evaluation claims should be interpreted. It also identifies observations that would directly falsify the framework or its central empirical claims.

  • Reproducibility: Every reported number and figure is regenerated by a CPU-only, fixed-seed script whose outputs are written to JSON.Separate commands reproduce the uncertainty self-test, entropy summary, localization and forecasting tables, and data figures.
  • Outcome definition: The primary localization criterion counts gold-file containment, while exact set equality falls from 93.99% to 15.23% because agents name excluded test files.A filtered comparison that removes test paths matches the primary criterion within rounding, making exact equality diagnostic of naming behavior.
  • Outcome definition: Hard-resetting repositories to their base commits prevents agents from reading answers from their own working trees and inflating localization accuracy.Running the same pipeline on distributed snapshots would measure leakage rather than the intended localization ability.
  • Attribution limitations: Monte Carlo error attenuates estimated contributions toward zero, so the data-flow reachability null requires SIMEX or larger m before it can be asserted conclusively.The stated bias direction is favorable to the reported null, but the current result remains an observation subject to measurement error.
  • Attribution limitations: Equidistant estimation points produce segment effects whose sum need not satisfy Eq. equation 9 exactly.The paper therefore omits additivity checks as evidence; contiguous windows would be required for a genuine test.
  • Pre-registration: Four preregistered entries failed as stated: monotone entropy decay, the forecasting usefulness threshold, the degenerate-outcome range, and D1.Three directional predictions exceeded their fixed intervals, which the paper reports as intervals that were too conservative rather than as successes.
  • Pre-registration: The paper withdraws post hoc uses of R, mean blamed contribution, and Z because they were introduced after observing displacement and rely on an invalid causal reference.The provenance-class enrichment and information-set isolation contrast remain the adjudicating comparisons and required no additional agent execution.
  • Design correction: A corrected prefix-only judge uses an advance-fixed truncation fraction independent of the causal estimate after the original design mechanically forced positive displacement.The initial truncation ended just before the selected step, capping reachable positions even under a null of no descendant-conditioning effect.
Loading 2608.22960v1…