PaperlayerView exact revision on arXiv

Source-linked AI summary

The Error You See Is Not the Error You Made: Progression-aware Reasoning Origin for Reasoning Error Localization

Yiguo Wang, Ziyuan Yang, Yi Zou, Dan Lin, Rongsheng Li, Yi Zhang

arXiv:2609.33297v1cs.AI

TL;DR

Holistic and forward-sequential verification can provide limited evidence about where an error originated, since a later rejected step may only expose an earlier mistake. PRO combines incoming and outgoing evidence, selectively refines ambiguous regions, and attributes the source; the paper reports stronger first-error localization across verification methods and backbones and improvements across diverse reasoning settings.

  • Problem

    Holistic verifiers provide little positional evidence, and forward sequential checking can treat a later exposure point as the error source.

    Source: §1 · pp. 1–2

  • Method

    PRO is a training-free framework that combines incoming support and outgoing compatibility, selectively refines mismatches, and uses detector-conditioned attribution to localize error sources.

    Source: §1 · p. 2, §4.2 · pp. 4–5, §4.4 · p. 7

  • Results

    Experiments across diverse reasoning settings validate PRO's design, and open-form results report stronger first-error localization while maintaining competitive performance on correct traces.

    Source: §5.2 · p. 8, §6 · p. 9

  • Takeaways & Limitations

    The findings support progression-aware source attribution as a more faithful formulation of reasoning verification than treating the first rejected step as the error origin.

    Source: §6 · p. 9

Abstract

from arXiv · show

Verifying multi-step LLM reasoning requires more than determining whether a trace is correct: a useful verifier should identify where the reasoning first goes wrong. However, existing holistic methods provide little positional evidence, while forward sequential verification often treats the first rejected step as the error source. Under error propagation, this assumption can fail, since an earlier mistake may remain locally plausible and become observable only through its downstream consequences. We therefore rethink reasoning verification as a progression-aware error-source localization problem: rather than asking only where a reasoning trace first appears inconsistent, we ask which earlier step best explains how that inconsistency emerges along the trajectory. Based on this view, we propose Progression-aware Reasoning Origin (PRO), a training-free framework for first-error localization. PRO jointly models incoming support from the preceding context and outgoing compatibility with subsequent reasoning, selectively refines regions where these signals disagree, and finally performs detector-conditioned source attribution with intervention-based evidence to distinguish the true error origin from its propagated manifestations. We further formalize the gap between forward rejection and structural exposure, showing why incoming-side evidence alone is insufficient for reliable localization under error propagation. Experiments across open-form, medical, and structured reasoning tasks demonstrate consistent improvements over strong verification baselines, supporting progression-aware source attribution as a more faithful formulation of reasoning verification.

1 INTRODUCTION

Holistic verification gives little positional evidence, while forward checking can mistake a later exposure point for the step that introduced an error. PRO localizes error origins by combining progression-aware evidence, selective refinement, and source attribution.

1 INTRODUCTION

  • Under error propagation, the first rejected step may be a downstream exposure point rather than the step that introduced the error.A locally plausible step can become suspicious only when its consequences appear later in the reasoning.Source: §1 · pp. 1–2
  • PRO treats each reasoning step as both a conclusion supported by preceding context and a premise compatible with subsequent reasoning.Incoming and outgoing evidence jointly help distinguish the true source from an exposed failure.Source: §1 · p. 2
  • PRO selectively refines inconclusive regions, then uses the observed verification pattern to distinguish error sources from steps that inherit or expose errors.The process begins with coarse-grained trajectory verification and narrows candidate sources.Source: §1 · p. 2
  • The paper formalizes the discrepancy between forward rejection and structural exposure to motivate outgoing evidence in earliest-error localization.The introduction frames first-error localization as requiring more than purely forward local checking.Source: §1 · p. 2

2 RELATED WORK

Prior verification work ranges from holistic solution scoring to process-level assessment and inference-time critique, but these lines generally do not localize the earliest corrupted step. Process supervision also supplies step-level feedback and supports verifier training and reranking.

2 RELATED WORK

  • Early verifier work scores complete sampled solutions and reranks candidates, while newer methods assess steps, blocks, or structural relations.GoV uses graph-structured blocks with adaptive granularity; ProcessBench and DeltaBench study earliest-error detection.Source: §2 · p. 2
  • Process supervision provides step-level feedback, with automated labeling methods such as Math-Shepherd and OmegaPRM reducing annotation cost.These methods support verifier training, reranking, and reinforcement learning.Source: §2 · pp. 2–3
  • Critique, revision, and inference-time verification methods aim to improve or stabilize reasoning rather than locate its earliest erroneous step.The cited analysis also reports limits to intrinsic self-correction without strong external feedback or explicit verification signals.Source: §2 · p. 3

3 DETECTOR-STAGE GAP IN SEQUENTIAL VERIFICATION

The section distinguishes incoming admissibility from outgoing compatibility to explain why forward rejection can occur after an error has entered a trace. Its formal results show that structural exposure can precede forward rejection and cannot always be recovered from incoming-side evidence alone.

3.1 INCOMING ADMISSIBILITY AND STRUCTURAL EXPOSURE

  • Forward-only checking can reject a block after an earlier error has entered the trajectory, because that error may first become visible through a later transition.An exposed block may itself be erroneous or may reflect a consequence of an earlier error, so source attribution remains necessary.Source: §3.1 · p. 3
  • Incoming admissibility tests whether a block is supported by the problem and preceding reasoning, while outgoing compatibility tests whether it supports the next block.The mismatch set contains blocks that pass incoming checking but fail to support their immediate continuation.Source: §3.1 · p. 3

3.2 FORMAL PROPERTIES OF THE DETECTOR-STAGE GAP

  • Structural exposure occurs no later than forward rejection, and the inequality is strict for some traces.No detector relying only on incoming-side signals can recover structural exposure correctly for every decomposed reasoning trace.Source: §3.2 · pp. 3–4
  • The formal results motivate augmenting a forward-anchored detector with an outgoing-side structural test before source localization.PRO refines exposure candidates before using the surviving detector output to locate the source.Source: §3.2 · p. 4

4 PROGRESSION-AWARE REASONING ORIGIN

PRO localizes reasoning errors by tracking both support from prior context and compatibility with later reasoning. It refines ambiguous regions and uses detector-conditioned candidate attribution to distinguish an error’s origin from where it becomes observable.

4.1 OVERVIEW

  • PRO combines incoming support, outgoing compatibility, selective refinement, and origin-oriented attribution in a three-stage training-free framework.It separates where an inconsistency becomes observable from where the error is introduced.Source: §4.1 · p. 4

4.2 PROGRESSION-AWARE EVIDENCE MODELING

  • Incoming evidence tests whether a block follows from the problem and verified prefix, while outgoing evidence tests whether it supports the continuation.A mismatch marks an unresolved region: forward checking accepts the block, but its continuation is incompatible.Source: §4.2 · pp. 4–5

4.3 MISMATCH-TRIGGERED GRANULARITY REFINEMENT

  • PRO refines only mismatch regions by decomposing the adjacent-block window into finer units and checking whether the inconsistency persists.This distinguishes a coarse-decomposition artifact from a more robust local failure.Source: §4.3 · pp. 5–6

4.4 ORIGIN-ORIENTED CANDIDATE ATTRIBUTION

  • The refined detector position anchors a shortlist of source candidates, whose local repairs provide evidence for tracing the observed failure to its origin.A repair that removes the local inconsistency supports a candidate as upstream of the exposure; the detector position remains a competing hypothesis.Source: §4.4 · pp. 6–7

5 EXPERIMENTS

Across open-form, medical, and structured reasoning tasks, PRO generally improves first-error localization, with gains persisting across backbones and increasing with reasoning depth. The experiments also examine PRO’s components and use matched verification settings.

5.1 EXPERIMENTAL SETTING

  • The evaluation covers ProcessBench, a 450-trace MedReason benchmark, and Number Triangle Summation at four reasoning depths; PRO uses a detector-conditioned shortlist of k = 5.Experiments use deterministic decoding at temperature 0; ProcessBench’s holistic baseline uses the official leaderboard result.Source: §5.1 · p. 7

5.2 OPEN-FORM REASONING EXPERIMENT

  • PRO generally localizes first errors more strongly than holistic and forward-local verification on ProcessBench while retaining competitive correct-trace performance.The reported advantage persists across different backbone choices.Source: Table 1 · p. 7, §5.2 · p. 8, Figure 2 · p. 8

5.3 MEDICAL REASONING EXPERIMENT

  • On MedReason, PRO consistently improves first-error localization across backbones while reliably recognizing correct reasoning.The comparison is reported in Table 2; Figure 3 shows a stable advantage across models.Source: §5.3 · p. 8, Table 2 · p. 8, Figure 3 · p. 8

5.4 CROSS-BACKBONE STATISTICS

  • PRO improves over all baselines on ProcessBench and MedReason across backbones, with gains more pronounced on MedReason.The cross-backbone gains are averaged across four evaluated models.Source: §5.4 · p. 9, Figure 4 · p. 9

5.5 STRUCTURED ARITHMETIC REASONING

  • On Number Triangle Summation, PRO and GoV are comparable at short depths, while PRO’s advantage grows as reasoning depth increases.The authors associate this trend with greater distance between error origin and exposure.Source: §5.5 · p. 9, Table 3 · p. 9

5.6 ABLATION STUDY

  • Removing each PRO component weakens first-error localization, with the largest degradation occurring without top-k reranking.Removing outgoing evidence weakens discrimination of progression-inconsistent blocks, while removing refinement makes ambiguous detector outputs harder to resolve.Source: §5.6 · p. 9, Table 4 · p. 9

6 CONCLUSION

The conclusion frames PRO as a training-free approach to tracing exposed failures back to their likely origins. The accompanying problem setting and challenges distinguish source localization from detecting inconsistency and highlight the difficulty of choosing reasoning granularity.

6 CONCLUSION

  • PRO combines progression-aware evidence, selective refinement, and source-oriented attribution to trace exposed failures to likely origins.The conclusion says experiments across diverse reasoning settings validate this design.Source: §6 · p. 9

A.1 REASONING VERIFICATION

  • Holistic verifiers give a global judgment, while decomposition-based methods expose intermediate units; earliest-error localization remains difficult even for strong process verifiers.The discussion cites ProcessBench and DeltaBench for this difficulty.Source: §A.1 · p. 12

B.1 PROBLEM SETTING

  • First-error localization returns the earliest incorrect block, or −1 when the trace is valid, across arithmetic, free-form mathematical, and medical traces.The problem setting distinguishes this task from judging only the final output.Source: §B.1 · p. 13
  • The first exposed inconsistency may be downstream of the source error, so local suspicion alone cannot identify the origin reliably.The detector’s suspicious block is not necessarily the block that introduced the inconsistency.Source: §B.1 · p. 13

B.2 GOALS

  • The stated goals are faithful source localization, robustness to varying trace structure, and deployment without additional supervision or verifier training.The setting is black-box verification.Source: §B.2 · p. 13

B.3 CHALLENGES

  • Variable granularity creates a trade-off: fine units expose local faults but can lose causal context, while coarse units preserve context but blur the error location.The paper motivates refining granularity where ambiguity arises while retaining enough context for attribution.Source: §B.3 · p. 14

C THEORETICAL PROOF

The theoretical comparison shows that PRO’s structural-exposure index can precede ordinary forward rejection. Outgoing-side mismatch can expose a suspicious bridge before any incoming-side rejection occurs.

C.1 FORWARD REJECTION VERSUS STRUCTURAL EXPOSURE

  • Structural exposure can occur no later than forward rejection, and an outgoing-side mismatch can make it strictly earlier.A trace whose incoming-side tests all pass but has an outgoing-side mismatch witnesses strictness.Source: §C.1 · p. 14
  • The proof compares minimized index sets, using candidate-set inclusion to establish the ordering between structural exposure and forward rejection.The inclusion adds the mismatch set to the candidates for structural exposure.Source: §C.1 · p. 14

C.2 NECESSITY OF OUTGOING-SIDE INFORMATION

Incoming-side signals alone cannot universally recover PRO’s detector-stage structural-exposure index. PRO’s test-time procedure combines bidirectional checking and refinement with detector-conditioned candidate attribution.

C.2 NECESSITY OF OUTGOING-SIDE INFORMATION

  • Two traces can share identical incoming-side signals yet have different detector-stage structural-exposure indices, so incoming-only rules cannot recover the target for both.The proof attributes the difference to outgoing-side structure.Source: §C.2 · pp. 14–15

D THE MAIN STEPS OF PRO

  • PRO checks incoming admissibility and outgoing support against the immediate continuation, refining locally when the signals disagree instead of immediately rejecting.The theoretical pre-refinement exposure index is τ⋆; the implementation uses refined detector output τdet.Source: D THE MAIN STEPS OF PRO · p. 15
  • For surviving suspicious regions, PRO builds a detector-conditioned top-k shortlist, rewrites and re-verifies candidates under the same verified prefix, then reranks using repair evidence.If reranking returns no valid shortlist candidate, PRO falls back to the original detector index.Source: D THE MAIN STEPS OF PRO · p. 15
  • Across datasets, PRO uses the same prompt family, changing only the verification-unit definition; it assigns the terminal block outgoing score 1.This prevents the absence of a successor from downgrading the final block.Source: D THE MAIN STEPS OF PRO · p. 15

E.1 INFERENCE AND DATA CONFIGURATION

The inference and data configuration uses matched backbones within each setting and deterministic decoding for the main runs. Evaluations include complete ProcessBench test subsets and a balanced MedReason set.

E.1 INFERENCE AND DATA CONFIGURATION

  • All compared methods use the same backbone within each setting, with deterministic decoding at temperature 0 for main runs.Retry behavior follows the released task-specific scripts; PRO uses a detector-conditioned shortlist of k = 5.Source: §E.1 · p. 15
  • ProcessBench evaluation covers its complete GSM8K, MATH, and OlympiadBench test subsets.Source: §E.1 · p. 15
  • MedReason evaluation uses 450 traces: 225 unmodified and 225 corrupted counterparts balanced across nine source partitions.The injected error position is the first-error label, and the same evaluation set is used for every compared backbone.Source: §E.1 · p. 15

E.2 METRICS

The evaluation reports exact first-error localization F1 alongside Correct Accuracy and Error Accuracy when annotations allow. The metrics distinguish recognizing correct traces from identifying their first incorrect step.

E.2 METRICS

  • F1 denotes exact first-error localization F1 and is the primary metric, computed as the harmonic mean of Correct Accuracy and Error Accuracy.Source: §E.2 · pp. 15–16
  • Together, these metrics reflect recovery of the true source step rather than merely exposing a downstream failure.Source: §E.2 · p. 16
  • Correct Accuracy measures the proportion of fully correct reasoning traces judged as error-free.Source: §E.2 · p. 16
  • Error Accuracy measures the proportion of erroneous traces for which the method identifies the first incorrect step.Source: §E.2 · p. 16

F QUALITATIVE CASE STUDY

In a MedReason Type 1 diabetes case, an unsupported early causal commitment remains obscured by plausible intermediate reasoning until a factual error becomes explicit downstream. PRO recovers the annotated source step where GoV does not.

F QUALITATIVE CASE STUDY

  • The corrupted trajectory’s Steps 3 and 4 remain locally plausible, introducing HLA susceptibility, antigen presentation, CD8+ T cells, and inflammatory cytokines, while an unsupported causal commitment starts in Step 2 and becomes explicit at Step 5.Source: F QUALITATIVE CASE STUDY · p. 16
  • PRO identifies Step 2 as the source error, while GoV attributes the failure to Step 5, where the factual mistake becomes explicit.The figure’s example concerns a Type 1 diabetes reasoning trajectory.Source: Figure A1 · p. 17, F QUALITATIVE CASE STUDY · p. 17
  • PRO revisits the suspicious region with candidate-conditioned rewriting, checks consistency with the preceding context and final conclusion, then reranks surviving candidates by source plausibility.The inspected example is one in which GoV fails to recover the annotated source step and PRO localizes it correctly after rewriting and reranking.Source: F QUALITATIVE CASE STUDY · pp. 16–17

Related papers

Loading 2609.33297v1…