PaperlayerView exact revision on arXiv

Source-linked AI summary

Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores

Qiyao Yan, Chenpeng Wang, Liangming Pan

arXiv:2608.31068v1cs.AI

TL;DR

Wrong final answers do not necessarily indicate missing reasoning, because correct answer evidence may survive internally but fail to appear in output scoring. This paper diagnoses that gap with a target-label-free score correction, recovering 9–34 accuracy points for Qwen3.5 models and transferring to OLMo and Llama.

  • Problem

    Whether a wrong final answer reflects absent reasoning or failed expression of internal evidence is difficult to distinguish, making output-only evaluation potentially misleading.

    Source: §1 · p. 1

  • Method

    A three-stage diagnostic tests hidden-state evidence, frozen output readouts, and whether a two-parameter, target-label-free additive correction recovers per-example decisions.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Across controlled logic, ProofWriter, ANLI, and FOLIO, correction recovers 9–34 accuracy points for Qwen3.5 models, transfers to OLMo and Llama, and exceeds count-preserving permutation baselines.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Benchmark errors should be diagnosed across internal representations and answer scoring before being attributed to missing reasoning capability.

    Source: §6 · p. 9

    The correction assumes a supplied label prior and is tested only on label-balanced evaluation splits, leaving its use with unknown or imbalanced label distributions unresolved.

    Source: Limitations · p. 9

Abstract

from arXiv · show

When a large language model fails a reasoning task, it is often assumed to lack the underlying capability. However, this conflates a genuine absence of reasoning with a late-stage output bottleneck. We observe a consistent readout gap across diverse reasoning benchmarks: hidden-state probes successfully decode correct answers even when native sequence scoring completely collapses due to structural biases. To test whether instance-specific logic survives this collapse, we introduce a diagnostic protocol using a minimal, target-label-free additive correction. Fitting just two parameters on as few as 25 unlabeled examples recovers 9--34 accuracy points for Qwen3.5 models, transferring successfully to OLMo-2-1B and Llama-3.1-8B. Crucially, these recovered decisions persist on hard instances unresolved by simple lexical overlap and significantly exceed count-preserving permutation baselines. Our results show that many apparent zero-shot reasoning deficits are expression failures masking intact internal logic, urging a narrower interpretation of benchmark evaluations.

1 Introduction

The introduction distinguishes reasoning ability from final-answer expression, arguing that a late-stage scoring bottleneck can hide correct instance-specific logic. It presents a diagnostic correction and reports substantial recovery despite collapsed normal output scoring.

1 Introduction

  • The paper frames wrong final answers as potentially misleading because vocabulary-level structural biases can overwrite correct internal answers and produce apparently random predictions.The cited background describes correct reasoning as potentially present in hidden layers before the final output stage.Source: §1 · p. 1
  • The introduction distinguishes evidence that answers are represented from proof that native generation uses them, and aggregate calibration gains from repaired instance-level reasoning.Global answer-distribution shifts, such as balancing true and false outputs, do not guarantee that individual questions’ underlying logic was fixed or used.Source: §1 · pp. 1–2
  • The protocol compares hidden-state probes, single-token logits, and sequence scores, then applies a target-label-free additive correction that shifts global boundaries without changing prompt-specific candidate rankings.The correction uses only two parameters and no ground-truth labels, so it cannot teach new reasoning skills; recovery is offered as evidence that correct logical alignment remained in native outputs.Source: §1 · p. 2
  • With as few as 25 unlabeled examples, the two-parameter correction recovers 9–34 accuracy points on Qwen3.5 and transfers to OLMo-2-1B and Llama-3.1-8B.The introduction reports that probes decode correct answers while normal output scoring fails completely, and says stress tests indicate the recovery reflects semantic reasoning rather than shallow word matching.Source: §1 · p. 2

2 Related Work

Related work shows that models can encode reasoning-relevant information before producing an answer, while native scoring may fail to express it. This section motivates diagnosing output bottlenecks with per-example recovery tests and controlled logical-reasoning benchmarks.

Probing versus output scoring

  • Linear probes decode reasoning-relevant information before the output token, but bypass native scoring and establish capacity rather than practical use (Burns et al., 2023).Probe accuracy can also reflect statistical features ignored by the model’s native prediction path.Source: §2 · p. 2

Softmax bottlenecks and unembedding misalignment

  • The softmax bottleneck can distort a reasoning manifold through the static unembedding, while post-training may bias native scores toward frequent structural labels.The cited account links output-layer expressiveness limits to instruction-tuning and RLHF shifts in baseline probabilities.Source: §2 · p. 2

Output bias and per-example recovery

  • Per-example gain counts and label-count permutation baselines test whether additive offsets recover individual instances rather than merely reshape aggregate label distributions.Prior-conditioned offsets are evaluated through held-out rescue and permutation controls.Source: §2 · pp. 2–3, Figure 1 · p. 3

Logical-reasoning benchmarks as testbeds

  • Controlled deduction tasks and benchmarks including ProofWriter, FOLIO, and ANLI enable auditing prediction-pipeline components instead of equating wrong answers with absent reasoning.Their known ground truth, balanced labels, and controllable proof depth support separating representation failures from scoring bottlenecks.Source: §2 · p. 3

3 Method

The method diagnoses whether correct answers are encoded in frozen hidden states but obscured by native output scoring, then tests whether a minimal target-label-free offset can recover them. Its readouts and correction operate on frozen outputs rather than changing the model.

Diagnostic pipeline

  • The three-stage pipeline tests hidden-state answer encoding, frozen output-layer exposure, and per-example recovery under a minimal target-label-free correction.No training objective is added to the model; every component operates on frozen outputs.Source: §3 · p. 3

Readouts

  • At the answer slot, native logits follow the frozen unembedding geometry, while an independent logistic-regression probe tests whether the shared hidden state linearly separates labels.The shared answer-slot state avoids cascading effects from generated tokens; a prompt-end probe applies the same readout before the answer tag is appended.Source: §3 · pp. 3–4

Prior-conditioned offset correction

  • The correction fits two global offsets on unlabeled in-domain scores by grid-searching toward a target label prior, shifting thresholds without changing within-label rankings.One reference offset is fixed at zero, leaving two free parameters for three labels; the offsets cannot learn a new task mapping.Source: §3 · p. 4

Evaluation protocol and controls

  • Recovery is assessed on a disjoint evaluation set and must show positive per-example gains while surviving TF-IDF-missed-slice, label-count-permutation, and small-sample subsampling controls.The subsampling control fits offsets on 25 to 500 examples to test for a pre-existing threshold shift rather than data-intensive relearning.Source: §3 · p. 4

4 Experiments

Across four reasoning tasks, native sequence scores can collapse despite answer evidence remaining in hidden states, and a two-parameter unlabeled offset correction recovers per-example accuracy. Recovery is strongest when raw predictions are collapsed or biased and probes indicate the answer is encoded.

4.1 Setup and design rationale

  • The evaluation scales from controlled logic to ProofWriter, ANLI, and FOLIO, using forced-choice literal answer slots and a post-trained Qwen3.5-2B/4B/9B sweep.Cross-family OLMo-2-1B, Llama-3.1-8B, and Pythia tests are limited to ProofWriter to probe generalization boundaries.Source: §4.1 · pp. 4–5

4.2 Readout gap: string scoring fails while answer-slot evidence remains

  • Qwen3.5-2B answer-slot probes reach 0.918 in-domain and 0.661 lexical OOD while full-string scores remain at 0.333; Qwen3.5-9B probes reach 0.972/0.830 against the same collapsed scores.Base-checkpoint results and the audits further locate loss at both vocabulary projection and sequence aggregation; randomized-label probes remain at chance, and mean-normalized scores reproduce the collapse.Source: §4.2 · p. 5, Table 1 · p. 6

4.3 Label-bias correction recovers per-example evidence

  • Qwen3.5-4B/9B offset correction raises accuracy from chance to 0.570/0.602 on synthetic lexical OOD and to 0.653/0.678 on ProofWriter; every Table 2 configuration is significant (p ≤0.035).On adversarial ANLI, Qwen3.5-4B improves from 0.478 to 0.571. Gains in macro-F1 show that rebalancing label thresholds reorders candidate scores rather than defaulting to one frequent class.Source: §4.3 · p. 5, Table 2 · p. 6, Figure 2 · p. 7

4.4 Sample efficiency

  • With only 25 unlabeled examples, Qwen3.5-4B/9B improve ProofWriter accuracy by +0.216/+0.277, and increasing the fit budget to 1,000 changes held-out accuracy by at most 0.022.Recovery is positive for every collapsed Qwen3.5 row in all 30 subsamples; near-ceiling ANLI-9B is an exception because it has little remaining headroom.Source: §4.4 · p. 6, Table 3 · p. 7

4.5 Shortcut and label-count controls

  • On ProofWriter examples missed by TF-IDF, 25-example correction reaches 0.622/0.643 for Qwen3.5-4B/9B, with full-budget gaps above count-preserving nulls of +0.287/+0.305.The broader controls also find positive worst-case gains under prior perturbation, persistent collapse across prompt/verbalizer variants, and recovery across independently generated splits.Source: §4.5 · pp. 6–7, Figure 3 · p. 7, Table 4 · p. 8

4.6 Complementarity with source-task transfer

  • A source-task ensemble reaches 0.700 on ProofWriter OOD, and adding target-label-free corrected sequence scores raises hybrid accuracy to 0.720, a net gain of 20 examples.The margin is modest without a significance claim, but the hybrid also improves on TF-IDF-missed examples (0.682 to 0.707) and deepest proof strata (0.483 to 0.557); OLMo and Llama gains exceed permutation nulls, unlike unreliable Pythia changes.Source: §4.6 · pp. 7–8, Table 6 · p. 8

4.7 Boundary conditions: when recovery fails

  • Correction recovers per-example evidence only when probes indicate the answer is encoded and raw predictions are collapsed or strongly biased, rather than merely inaccurate.The paper notes that correction is not expected to help universally, with near-ceiling performance among the stated boundary cases.Source: §4.7 · p. 8

5 Discussion

Raw likelihood scoring can underestimate reasoning representations, but probe accuracy and calibration gains must be interpreted together: gains matter only when they exceed the label-count permutation baseline. The additive correction diagnoses collapsed distributions rather than providing a universal fix, and its partial recovery motivates more careful interpretation of zero-shot failures.

The conjunctive nature of evidence

  • Probe accuracy alone does not show that native scoring uses decoded answers, and calibration gains are meaningful only when they exceed the label-count permutation baseline.Pythia checkpoints fail this test, while high-baseline Qwen3.5-9B on ANLI shows no recoverable gain; the offset is therefore a descriptive diagnostic, not a universal fix.Source: §5 · p. 8

Mechanistic origins of the scoring collapse

  • The two-parameter additive correction removes a shared label-level threshold shift while preserving within-label rankings, allowing surviving task-relevant score differences to surface without teaching new reasoning paths.The collapse precedes instruction tuning in Base-model results (Table 11); structural-token frequency priors may originate in pretraining and be amplified by instruction tuning and safety alignment.Source: §5 · pp. 8–9

Why recovery is partial

  • On controlled-logic lexical-OOD, corrected accuracy reaches 0.57–0.60, below answer-slot probe accuracy of 0.77–0.83, because shared offsets cannot fix per-example scoring distortions.Surface-form competition between candidate strings (Holtzman et al., 2021) and context-dependent unembedding misalignment may contribute to the residual gap, which marks a next diagnostic target.Source: §5 · p. 9

Connections to chain-of-thought prompting

  • The proposed connection to chain-of-thought prompting (Wei et al., 2022) is speculative: intermediate tokens may distribute deduction beyond the single classification-token unembedding step where collapse occurs.The account predicts that collapse should shrink as evaluation moves from single-token to free-form answers.Source: §5 · p. 9

Implications for evaluation practice

  • A collapsed prediction histogram may mask reasoning rather than its absence, so inspect it before interpreting near-chance zero-shot accuracy as a capability failure.A lightweight, target-label-free offset fitted on a small unlabeled set can help distinguish masked capability from output-layer artifacts; default calibration may also affect leaderboard rankings.Source: §5 · p. 9

6 Conclusion

Across controlled logic, ProofWriter, ANLI, and FOLIO, a low-capacity, target-label-free offset recovers accuracy for larger Qwen3.5 models and transfers to OLMo and Llama. The findings support diagnosing internal representations and answer scoring before attributing benchmark errors to missing reasoning capability.

6 Conclusion

  • A low-capacity, target-label-free offset exposes held-out, example-specific structure and recovers accuracy for larger Qwen3.5 models across controlled logic, ProofWriter, ANLI, and FOLIO, with transfer to OLMo and Llama.The conclusion frames these results as evidence that a wrong prediction need not mean the underlying answer evidence is absent.Source: §6 · p. 9
  • The gains persist on TF-IDF-missed examples and exceed count-preserving permutation baselines, while Pythia and near-ceiling configurations delineate clear limits.Source: §6 · p. 9
  • Benchmark errors should be diagnosed across internal representations and answer scoring before being attributed to missing reasoning capability.Source: §6 · p. 9

Limitations

The diagnostic establishes that collapsed scores retain per-example evidence within a known label-balanced regime, but does not identify the underlying neural circuits or establish universality. Its assumptions, observational controls, and forced-choice scope limit what can be concluded or deployed.

Limitations

  • The correction assumes a supplied label prior; because evaluation splits are label-balanced, results do not establish a solution for imbalanced or unknown distributions, where jointly identifying priors and offsets remains open.The method is tested within the operating regime described by its prior.Source: Limitations · p. 9
  • Probes, same-position logits, and corrected scores are observational rather than causal; permutation gaps show example-specific alignment, not human-like semantic processing, and controls cannot eliminate shortcut explanations.Lexical, metadata, and matched counterfactual controls reduce, but cannot rule out, shortcut explanations.Source: Limitations · p. 9
  • The position-matched chain centers on Qwen3.5; OLMo and Llama transfer is promising, Pythia's negative results show the phenomenon is not universal, and open-ended generation is out of scope.Open-ended use would require robust candidate-answer strings and matched permutation controls.Source: Limitations · p. 9

Ethics Statement · A Appendix Overview

The study uses synthetic data, public benchmarks, and open-weight models, framing corrected scores as diagnostic evidence rather than a reliability guarantee. The appendix records the study’s procedures and the controls used to delimit interpretation.

Ethics Statement

  • Corrected scores recovering accuracy are diagnostic evidence, not proof of safe or faithful reasoning or a reliability guarantee.The authors identify overclaiming as the primary ethical risk and report scope controls, shortcut audits, and explicit boundaries to mitigate it.Source: Ethics Statement · p. 10
  • The study uses synthetic data, public benchmarks, and open-weight Qwen3.5, Pythia, OLMo, and Llama checkpoints under their respective research-use licenses.Source: Ethics Statement · p. 10
  • The study collects no personal data and introduces no deployed decision system.Source: Ethics Statement · p. 10

A Appendix Overview

  • The appendix documents data construction, model revisions, score computation, offset fitting, and uncertainty procedures.Source: A Appendix Overview · p. 12
  • Appendix controls cover surface and verbalizer effects, expanded readout audits, controlled-logic robustness, ProofWriter transfer, and ANLI/FOLIO diagnostics.Source: A Appendix Overview · p. 12
  • The appendix presents raw scores before label-free correction, followed by TF-IDF-missed, permutation, prior-sensitivity, and model-family controls.These controls are described as delimiting the interpretation.Source: A Appendix Overview · p. 12

B Data, Models, and Reproducibility · C Surface and Verbalizer Controls

The paper documents a reproducible, label-free evaluation pipeline and tests whether surface features, answer length, or prompt wording explain the observed sequence-score collapse. The controls show that the collapse persists across prompt and equal-token verbalizer variants.

B Data, Models, and Reproducibility

  • Held-out evaluations use at least 1,000 examples except the labeled FOLIO diagnostic; all evaluation splits are disjoint from unlabeled offset-fit data.FOLIO offsets are fit on 203 validation examples and evaluated on a separate prepared split of 1,001 examples.Source: B Data, Models, and Reproducibility · p. 12
  • The study uses models from 410M to 12B parameters; cached-score offset fitting adds negligible cost to an artifact suite requiring roughly 400 H100 GPU-hours.The paper uses a selected subset of the broader artifact suite’s results.Source: B Data, Models, and Reproducibility · p. 12
  • A deterministic two-dimensional search fits additive prior offsets label-free, freezes them before held-out evaluation, and uses bootstrap intervals, paired tests, sampled-ID repeats, and TF-IDF baselines for robustness.One label offset is fixed to zero; candidate pairs are ranked by prior fit, preferring lower collapse and smaller norm.Source: B Data, Models, and Reproducibility · p. 12
  • The Qwen3.5-4B answer-slot rerun uses 2,000 probe-training rows across three 1,000-example held-out splits, with a pinned revision and fixed prompt suffix.The suffix is literal “Answer: <answer>”; reruns use left padding and a 1,024-token maximum, excluding earlier values with misaligned prompt-format flags.Source: B Data, Models, and Reproducibility · pp. 12–13, Table 7 · p. 13
  • Controlled logic uses balanced true/false/unknown labels, nonce terms, and distractor rules, with ID, depth-OOD, lexical-OOD, and stress/counterfactual evaluation splits.The splits vary reasoning depth, template syntax, and generator seeds; the stress tests include counterfactual rule inversions.Source: B Data, Models, and Reproducibility · p. 13

C Surface and Verbalizer Controls

  • Surface-leakage tests examine metadata and template structure, and strong metadata baselines motivate TF-IDF-missed slices and permutation nulls as primary controls.Table 8 is the surface-leakage audit of TF-IDF and metadata baselines.Source: C Surface and Verbalizer Controls · p. 13, Table 8 · p. 13
  • Sequence-score collapse persists across prompt and equal-token verbalizer variants, ruling out answer-string length and formatting artifacts.The equal-token verbalizer audit holds verbalizer token length constant.Source: C Surface and Verbalizer Controls · p. 13, Table 9 · p. 13

D Expanded Readout Audit

The expanded audit finds that answer-slot probes retain an advantage over native sequence scoring, including in Qwen3.5 Base, and tests the robustness and limits of this probe-based readout.

D Expanded Readout Audit

  • Answer-slot probes remain well ahead of the best label-logit directions at the best native position; Qwen3.5 Base also shows probe–scoring divergence and collapse when partially informative logits are aggregated over the full candidate string.Selecting the best native readout position narrows the gap only slightly.Source: D Expanded Readout Audit · p. 13, Table 11 · p. 14, Table 12 · p. 14
  • Random-label fits and randomized or auxiliary targets bound spurious decodability, while cross-split answer-direction transfer measures probe variance under distribution shift and limits their diagnostic use.These controls help distinguish answer-related structure from arbitrary side-feature memorization.Source: D Expanded Readout Audit · p. 13, Table 14 · p. 14, Table 15 · p. 14
  • The audit quantifies the readout gap with bootstrap intervals and paired tests, and measures how probe accuracy depends on labeled training-set size.Source: D Expanded Readout Audit · p. 13
  • The collapse and label-free recovery persist across 20 prompt/verbalizer variants, rather than depending on one surface template.Source: Table 10 · p. 14

E Controlled-Logic Parallel Results and Label-Bias Details

Controlled-logic results test whether offset recovery survives seed changes, score normalization, prior perturbations, and lexical-shortcut and label-count controls. Across these checks, Qwen3.5 recovery is reported as consistent and tied to instance-specific alignment.

Controlled-Logic Parallel Results and Label-Bias Details

  • Mean-normalized candidate scores leave both the collapse and offset recovery intact, ruling out token-length normalization as their explanation.The section reports that this result is replicated across dependent generators and seeds.Source: E Controlled-Logic Parallel Results… · p. 14, Table 13 · p. 14, Table 21 · p. 16
  • Count-preserving permutation baselines support instance-specific alignment beyond label-distribution matching.The null compares corrected predictions with random permutations sharing the same label counts.Source: E Controlled-Logic Parallel Results… · p. 14, Table 22 · p. 16
  • Larger Qwen3.5 checkpoints recover held-out accuracy and macro-F1 under unlabeled offset correction, while Pythia rows mark the boundary.Positive Qwen3.5 margins persist when the target prior is perturbed, indicating reliance on internal rankings rather than a precisely tuned prior.Source: Table 18 · p. 15, Table 19 · p. 15
  • Held-out recovery is already positive with 25 unlabeled fit examples and saturates quickly, consistent with the two-parameter threshold shift.The larger Qwen3.5 checkpoints are above chance at 25 examples; the near-ceiling ANLI-9B row is a scope control.Source: Figure 4 · p. 14, Table 20 · p. 15
  • Across regenerated synthetic seeds, Qwen3.5-9B offset recovery is described as a model property, with positive margins persisting on recomputed TF-IDF-missed subsets.The seed-specific TF-IDF-missed nulls additionally compare predictions with label-count permutation baselines as the protocol’s strictest combined control.Source: Table 23 · p. 16, Table 24 · p. 16, Table 25 · p. 16

F ProofWriter Sequence Scores and Source-Task Evidence · G ANLI and FOLIO Diagnostics · H Auxiliary Evidence and Scope Controls

Across ProofWriter, ANLI, and FOLIO, the appendix tests whether offset-based recovery survives shortcut controls, sparse fits, and changes in task structure. Auxiliary tests also delimit when the recovery is interpretable and which scoring-path components are surface-sensitive.

F ProofWriter Sequence Scores and Source-Task Evidence

G ANLI and FOLIO Diagnostics

  • In adversarial, less-templated ANLI, recovery persists under permutation baselines and small-budget fits, extending the diagnostic beyond controlled logic tasks.The appendix reports positive Qwen3.5 recovery gaps beyond matched label-count permutations.Source: G ANLI and FOLIO Diagnostics · p. 17, Table 40 · p. 20
  • On FOLIO, offsets fitted on 203 validation examples are evaluated on 1,001 prepared training examples and clear label-count permutation nulls.The sparse-fit diagnostic retains gains under prior shifts and on TF-IDF-missed slices.Source: G ANLI and FOLIO Diagnostics · p. 17, Table 43 · p. 17, Table 44 · p. 17

H Auxiliary Evidence and Scope Controls

  • Auxiliary ablations and prompt grids diagnose mechanisms and surface sensitivity; complete held-out sequence-score results remain the primary evidence, separated from prompt or label-prior effects by per-example gains and permutation nulls.Adapter sweeps complement this evidence by showing target-domain signal in transferred representations before native scores are added.Source: H Auxiliary Evidence and Scope Controls · pp. 17–18

F ProofWriter Sequence Scores and Source-Task Evidence

  • ProofWriter mean-normalized scores replicate synthetic-task recovery, while fusing frozen source adapters with target sequence scores yields additive gains.The appendix also tests prior sensitivity and label-count permutation controls for ProofWriter recovery.Source: Table 28 · p. 18, Table 31 · p. 18, Table 29 · p. 18

H Auxiliary Evidence and Scope Controls

  • Positive permutation gaps for larger Qwen3.5, OLMo, and Llama configurations coexist with null or negative Pythia gaps and near-ceiling cases, ruling out a universal calibration effect.The narrower conclusion is that global offsets can expose example-specific evidence when raw rankings collapse but representations remain informative.Source: H Auxiliary Evidence and Scope Controls · p. 18
  • On ProofWriter hard slices, the sequence-augmented source-task ensemble exceeds frozen-baseline and permutation-null results across deeper-proof strata and TF-IDF-missed variants.Ablations attribute hybrid gains to complementary sequence-score evidence rather than a single architecture, with improvement across a wide fusion-weight plateau.Source: Table 32 · p. 19, Table 33 · p. 19, Table 34 · p. 19
  • ProofWriter, ANLI, and FOLIO diagnostics retain gains at small fitting budgets; ANLI Qwen3.5 recovery also exceeds matched label-count permutations.The cited diagnostics include small-budget ProofWriter hard slices, ANLI R2/OOD evaluation, and fractional FOLIO subsamples within the 203-example fit set.Source: Table 35 · p. 19, Table 38 · p. 20, Table 42 · p. 20

Related papers

Loading 2608.31068v1…