Source-linked AI summary
Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores
Qiyao Yan, Chenpeng Wang, Liangming Pan
TL;DR
The paper asks whether wrong benchmark answers necessarily indicate missing reasoning, given that internal representations may retain correct answers while output scoring collapses. It introduces a constrained, target-label-free additive diagnostic and finds substantial recovery across tasks and models, while identifying scope limits around label priors and causal interpretation.
Problem
Wrong final answers may conflate absent reasoning with a late-stage output bottleneck, while probe evidence alone does not show that native generation uses represented information.
Method
The paper compares probes and native scores, then fits a two-parameter, target-label-free additive offset on unlabeled examples while testing held-out and permutation controls.
Results
The protocol recovers 9–34 accuracy points from Qwen3.5 models using as few as 25 unlabeled examples, with transfer to OLMo-2-1B and Llama-3.1-8B.
Takeaways & Limitations
Benchmark errors should be diagnosed across internal representations and answer scoring before being attributed to missing reasoning capability.
Takeaways & Limitations
The diagnostic is tested under label-balanced splits with a supplied prior and does not establish causal neural circuits or guarantee semantic processing.
Abstract
from arXiv · showhide
When a large language model fails a reasoning task, it is often assumed to lack the underlying capability. However, this conflates a genuine absence of reasoning with a late-stage output bottleneck. We observe a consistent readout gap across diverse reasoning benchmarks: hidden-state probes successfully decode correct answers even when native sequence scoring completely collapses due to structural biases. To test whether instance-specific logic survives this collapse, we introduce a diagnostic protocol using a minimal, target-label-free additive correction. Fitting just two parameters on as few as 25 unlabeled examples recovers 9--34 accuracy points for Qwen3.5 models, transferring successfully to OLMo-2-1B and Llama-3.1-8B. Crucially, these recovered decisions persist on hard instances unresolved by simple lexical overlap and significantly exceed count-preserving permutation baselines. Our results show that many apparent zero-shot reasoning deficits are expression failures masking intact internal logic, urging a narrower interpretation of benchmark evaluations.
1 Introduction
The paper argues that wrong final answers can reflect a late-stage output bottleneck rather than absent reasoning. Its diagnostic protocol tests this distinction and finds substantial recovery from a minimal unlabeled correction.
- Hidden-state probes can decode correct answers even when native output scoring selects the wrong label.
- The paper addresses the gap between probe evidence and native generation by tracing where reasoning signals collapse.
- A two-parameter, target-label-free additive correction cannot teach new reasoning or change within-prompt candidate rankings.
- 9–34 accuracy points are recovered using as few as 25 unlabeled examples across controlled and natural-language reasoning tasks.
- Recovery survives shortcut stress tests and transfers from Qwen3.5 to OLMo-2-1B and Llama-3.1-8B.
2 Related Work
Prior work shows that models may encode reasoning information internally while output scoring is distorted by external observers, output-layer constraints, and systematic biases. This paper combines diagnostic scoring corrections with per-example controls on logic benchmarks to test whether recovered decisions reflect individual-instance structure.
- Hidden-state probe accuracy demonstrates representational capacity but can reflect features ignored by the model’s native prediction path.
- Softmax and unembedding constraints can distort a complex reasoning manifold when mapping hidden states into vocabulary logits.
- Existing calibration methods mitigate aggregate output biases, but typically do not establish whether individual corrected decisions use example-specific logic.
- Figure 1 frames additive offsets, held-out rescue, and permutation controls as tests of whether collapsed scores retain usable structure.
- Controlled deduction, ProofWriter, FOLIO, and ANLI provide balanced labels and known ground truth for auditing prediction components.
3 Method
The method localizes answer failures across hidden states, native logits, and full candidate-string scores, then applies a constrained prior-conditioned offset to frozen outputs. Held-out evaluation and per-example controls test whether any recovery reflects retained structure rather than label-count or lexical shortcuts.
- The three-stage pipeline probes hidden states, tests frozen output readouts, and applies a target-label-free correction to frozen outputs.
- Candidate answers are evaluated by summing teacher-forced log-probabilities over fixed continuations, with the highest-scoring candidate selected natively.
- Native same-position label logits depend on the frozen unembedding geometry, so large structural biases can dominate logic encoded in the hidden state.
- Answer-slot probes use an independent logistic-regression hyperplane at the decision-point hidden state, outside vocabulary-projection constraints.
- The correction fixes one offset to zero and fits two remaining parameters by matching predicted label fractions to a target prior.
- Global offsets preserve within-label rankings and therefore act as threshold shifts rather than a learned reasoning-task mapping.
- In a Qwen3.5-9B example, offsets change the winning candidate from unknown to the gold label true.
- The fit and evaluation sets are disjoint, and valid recovery must show positive per-example gains while surviving lexical and permutation controls.
4 Experiments
Across controlled and natural-language reasoning tasks, the experiments show that collapsed sequence scoring can be substantially improved by a small additive correction, with gains persisting beyond lexical shortcuts and label-count matching.
- Readout gap: Answer-slot probes reach 0.918 in-domain and 0.661 lexical OOD for Qwen3.5-2B, while full candidate-answer string scores remain at chance (0.333).For Qwen3.5-9B, probes reach 0.972 and 0.830 against the same collapsed string scores.
- Readout gap: The mismatch is geometric rather than representational: hidden states encode the answer, but pretrained unembedding geometry fails to expose it.Base Qwen3.5 checkpoints show the same divergence, indicating it is not tied to instruction tuning.
- Label-bias correction: The two-parameter correction raises Qwen3.5-4B/9B accuracy from chance to 0.570/0.602 on synthetic lexical and 0.653/0.678 on ProofWriter.On ANLI, Qwen3.5-4B improves from 0.478 to 0.571; every Table 2 configuration is statistically significant at p ≤0.035.
- Label-bias correction: Before correction, Qwen3.5-9B predicts unknown for 999 of 1,000 lexical-OOD examples; after correction, score differences reorder candidates correctly without target labels.The parallel macro-F1 gain indicates the correction is not merely selecting a frequent class.
- Sample efficiency: At 25 examples, Qwen3.5-4B/9B improve on ProofWriter by +0.216/+0.277, while increasing the fit budget to 1,000 changes held-out accuracy by at most 0.022.Recovery is positive across all 30 subsamples, with near-ceiling ANLI-9B as the exception because additional data cannot create headroom.
- Shortcut and label-count controls: On TF-IDF-missed ProofWriter examples, Qwen3.5-4B/9B reach 0.622/0.643 and exceed count-preserving nulls by +0.287/+0.305.These controls remove premise–hypothesis overlap shortcuts and test whether corrected predictions align with individual examples rather than only label frequencies.
- Robustness and scope: Prior perturbations leave worst-case Qwen3.5-4B/9B gains positive on synthetic lexical (+0.193/+0.242) and ProofWriter (+0.281/+0.294), whereas Pythia-12B reaches −0.021.Across prompt and verbalizer variants, collapse persists in 87–96% of configurations, and regenerated splits preserve Qwen3.5-9B recovery.
- Hard-instance audits: On hard slices, the hybrid improves from 0.682 to 0.707 for TF-IDF-missed examples and from 0.483 to 0.557 on ProofWriter strata with ≥2 hops.The margin is described as modest, directionally consistent, and not assigned a significance claim.
5 Discussion
The discussion frames recovery as evidence that native sequence scores can preserve instance-specific logic beneath shared output biases, while emphasizing that the correction is diagnostic, partial, and not universally causal. It also connects the bottleneck to output-layer aggregation and cautions that evaluation should inspect internal representations and scoring together.
- Positive calibration gains are meaningful only when they exceed count-preserving permutation baselines; otherwise, the correction is merely descriptive.The authors treat high probe accuracy as evidence of decodability, not native utilization, and identify Pythia failures and a near-ceiling ANLI row as limits.
- The additive correction works because task-relevant score differences survive beneath a dominant label-dependent offset, without changing within-label rankings.The correction removes a shared threshold shift rather than introducing new reasoning paths.
- Corrected accuracy reaches 0.57–0.60 on controlled-logic lexical-OOD data, versus 0.77–0.83 for answer-slot probes, showing that global offsets expose only part of the gap.Per-example surface-form competition and context-dependent unembedding misalignment remain outside the correction’s scope.
- Mapping multi-hop deduction into one classification token may concentrate burden at the collapsed unembedding step, whereas chain-of-thought distributes it across intermediate tokens.The proposed connection to chain-of-thought prompting is explicitly speculative and yields a prediction for future evaluation.
- Prediction histograms and lightweight unlabeled offsets can help distinguish absent capability from output-layer artifacts before zero-shot failures are interpreted as reasoning deficits.The discussion warns that leaderboard rankings may penalize poor default calibration rather than poor reasoning.
6 Conclusion
Across controlled logic and natural-language benchmarks, the paper finds that wrong sequence predictions can coexist with recoverable answer evidence. A lightweight offset exposes held-out structure and transfers across models, but permutation tests and boundary cases constrain the interpretation.
- Wrong predictions need not indicate absent underlying answer evidence across controlled logic, ProofWriter, ANLI, and FOLIO.The conclusion reports recovery for larger Qwen3.5 models with transfer to OLMo and Llama.
- Recovered gains persist on TF-IDF-missed examples and exceed count-preserving permutation baselines, while Pythia and near-ceiling configurations mark clear limits.
- Benchmark errors should be diagnosed using both internal representations and answer scoring before being attributed to missing reasoning capability.
Limitations
The diagnostic assumes a known label prior and therefore establishes evidence recovery only for label-balanced evaluations, not unknown or imbalanced deployments. It is observational rather than causal, and its generality is limited across architectures and task formats.
- The correction is validated under label-balanced splits with a supplied prior, not under imbalanced or unknown label distributions.Inferring both the prior and offsets from the same unlabeled score matrix without additional supervision remains open.
- The protocol triangulates representation and scoring but does not isolate the neural circuits responsible for the bottleneck.Permutation gaps and shortcut controls do not establish human-like semantic processing or eliminate all shortcut explanations.
- The phenomenon is not universal across architectures, and extending the framework beyond forced-choice tasks to open-ended generation remains outside scope.Transfer results on OLMo and Llama are promising, while negative Pythia results mark an architectural boundary.
Ethics Statement
The study uses synthetic data, public benchmarks, and open-weight checkpoints for interpretability and evaluation, without collecting personal data or deploying a decision system. Its main ethical risk is overclaiming recovered accuracy as evidence of safe or faithful reasoning.
- The study evaluates synthetic data, public benchmarks, and open-weight checkpoints without collecting personal data or deploying a decision system.The checkpoints include Qwen3.5, Pythia, OLMo, and Llama, used within their respective research licenses.
- The primary ethical risk is that corrected-score accuracy could be misconstrued as proof of safe or faithful internal reasoning.The paper mitigates this risk through scope controls, shortcut audits, and explicit boundaries.
B Data, Models, and Reproducibility
The study uses balanced controlled-logic and benchmark evaluations with disjoint held-out splits, pinned model revisions, deterministic offset fitting, and audits against surface leakage and formatting artifacts. Experiments span multiple reasoning depths, lexical variants, counterfactuals, and model configurations.
- Data and splits: Principal held-out evaluations use at least 1,000 examples, with fixed three-way labels and a uniform prior; FOLIO uses 203 validation and 1,001 prepared test examples.Controlled-logic, ProofWriter, ANLI, and FOLIO use the same three-way diagnostic setup, with FOLIO explicitly labeled as an exception.
- Compute and fitting: The artifact suite required roughly 400 H100 GPU-hours, while cached-score offset fitting adds negligible cost through deterministic two-dimensional grid search.Models range from 410M to 12B parameters.
- Compute and fitting: Additive prior offsets are selected by deterministic two-dimensional search, then frozen before held-out evaluation with bootstrap and paired-test robustness reporting.One label offset is fixed to zero, leaving two free parameters for three labels.
- Reproducibility: Pinned checkpoint revisions and format-matched reruns specify prompt suffixes, split seeds, padding, truncation, and provenance in manifests and revision-summary files.Earlier values were excluded when training and evaluation formatting flags were misaligned.
- Data and tasks: The controlled-logic benchmark is a balanced three-way deductive task using nonce terms, shuffled facts and rules, and distractors requiring multi-hop reasoning.A depth-2 example derives bright through the chain marn → pavo → joto → bright.
- Robustness controls: Generalization tests include in-distribution, deeper-chain, paraphrased-lexical, seed-shift, and counterfactual-inversion splits, with TF-IDF and verbalizer audits checking shallow leakage and answer-length artifacts.Prompt and verbalizer variants show that collapse persists rather than depending on one surface template.
D Expanded Readout Audit
The expanded audit tests whether the readout gap reflects answer-related structure rather than probe artifacts or formatting choices. Across native-readout, prompt, verbalizer, selectivity, stability, and statistical checks, sequence scoring remains the weakest link.
- Native readout: Answer-slot probes remain well above the best label-logit directions even when the best native readout position is selected.The position-matched Base comparison shows that aggregating partially informative label logits over full candidate strings recreates collapse.
- Probe controls: The audit uses random-label and auxiliary-target controls to bound spurious probe decodability and distinguish answer-related structure from arbitrary side-feature memorization.Direction transfer across splits additionally measures probe variance under distribution shift.
- Statistical audit: Bootstrap intervals and paired tests quantify the readout gap, while probe accuracy as a function of labeled training-set size completes the expanded audit.These analyses complement the selectivity and direction-stability controls.
- Prompt and verbalizer robustness: Collapse and label-free recovery persist across 20 prompt/verbalizer variants, while equal-token verbalizers rule out answer-length artifacts.These controls support interpreting the effect as an output-layer bottleneck rather than a formatting artifact.
E Controlled-Logic Parallel Results and Label-Bias Details
Controlled-logic experiments show that Qwen3.5 models recover consistently from collapsed sequence scores, while normalization, permutation, seed, and probe audits distinguish example-specific recovery from calibration artifacts.
- Qwen3.5 models recover consistently across independent generators, arguing against score-normalization artifacts.
- Permutation controls: Corrected predictions exceed count-preserving permutation baselines, supporting per-example recovery beyond label-histogram matching.
- Readout diagnostics: Probe audits quantify gaps among hidden-state probes, native readouts, and full-string scoring, while training-size tests characterize probes as diagnostic upper bounds.
- Sample efficiency: Offset recovery is already positive with 25 unlabeled fit examples and saturates quickly, consistent with a two-parameter threshold shift.
- Normalization robustness: Mean-normalized scores preserve both the candidate-score collapse and the recovery, ruling out token-length normalization as the explanation.
- Shortcut controls: TF-IDF-missed evaluations recompute shortcut-controlled subsets across independent seeds, with positive margins persisting against permutation nulls.
F ProofWriter Sequence Scores and Source-Task Evidence
ProofWriter and broader natural-language diagnostics extend the offset results beyond controlled logic, while hard-slice, permutation, prior, and model-family controls delimit when recovery is informative.
- ProofWriter controls: ProofWriter gains exceed label-count permutation baselines and persist on TF-IDF-missed slices, isolating example-specific structure beyond surface matching.
- ProofWriter results: 0.622 calibrated accuracy is reached by Qwen3.5-4B on ProofWriter with 25 ID examples, compared with 0.321 for Pythia-410M.
- ProofWriter results: 0.643 calibrated accuracy is reached by Qwen3.5-9B on ProofWriter with 25 ID examples, rising to 0.651 with 1000 ID examples.
- ANLI diagnostics: ANLI persistence under permutation baselines and small-budget fits indicates extension to less templated natural-language inference.
- FOLIO diagnostics: FOLIO uses a 203-example fit set, with positive gains retained at fractional subsamples and after label-count, prior, and TF-IDF-missed checks.
- Interpretive boundaries: The auxiliary ablations are diagnostic rather than additional benchmark-performance claims, whereas label-free sequence-score tables evaluate the complete held-out split.
- Model-family boundaries: Positive permutation gaps for larger Qwen3.5, OLMo, and Llama configurations coexist with null or negative Pythia gaps and near-ceiling rows, delimiting a non-universal effect.