PaperlayerView exact revision on arXiv

Source-linked AI summary

Reasoning Instructions Can Break Answer Decoding in Vision--Language Models

Zeyan Li, Siyuan Qiu, Jianfeng Xu

arXiv:2609.29278v1cs.CLcs.AI

TL;DR

CoT prompting is widely used in multimodal reasoning, but the specific mismatch of requesting a continuation and immediately scoring answer labels before any rationale has not been systematically examined. The paper compares direct scoring, CoT-prefix label scoring, and free reasoning, alongside condition-matched probes and controls including option-content permutation. Under CoT-prefix scoring, native accuracy fell from 80.76% to 45.48%; a condition-matched probe reached 78.94%, and the CoT-prefix A/first-slot rate across permutations was 93.54 ± 0.43%.

  • Problem

    CoT prompting is widely used in multimodal reasoning, but the specific mismatch of requesting a continuation and immediately scoring answer labels before any rationale has not been systematically examined.

    Source: §1 · p. 1

  • Method

    The paper compares direct scoring, CoT-prefix label scoring, and free reasoning, alongside condition-matched probes and controls including option-content permutation.

    Source: §1 · p. 1

  • Results

    Under CoT-prefix scoring, native accuracy fell from 80.76% to 45.48%; a condition-matched probe reached 78.94%, and the CoT-prefix A/first-slot rate across permutations was 93.54 ± 0.43%.

    Source: §3 · p. 2

  • Takeaways & Limitations

    When a prompt requests a rationale, the scoring rule should evaluate the corresponding output event rather than immediate answer-label logits.

    Source: §1 · p. 1, §5 · p. 4

    The study covers a limited set of multiple-choice benchmarks, two VLM families, reasoning instructions, and answer interfaces; probes show decodability, not causal use by the native decoder.

    Source: §5 · p. 4

Abstract

from arXiv · show

Chain-of-thought (CoT) instructions can distort multiple-choice VLM evaluation when a scorer appends a reasoning cue but reads answer-label logits before the model generates any rationale. We call this CoT-prefix scoring. On ScienceQA, Qwen2.5-VL-7B drops from 80.76% to 45.48%, and across five option-content permutations 93.54% of CoT-prefix predictions select the first slot. Condition-matched linear probes recover 78.94% from the same hidden states, while free generation restores 75.24%, showing that the answer often survives the prefix and the immediate readout fails. Vocabulary and layer diagnostics explain the mismatch: probability mass moves toward continuation tokens, while answer information remains linearly accessible in late layers. The effect recurs with varying severity across datasets and models, though not universally. These results show that CoT-prefix scoring can confound model knowledge with an evaluation-interface mismatch and should be avoided unless the requested and scored output events are aligned.

1. INTRODUCTION

CoT-prefix scoring appends a reasoning cue but reads answer-label logits before any rationale is generated, creating a mismatch between the requested and scored events. The paper investigates whether the resulting low scores reflect lost answers or a failure of immediate readout.

1. INTRODUCTION

  • CoT-prefix scoring appends an instruction such as “Let me think step by step.” and scores answer-label logits before the model generates an explanation.The prompt requests an explanation, while the scorer treats the first token as the final answer.Source: §1 · p. 1
  • Across five option-content permutations, most CoT-prefix predictions select the first option regardless of its content.The suffix changes while the questions remain unchanged.Source: §1 · p. 1
  • Earlier work examined selection bias, position effects, and prompt-format flaws, but did not isolate immediate label scoring after a continuation-inducing suffix.The paper uses matched probes and controls for content permutation, free reasoning, alternative output events, and replication across datasets and model families.Source: §1 · p. 1
  • Figure 1 contrasts direct prompting, which requests and scores B, with a CoT prefix, which requests a rationale but can yield A under immediate label scoring.Free generation and a matched probe can recover B.Source: Figure 1 · p. 1
  • Matched probes recover most lost points from the same hidden states, and free generation restores most direct-answer performance, supporting readout failure rather than answer loss.The effect also recurs, sometimes more severely, on other datasets and models, while one counterexample shows the suffix can help.Source: §1 · p. 1
  • The paper traces the failure to probability mass shifting toward continuation tokens and recommends aligning the requested output event with the scored event.Its contributions also identify CoT-prefix scoring and use probes and generation controls to test whether the answer survives the prefix.Source: §1 · p. 1

2. EVALUATION INTERFACES AND DIAGNOSTIC PROTOCOL

The diagnostic protocol compares immediate answer-label scoring with free reasoning and uses condition-matched probes to test whether answers remain accessible in the final hidden state. It also tests the stability and limits of the first-slot pattern through content shuffling.

2. EVALUATION INTERFACES AND DIAGNOSTIC PROTOCOL

  • Direct scoring reads answer-token logits after the choices, CoT-prefix scoring reads them immediately after a reasoning cue, and free CoT extracts an answer after generation.Direct and CoT-prefix differ only in terminal text; free CoT also changes the output event.Source: §2 · p. 2
  • The formalization uses the complete condition-specific prompt and the hidden state at its final input position to describe the model's next-token distribution.The valid answer-label token set defines immediate scoring.Source: §2 · p. 2
  • Immediate label scoring and free reasoning query different conditional events: the next-position label distribution versus generating a rationale and then an answer.CoT-prefix scoring applies immediate label scoring even though the prompt requests the rationale-then-answer event.Source: §2 · p. 2
  • Condition-specific linear probes map each final-layer, last-input-position state to five choice logits while the VLM remains frozen.Invalid choices are masked, and the condition gap measures condition-specific recovery; probe decodability does not establish what the native decoder uses.Source: §2 · p. 2
  • Probe fitting uses the image-present official ScienceQA training split, validation selects checkpoints, and reported results use the complete official test split.The test split is not used to fit or select the primary decoder.Source: §2 · p. 2
  • Five fixed-seed permutations shuffle option contents and remap the gold label while regenerating labels in display order, leaving A in the first slot.A high coupled A/first-slot rate indicates a stable default toward that label-position event, but this control alone cannot distinguish position bias from label-token bias.Source: §2 · p. 2

3. RESULTS

On ScienceQA, CoT-prefix scoring sharply lowers native accuracy, while probes and generated reasoning recover much of the apparent loss. Controls and cross-dataset/model checks show that the effect depends on the scored output event and varies across settings.

3. RESULTS

  • Native accuracy falls from 80.76% Direct to 45.48% CoT-prefix; the CoT-prefix probe reaches 78.94%, and 93.54 ± 0.43% of five-permutation predictions select the first slot.Across the same permutations, Direct accuracy is 80.26 ± 0.13% and CoT-prefix accuracy is 45.78 ± 0.66%.Source: §3 · p. 2
  • Free-CoT extraction restores accuracy to 75.24% on all 4,241 items, with 215 parse failures counted as errors.The A/first-slot rate falls from 94.08% to 42.09%.Source: §3 · p. 2
  • Rank-32 LoRA on the vocabulary projection raises CoT-prefix accuracy from 45.48% to 70.90%, while Direct stays at 80.76% to 80.60%; the collapse is severe but not universal.The cross-dataset/model checks include a 54.02-point collapse on AI2D and a reversal on MMBench.Source: Table 1 · p. 3, §3 · p. 3
  • Full-string likelihood reverses the ordering (52.32% Direct, 55.41% CoT-prefix), while eleven endings yield 76.75%–87.46% generated-answer accuracy.Answer-eliciting endings perform well under immediate scoring, unlike continuation-inducing phrases.Source: §3 · p. 3
  • For Q3 ScienceQA image items, the CoT-prefix accuracy gap narrows from 26.9 points with two choices to 14.2 with four; the five-choice slice has only 38 items.The coupled first-slot default weakens as more choices are displayed.Source: §3 · p. 3

4. MECHANISTIC AND EXPLORATORY DIAGNOSTICS

The diagnostics show that CoT-prefix both shifts probability toward continuation tokens and distorts the distribution among answer labels. Answer information remains linearly accessible in late-layer states even as the immediate readout diverges.

4. MECHANISTIC AND EXPLORATORY DIAGNOSTICS

  • Correct-label probability is preserved mainly when the gold answer occupies the first slot, while later positions are strongly suppressed.The position-conditioned comparison groups items by the displayed position of the gold label.Source: Figure 2 · p. 3, §4 · pp. 3–4
  • For one representative item, Direct assigns 0.904 probability to correct option C, but CoT-prefix assigns 0.980 to first-slot option A; free reasoning and the probe recover C.The probe assigns C probability 0.9996.Source: §4 · p. 3
  • On the full test split, four explicit step-by-step requests cause large drops and A/first-slot concentration, while the minimal cue “Reasoning:” remains close to Direct, indicating an effect of continuation strength.Source: §4 · p. 3
  • Probe accuracy rises in late layers, where the Direct–CoT-prefix gap widens, while answer information remains linearly organized.The shuffle control drops linear accuracy from 86.25% to 33.75%, near the 32.65% random-choice baseline, indicating recovery depends on example-specific state information.Source: Figure 3 · p. 4, §4 · p. 4
  • MA falls about fourteen-fold and MR rises four-fold, while the mean rank of the correct answer token worsens by more than twenty thousand positions.Separately, the suffix changes the conditional distribution within valid answer labels, so renormalizing over those labels does not recover the original decision.Source: §4 · p. 4, Table 3 · p. 4

5. CONCLUSION

CoT-prefix scoring can misread a continuation-seeking prompt: answer identity remains linearly organized in late layers even as probability shifts toward continuation tokens. The study recommends aligning the scored output event with the requested one, while noting limits in its benchmarks, model families, and evaluation settings.

5. CONCLUSION

  • More than 35 points: CoT-prefix scoring lowers Qwen2.5-VL-7B accuracy on ScienceQA and drives predictions toward the first slot regardless of its content.Condition-matched probes recover most of the answer from the same hidden states, while free generation restores most direct-answer performance; diagnostics locate the mismatch in probability mass shifting toward continuation tokens despite linearly organized answer identity in late layers.Source: §5 · p. 4
  • When a prompt requests a rationale, evaluations should score the corresponding output event rather than immediate post-prefix label logits.The paper suggests reporting the interface, label tokenization, and whether decoding is conditioned on an intermediate rationale.Source: §5 · p. 4
  • The study covers multiple-choice benchmarks and two VLM families; its probes establish decodability, not causal use by the native decoder.It also studies a limited set of reasoning instructions and answer interfaces.Source: §5 · p. 4
Loading 2609.29278v1…