PaperlayerView exact revision on arXiv

Source-linked AI summary

When Helpful Text Hurts: Option-Redirecting Bias in Vision-Language Models

Tam Le Thi Thanh, Hoang Tran Van, Hong-Hanh Nguyen-Le, Thanh Duc Ngo

arXiv:2609.32489v1cs.AIcs.CV

TL;DR

Auxiliary text is often used in tri-modal VQA, but its reliability is uncontrolled and its effects on answer selection under fixed contexts are not well isolated. This paper introduces a controlled ladder and an inference-time intervention; it finds that question-aligned, image-inconsistent text supporting a distractor is especially harmful. The findings motivate decision-level analysis of auxiliary-text reliability.

  • Problem

    How unreliable auxiliary text affects answer selection under fixed image-question-option contexts remains unclear, despite its common use and often uncontrolled reliability.

    Source: §1 · p. 1, §2.1 · p. 3

  • Method

    The Textual Reliability Ladder varies auxiliary text along image consistency, question relevance, and option support, alongside a targeted post-hoc inference-time steering intervention.

    Source: §1 · p. 2

  • Results

    Option-Redirecting text induces the largest and most consistent accuracy drops across models and datasets, concentrating errors on specific incorrect options.

    Source: §1 · p. 2, §5.2 · p. 7, §6 · p. 8

  • Takeaways & Limitations

    Tri-modal VQA robustness requires decision-level analysis because auxiliary text can redirect predictions toward a specific distractor.

    Source: §6 · p. 8

    Robustness-oriented steering can mildly degrade performance under Faithful text by attenuating beneficial cues.

    Source: §5.3 · p. 8

Abstract

from arXiv · show

In tri-modal visual question answering (VQA), auxiliary text is commonly used to complement visual and textual inputs, yet its reliability is often uncontrolled. While prior work studies modality conflicts in general, it remains unclear how different types of unreliable auxiliary text affect answer selection under fixed image-question-option contexts. In this work, we show that the most harmful auxiliary text is not necessarily the most factually incorrect, but the one that aligns with the question while contradicting the image and favoring a specific distractor, leading to systematic redirection of model predictions. To isolate this effect, we introduce the Textual Reliability Ladder, a controlled diagnostic protocol that decomposes auxiliary text along three axes: image consistency, question relevance, and option support. Across multiple datasets (ScienceQA, VCR, A-OKVQA, Causal-VidQA) and recent VLMs, we find that such distractor-supporting text induces the largest accuracy drops (up to 53.1%) and concentrates errors on specific incorrect options. To mitigate this failure mode, we propose a training-free inference-time intervention that explicitly counteracts this redirection effect via noise-stability steering and dynamic grounding, reducing redirected errors while largely preserving performance under faithful text. Our results highlight that auxiliary-text reliability must be understood at the decision level, rather than solely through factual correctness, and provide a practical pathway toward more robust tri-modal reasoning.

1 Introduction

Auxiliary text can systematically redirect answer selection when it is question-aligned, image-inconsistent, and supportive of a distractor. The paper introduces a controlled diagnostic framework and a targeted inference-time intervention to study and mitigate this bias.

1 Introduction

  • The central gap is how unreliable auxiliary text changes answer selection when the image, question, and candidate options remain fixed.Prior work treats textual noise or contradiction as a uniform source of error.Source: §1 · p. 1
  • The Textual Reliability Ladder varies auxiliary text while holding the image, question, and options fixed, separating image consistency, question relevance, and option support.This isolates changes in predictions attributable to textual properties.Source: §1 · p. 2
  • Option-Redirecting text produces the largest and most consistent accuracy drops across models and datasets, concentrating errors on a specific incorrect option.This holds under style control and lexical filtering.Source: §1 · p. 2
  • The proposed inference-time intervention uses post-hoc latent-space steering, requires intermediate activations, and needs neither gradients nor parameter updates.It targets Option-Redirecting bias rather than serving as a universal solution.Source: §1 · p. 2

2 Related Work

Prior work examines hallucinations, modality conflicts, language priors, and steering, but does not isolate how externally provided auxiliary text redirects discrete answer choices. This paper addresses that decision-level gap with controlled text variants and a targeted test-time intervention.

2.1 Bias from Auxiliary Text

  • Prior hallucination surveys focus on generated-text faithfulness, whereas this work studies how imperfect input text distorts downstream answer selection [4] [20].Auxiliary inputs in VQA can include captions, OCR/ASR, and metadata [6] [16].Source: §2.1 · p. 2
  • Conflict-focused benchmarks examine contradiction recognition, but do not explain which incorrect answer is selected or isolate properties driving decision-level bias [34] [42].The paper introduces controlled variants factoring image grounding, question alignment, and option-targeted support.Source: §2.1 · pp. 2–3

2.2 Latent-Space Steering and Test-Time Intervention

  • Prior steering and test-time methods target hallucination, grounding, or trusted-prompt performance; this work instead targets auxiliary-text-induced decision bias [31] [37].The study uses frozen VLMs and a test-time intervention to avoid fine-tuning confounds.Source: §2.2 · p. 3

3 Textual Reliability Ladder

The Textual Reliability Ladder controls auxiliary-text properties under a fixed image-question-option context to distinguish coherent reliability regimes and diagnose their effects on discrete predictions. Its construction uses generation and verification controls, while frozen VLM evaluation limits training-time confounds.

3 Textual Reliability Ladder

  • Predictions are compared under fixed image-question-option contexts so changes can be attributed to auxiliary-text properties.Multiple-choice options enable analysis of whether text shifts probability toward the correct answer or a distractor.Source: §3 · p. 3
  • The ladder distinguishes auxiliary-text regimes using image grounding, question relevance, and implicit support for a correct or incorrect option.Table 1 presents regimes spanning reliable, non-informative, and decision-level misleading text.Source: Table 1 · p. 3, §3 · p. 3
  • The ladder is a diagnostic subset of coherent regimes, not an exhaustive combination of all cue values.It excludes grounded, question-aligned text supporting no option and option-supporting text that is not question-aligned.Source: §3 · p. 3
  • Option-Redirecting text is question-aligned and image-inconsistent, with implicit descriptive support for one incorrect option rather than verbatim answer leakage.The target distractor is selected from the incorrect options.Source: §3 · p. 4
  • Option-Redirecting text is token-length matched to Faithful text, while other conditions use fixed prompts, human verification, and lexical and style controls.Faithful annotations are screened, and instances that trigger option-cue or leakage filters are removed rather than edited.Source: §3 · p. 4
  • The ladder separates off-question distraction, non-targeted contradiction, and option-targeted misdirection instead of aggregating them as generic textual corruption.Evaluating frozen VLMs helps attribute behavioral changes primarily to auxiliary text rather than altered supervision or parameter updates.Source: §3 · p. 4

4 Inference-Time Intervention

The intervention addresses global text dominance and instance-specific grounding conflicts in frozen VLMs without retraining. It combines offline Noise-Stability Steering with online, question-aware Dynamic Grounding.

4 Inference-Time Intervention

  • The two-stage intervention pairs a global, static correction with an instance-level, question-aware correction to address distinct sources of auxiliary-text bias.It is designed to mitigate misleading text while preserving performance under non-harmful auxiliary text.Source: §4 · pp. 4–5, Figure 2 · p. 5

4.1 Stage 1: Noise-Stability Steering (NSS) with Offline Pre-computation and Pre-test Incorporation

  • Noise-Stability Steering estimates modality-specific directions from paired demonstrations and applies them to frozen VLM representations using forward hooks.The paired passes hold the context fixed while varying one factor; at each layer, the method subtracts the visual direction and adds the textual direction associated with correct continuations.Source: §4.1 · pp. 5–6

4.2 Stage 2: Dynamic Grounding (DG) with Online Test-Time

  • Dynamic Grounding applies an online correction per test instance, using question-relevant visual evidence to modulate auxiliary-text influence.It selects image tokens most similar to a pooled question representation and uses the final auxiliary-text decoder token; the correction is negligible when text is faithful and stronger when it is visually inconsistent.Source: §4.2 · p. 6

5 Experiments

Experiments show that question-aligned, image-inconsistent auxiliary text supporting a distractor causes the most consistent accuracy drops, while the two-stage intervention concentrates its gains in that regime. The ablations support a complementary coarse-to-fine design, with DG providing the primary correction under Option-Redirecting text.

5.1 Experimental Setup

  • The evaluation uses four multiple-choice VQA benchmarks and frozen VLMs from three model families, with 1,000 valid instances sampled per dataset.A fixed prompt and shared per-instance option permutation are used across auxiliary-text conditions.Source: §5.1 · p. 6

5.2 Textual Reliability Ladder Results

  • Faithful auxiliary text consistently improves accuracy over No-Aux across benchmarks, showing that frozen VLMs can use text beneficially when it aligns with image and question.Larger models improve base accuracy under Faithful and No-Aux conditions, but scaling does not monotonically improve robustness to misleading text.Source: §5.2 · pp. 7–8
  • Option-Redirecting text produces the largest and most consistent accuracy drops across models and datasets, unlike unreliable conditions that typically cause smaller changes.It remains question-relevant, conflicts with the image, and implicitly supports a specific distractor; construction controls include style matching, human verification, and token-length matching to Faithful text.Source: §5.2 · pp. 7–8

5.3 Two-stage Inference-Time Intervention

  • Under Option-Redirecting text, the two-stage intervention improves accuracy and reduces selection of the redirected distractor, while other conditions remain largely unchanged.A few model–dataset pairs show mild negative deltas under Faithful text, reflecting a trade-off in suppressing misleading text cues.Source: §5.3 · p. 8
  • NSS provides global bias correction against textual perturbations, while question-aware DG resolves instance-level grounding conflicts when text is relevant but visually inconsistent.NSS yields modest consistent gains; DG produces the largest improvements under Option-Redirecting text.Source: §5.3 · p. 8
  • The combined NSS+DG method achieves the best overall trade-off between robustness and performance, with gains concentrated under Option-Redirecting text and behavior largely stable elsewhere.DG supplies the primary correction under Option-Redirecting text; NSS contributes conservative global stabilization.Source: Table 2 · p. 8, §5.4 · p. 8, §5.3 · p. 8

5.4 Ablation Study

  • Stage-wise ablation shows NSS offers limited Option-Redirecting recovery, whereas DG supplies most targeted gain with a modest Faithful-text cost; combining them gives the strongest recovery while remaining stable on non-OR conditions.The ablation supports the proposed coarse-to-fine division of labor.Source: §5.4 · p. 8
  • Using the last auxiliary-text token for DG performs consistently better than mean pooling, especially under Option-Redirecting text.Source: §5.4 · p. 8

6 Conclusion

The conclusion identifies Option-Redirecting text as a dominant failure mode in tri-modal multiple-choice VQA and presents the Textual Reliability Ladder as a controlled way to study it. The appendix details how ladder conditions are constructed and verified while holding image, question, and options fixed.

6 Conclusion

  • The paper concludes that unreliable auxiliary text has nonuniform effects, with question-aligned, image-inconsistent text favoring a distractor producing the most severe option-targeted failures.The proposed inference-time intervention mitigates this bias while preserving performance under reliable auxiliary text.Source: §6 · p. 8

A Construction of the Textual Reliability Ladder

  • The Textual Reliability Ladder varies auxiliary text while holding image, question, and answer options fixed, enabling controlled comparisons of text conditions.Unreliable variants are generated with fixed prompts and verified; dataset-provided Faithful text that violates constraints leads to dropping the instance.Source: A Construction of the Textual… · p. 11, §A.7 · p. 12
  • The appendix lists Faithful, Off-Question, Contradictory, Option-Redirecting, Irrelevant, and No-Aux as the ladder conditions.No-Aux contains no auxiliary text.Source: A Construction of the Textual… · p. 11

A.1 Faithful

  • Faithful text comes from dataset annotations and is grounded in the image, relevant to the question, and typically supportive of the correct option.Source: §A.1 · p. 11

A.2 Off-Question; A.3 Contradictory

  • Off-Question text remains visually grounded but is irrelevant to the question, whereas Contradictory text is question-consistent but fabricated without image grounding.Source: §A.2 · p. 11, §A.3 · p. 11

A.4 Option-Redirecting

  • Option-Redirecting text is question-relevant and image-inconsistent, and is crafted to implicitly support a specific incorrect option sampled uniformly from the distractors.Source: §A.4 · p. 11

A.6 Human Verification

  • Two annotators independently verify each constructed text against condition-specific criteria, and ambiguous or noncompliant examples are discarded or regenerated.Both annotators must agree for acceptance; Option-Redirecting is checked for targeted support without verbatim option leakage.Source: §A.6 · p. 12
  • For the key Option-Redirecting versus Faithful comparison, text lengths are matched instance-wise in the target model's token space.Other conditions receive only a coarse length bound.Source: §A.6 · p. 12

A.8 Auxiliary Text Statistics

The appendices document controls for auxiliary-text construction and explain why frozen models are needed to isolate text bias. They also detail the paired-demonstration steering directions and the static NSS and instance-specific DG components.

A.8 Auxiliary Text Statistics

  • Filtering removes answer markers, option copies, leakage, violations of intended ladder semantics, and low-quality generations.The controls include repetitions, metainstructions, and dataset-referential text.Source: §A.8 · p. 14
  • Option-Redirecting text is matched instance-by-instance to Faithful in token length, while other conditions receive only coarse length control.The reported statistics use 1,000 instances per dataset and per-instance token counts under the inference model’s tokenizer.Source: §A.8 · pp. 12–13, Table 9 · p. 15

B Why Fine-tuning Is Not a Fair Test of Textual Bias

  • Frozen-model evaluation avoids conflating changes in text reliability with capability gains, dataset adaptation, prompt learning, or training-dependent heuristics.The authors also note that bluntly distrusting auxiliary text could reduce performance when the text is Faithful.Source: B Why Fine-tuning Is Not a Fair Test… · p. 15

C.1 NSS Steering Vector Construction

  • NSS steering directions come from paired correct and redirected continuations under an otherwise fixed context, with residual differences aggregated and reduced to a dominant principal component.The resulting direction is applied uniformly at inference time.Source: §C.1 · pp. 15–16

C.2 Stage 1: Noise-Stability Steering (NSS)

  • NSS uses fixed, demonstration-estimated directions across inputs; its textual residual contrasts correct and incorrect answer trajectories under the same context.The visual direction captures drift under controlled visual corruption, while the textual direction targets misleading continuations.Source: §C.2 · p. 16

C.3 Stage 2: Dynamic Grounding (DG)

  • DG is instance-specific and question-aware, and the authors use the last auxiliary-text token because it performs better than mean pooling across models and datasets.The last token is closer to answer generation and aggregates preceding auxiliary-text information; the authors suggest mean pooling can dilute decision-relevant content.Source: §C.3 · p. 16

D.1 Experimental Setup

The evaluation uses one prompt template across datasets, models, and text conditions, with an explicit warning that auxiliary text may be unreliable.

D.1 Experimental Setup

  • A fixed prompt template is used across all datasets, models, and textual conditions to avoid prompt-induced confounds.The template cautions that auxiliary text may be unreliable.Source: §D.1 · pp. 16–17
  • The warning is intended to make observed errors reflect internal reliance on textual cues rather than prompt-induced misinstruction.Source: §D.1 · pp. 16–17
  • Figure 6 presents percentage accuracy improvement over base models using Stage 1 NSS across auxiliary-text conditions.Source: Figure 6 · p. 17

D.2 Textual Reliability Ladder Results

On frozen VLMs, accuracy follows a reliability gradient, with Option-Redirecting text producing the most severe degradation. The evaluation excludes additional baselines that would change the controlled comparison or address different bias settings.

D.2 Textual Reliability Ladder Results

  • Model size improves performance in clean settings but does not eliminate susceptibility, which differs across model families.On ScienceQA under Option-Redirecting text, Qwen3VL-8B retains 59.0% accuracy, while InternVL3.5-8B has 35.9%.Source: §D.2 · p. 17
  • Option-Redirecting causes the most severe degradation across models and datasets, typically falling below No-Aux; Table 12 reports accuracy across six text conditions on 1,000 samples per dataset.Faithful text consistently outperforms No-Aux, while Irrelevant text remains close to No-Aux.Source: §D.2 · p. 17, Table 12 · p. 18

D.2.1 Why We Do Not Compare Against Additional Baselines.

  • The authors omit fine-tuning baselines because parameter updates can confound attribution of ladder changes to text reliability rather than capability gains or training heuristics.They also characterize prior dataset-level language-prior methods as structurally different from this externally supplied auxiliary-text setting.Source: §D.2.1 · p. 18
  • Counterfactual calibration and additional prompt baselines are not treated as direct comparisons because they alter the protocol or target different sources of bias.The study instead compares frozen VLMs under a fixed prompt, varying auxiliary-text type and the inference-time intervention.Source: §D.2.1 · pp. 18–19

D.3.1 Only Noise-Stability Steering (NSS) (Stage 1).

NSS produces modest average gains, with its largest reported gains under Option-Redirecting text and little change under Faithful text.

D.3.1 Only Noise-Stability Steering (NSS) (Stage 1).

  • +0.10 accuracy points is the average NSS improvement across models, datasets, and ladder conditions, while Faithful accuracy changes by an average of +0.11%.The authors report macro-averaged gains and say NSS rarely causes large regressions.Source: §D.3.1 · p. 19
  • +0.24% is the largest average NSS gain across text conditions, occurring under Option-Redirecting text.For Qwen2.5VL-3B on A-OKVQA, NSS changes Option-Redirecting accuracy from 67.4% to 69.2% (+1.8%).Source: §D.3.1 · p. 19
  • +0.60% is the reported Option-Redirecting average gain for NSS on A-OKVQA, the dataset where the pattern is most pronounced.Source: §D.3.1 · p. 20

D.3.2 Only Dynamic Grounding (DG) (Stage 2).

Dynamic Grounding (DG) applies an instance-dependent correction to auxiliary text and yields its largest gains when text redirects the model toward an incorrect option. Its effects are generally small when text is uninformative, but can be negative when text is faithful.

D.3.2 Only Dynamic Grounding (DG) (Stage 2).

  • DG uses the current image–question context to adjust auxiliary-text incorporation, aiming to suppress text-driven shortcuts when the text misleads.Unlike NSS, DG is instance-dependent; the two-stage design pairs DG with NSS as a conservative stabilization backbone.Source: §D.3.2 · pp. 20–21
  • ScienceQA gains reach +14.0% for Qwen2.5VL-7B and +15.1% for InternVL3.5-4B under Option-Redirecting text; DG also improves A-OKVQA and VCR.The passage reports additional gains of +7.7% on A-OKVQA for Qwen2.5VL-3B and +8.0% on VCR for Qwen2.5VL-7B.Source: §D.3.2 · p. 20
  • Under No-Aux, Irrelevant, and Off-Question conditions, changes are generally near zero, with benefits emerging mainly when strong text-induced bias needs correction.The cited account says DG does not introduce large distributional shifts when auxiliary text is uninformative.Source: §D.3.2 · p. 20
  • Faithful text can reduce DG accuracy, including by up to −3.5% on VCR for Qwen2.5VL-3B, so DG alone is not a universally safe default.The passage attributes this trade-off to suppressing text influence even when auxiliary text is correct and helpful.Source: §D.3.2 · p. 20

D.3.3 Combine Two-stage.

The combined intervention pairs global stabilization with instance-dependent grounding to target Option-Redirecting text while limiting shifts in other conditions. It can still modestly reduce performance under some Faithful-text settings, and its redirected-error reductions are reported across datasets and models.

D.3.3 Combine Two-stage.

  • Option-Redirecting gains reach +14.6% points on ScienceQA for InternVL3.5-4B and +14.3% on ScienceQA for Qwen2.5VL-7B with the combined method.Reported gains also reach +8.6% on A-OKVQA, +8.1% on VCR, and +6.6% on Causal-VidQA.Source: §D.3.3 · p. 21
  • Faithful-text performance is typically preserved, but some model–dataset pairs show negative deltas, reflecting the risk of downweighting helpful text.The authors describe these drops as limited in scope and report that the same hyperparameters yield gains in harmful regimes.Source: §D.3.3 · p. 21
  • Figure 8 contrasts coherent, global NSS directions with more localized, layer-dependent DG adjustments.The caption describes layer-wise magnitudes and cross-layer similarities for the stage-wise ablations and combined method.Source: Figure 8 · p. 22
  • Outside Option-Redirecting, changes typically stay within about ±1% for No-Aux, Irrelevant, Off-Question, and Contradictory conditions.Under Off-Question and No-Aux, the average effect is near zero.Source: §D.3.3 · p. 23
  • NSS stabilizes representations globally, while DG supplies instance-dependent re-grounding; severe option-targeted collapse can remain beyond NSS alone.This motivates DG’s question-aware correction, which selectively realigns auxiliary-text representations with question-relevant visual evidence.Source: §D.3.3 · pp. 21, 23

D.3.4 Option-Redirecting (Worst-Case Text Bias).

  • Redirect is the percentage of 1,000 samples predicted as the specific incorrect option favored by each Option-Redirecting text.Lower Redirect values indicate stronger robustness to option-targeted textual bias.Source: §D.3.4 · p. 23
  • Across models and datasets, the combined method consistently reduces redirected predictions, with reduction magnitudes varying by dataset and model scale.The reported reductions are typically larger on ScienceQA and A-OKVQA than on Causal-VidQA.Source: §D.3.4 · p. 23

D.3.5 Why Combining NSS and DG Is Necessary.

NSS and DG have distinct roles: NSS supplies global stability, while DG provides targeted, instance-level correction under Option-Redirecting text. Qualitative examples show both recovered visually grounded choices and residual failures when the redirected signal remains strong.

D.3.5 Why Combining NSS and DG Is Necessary.

  • NSS yields modest, consistent gains across text conditions, while DG’s strong improvements concentrate under Option-Redirecting text and are negligible for Faithful or weakly informative text.The stages therefore contribute global stability and selective instance-level correction, respectively.Source: §D.3.5 · p. 23
  • NSS stabilizes representations globally but is option-agnostic, whereas DG corrects decision-level option bias through instance-level grounding; neither stage is interchangeable.The combined method retains NSS’s stability while enabling DG’s targeted correction under Option-Redirecting text.Source: §D.3.5 · p. 23
  • Figure 8 shows near-identical NSS steering norms in the Only NSS and Combined settings, while DG adjustments remain sample-dependent and grow with depth.Its similarity maps show coherent NSS directions across layers and more localized DG adjustments.Source: §D.3.5 · p. 23

D.4 Qualitative Examples under Option-Redirecting

  • Qualitative cases include successful corrections that shift probability toward the visually grounded answer and failures where the redirected distractor remains dominant or competitive.The examples connect instance-level probability shifts to the aggregate robustness gains while showing that strong option-targeted text can resist correction.Source: §D.4 · p. 25, Figure 13 · p. 25, Figure 14 · p. 25
Loading 2609.32489v1…