PaperlayerView exact revision on arXiv

Source-linked AI summary

Which Source Wins? Task-Dependent Reliance in Vision-Language Models

Rodela Ghosh, Aviral Gupta, Guangjing Wang

arXiv:2608.17205v1cs.CLcs.CV

TL;DR

It is unclear how vision-language models reallocate reliance between conflicting image and text sources when one becomes harder to read. By degrading each modality in controlled conflicts, this paper finds task-dependent reversals: arithmetic models move away more from degraded text, while ChartQA-Conflict models move away more from degraded visual evidence.

  • Problem

    It remains unclear whether VLMs recalibrate reliance between conflicting visual and textual sources when one modality becomes less reliable.

  • Method

    The study degrades either source across four legibility levels in controlled GSM8K, SVAMP, and ChartQA-Conflict conflicts, measuring preference with answers and likelihood margins.

  • Results

    Five of six models shift more strongly from degraded text on arithmetic conflicts, whereas all six likelihood-scored models shift more strongly from degraded visual evidence on ChartQA-Conflict.

  • Takeaways & Limitations

    Modality reliance is setting-dependent rather than a fixed preference for text or vision, varying with task, evidence structure, model, and prompt framing.

  • Takeaways & Limitations

    The arithmetic and ChartQA-Conflict settings do not independently isolate task, representation, and conflict construction, preventing identification of a single causal mechanism.

Abstract

from arXiv · show

Vision-language models (VLMs) combine images and text, but when the two conflict and one becomes harder to read, it is unclear how a model shifts its reliance between them. We study this modality reallocation with a controlled setup: we degrade either the image or the text across four levels of legibility while keeping the other clean, and track how the model's preference changes. We build conflicts from GSM8K and SVAMP by pairing the rendered image of one arithmetic problem with the text of another, so the two sources support different answers. We also introduce ChartQA-Conflict, a manually reviewed benchmark of 229 chart-report conflicts with matched chart and table-image representations. We evaluate six open-weight VLMs using both generated answers and a length-normalized conditional log-likelihood margin. On GSM8K and SVAMP, five of six models shift more strongly away from degraded text than from degraded images. On ChartQA-Conflict, all six likelihood-scored models exhibit the opposite pattern, shifting more strongly away from the degraded visual source. This reversal persists after calibrating for unimodal accuracy loss and after replacing charts with plain table images. Two frontier API models, GPT-5.6-Luna and Gemini-3.5-Flash, behaviorally replicate the ChartQA-Conflict reversal, with GPT-5.6-Luna also matching the arithmetic direction. These results show that modality reliance in VLMs is not fixed, but varies across tasks, evidence structures, models, and evaluation settings. The source code is available at https://github.com/Ro-netizen004/multimodal-arbitration-artifact.

1 Introduction

The introduction asks whether VLMs reallocate reliance toward a cleaner modality when conflicting visual and textual sources differ in legibility. It presents a controlled evaluation showing that this reallocation is task-dependent and reverses between arithmetic and chart-based conflicts.

  • Research question: The motivating gap is whether VLMs appropriately recalibrate reliance when one modality becomes less reliable, and whether the shift is symmetric across degraded images and text.An asymmetric shift would indicate that modality reweighting depends on the degraded modality and task-specific evaluation.
  • Evaluation framework: The evaluation degrades either the image or text across four legibility levels while keeping the other source clean, measuring reliance with generated answers and a length-normalized conditional log-likelihood margin.The framework supports within-item comparisons of shifts away from each degraded modality.
  • Conflict construction: The study constructs conflicts by pairing a rendered GSM8K or SVAMP arithmetic problem with the text of another problem, so the sources support different answers.This controlled setup directly tests modality reallocation under source conflict.
  • Key finding: Five of six models shift more strongly away from degraded text than degraded images on GSM8K and SVAMP, while all six likelihood-scored models reverse this pattern on ChartQA-Conflict.The reversal persists after calibrating for unimodal accuracy loss and replacing charts with table images.
  • Benchmark: ChartQA-Conflict contributes a manually reviewed benchmark of 229 chart-report conflicts with matched chart and table-image representations.The benchmark extends the conflict-based evaluation beyond arithmetic problems.

2 Related Work

Prior work documents modality-use weaknesses and studies conflicting evidence, but does not directly test which source VLMs follow under matched cross-modal conflicts. This study addresses that gap by varying image or text legibility across four levels, using multi-step reasoning tasks and counterbalanced source labels.

  • VLM modality reliance preference: Prior studies report that VLMs often underperform on image-presented problems and under-use visual information, but do not test source preference when image and text conflict.SimpleOCR improves visual reading rather than inducing cross-modal conflict.
  • Methodological distinction: The study varies either image or text legibility across four levels and tracks how preference changes for the same item.Image and text receive nominally aligned degradation levels, while a secondary analysis adjusts for severity using measured unimodal-accuracy drops.
  • Methodological distinction: Counterbalanced Source A/B labels prevent either source from being framed as the primary input.This design separates source preference from arbitrary labeling or presentation asymmetry.
  • Properties of conflicting inputs: Existing conflict studies vary textual degradation, reasoning difficulty, uncertainty, or visual grounding, whereas this work compares matched degradation of image and text.The study examines multi-step reasoning rather than simple object recognition and tracks preference changes for the same item.
  • Methodological distinction: Unlike CMC-Bench, which varies conflict type to evaluate accuracy and abstention, this work keeps each conflict fixed while comparing preference changes across degradation arms.CMC-Bench creates image-text conflicts using ChartQA and MMMU.

3 Methodology

The methodology measures source reallocation by creating conflicts between image and text, degrading one source across matched legibility levels while keeping the other clean. It compares generated source choices and length-normalized CLL preference margins across arithmetic and chart/report settings.

  • Experimental pipeline: Models receive conflicting image and text sources while one modality is degraded across four legibility levels and the other remains clean.The levels are clean (L0), light (L2), moderate (L4), and heavy (L5).
  • Outcome measures: Source preference is measured using both generated answers and a length-normalized conditional log-likelihood margin.Generated responses are classified by whether they follow the image or text; undecidable responses are excluded from source-preference estimates.
  • Conflict construction: Arithmetic conflicts pair one problem’s rendered image with another problem’s native text, ensuring the sources support different answers.Pairs with identical answers are excluded, and source labels are counterbalanced across items.
  • Models: Six open-weight VLMs are evaluated with conditional log-likelihood, with generated-answer evaluation additionally conducted across the model set.The models span four families and range from 2–8B parameters.
  • ChartQA-Conflict: ChartQA-Conflict pairs a native chart with a report supporting a different answer to the same question, then tests a plain-table image control with identical facts.The chart preserves native visual structure, while the control changes only the visual form.
  • Statistical analysis: CLL changes are matched by item ID, summarized by median item-level L0-to-L5 changes, and tested with paired nonparametric, bootstrap, and randomization procedures.The paired-bootstrap confidence interval and randomization test each use 10,000 resamples or draws.

4 Results

Modality reallocation is task-dependent: arithmetic conflicts generally favor moving away from degraded text, whereas ChartQA-Conflict reverses this pattern, with models moving more strongly away from degraded visuals. The reversal survives accuracy calibration and persists with table images and frontier-model behavioral tests.

  • Measure consistency: 75.4% of 26,893 decidable trials matched CLL-margin and generated-answer source preferences, supporting CLL margins as the primary reallocation measure.The 95% CI was 74.4–76.4%, and both measures shifted in the same direction as degradation increased.
  • Arithmetic conflicts: Five of six models showed positive arithmetic asymmetry on GSM8K and SVAMP, indicating stronger reallocation under text than image degradation.Qwen2.5-VL-7B was the exception: its GSM8K estimate was A = +0.059, while its SVAMP asymmetry was A = −0.731 with a 95% CI excluding zero.
  • Robustness checks: Accuracy-loss calibration reproduced the task-dependent pattern: five of six arithmetic models had positive slope differences, whereas all six ChartQA-Conflict models had D < 0.Bootstrap intervals excluded zero in both cases; Qwen2.5-VL-7B remained the arithmetic exception with D < 0 on both arithmetic tasks.
  • ChartQA-Conflict: All six models showed negative ChartQA-Conflict asymmetry across 229 items, moving more strongly away from degraded charts than degraded reports.Values ranged from −0.280 to −2.175 nats/token, with every confidence interval below zero.
  • Robustness checks: Replacing charts with table images preserved negative asymmetry for all six models, showing that graphical chart marks were not necessary for the reversal.The question, report, conflict, and degradation levels were held unchanged.
  • Frontier-model replication: Both frontier APIs reproduced the negative ChartQA-Conflict direction, while GPT-5.6-Luna also showed positive arithmetic asymmetry on role-neutral GSM8K conflicts.Both results had p < .001; the ChartQA evaluation used 229 items and the GSM8K evaluation used 300 conflicts.

5 Conclusion

The conclusion finds that VLMs reallocate reliance asymmetrically when conflicting visual or textual evidence is degraded. This reliance varies with task, evidence structure, model, and prompt framing rather than reflecting a fixed text-or-vision bias.

  • Conclusion: The work contributes a problem formulation, curated conflict datasets, and reproducible evaluation methodology for diagnosing multimodal arbitration.The controlled framework presents conflicting visual and textual evidence, degrades each source in turn, and measures source preference.
  • Conclusion: Five of six models shifted more strongly away from degraded text on arithmetic conflicts across GSM8K and SVAMP.The framework measured source preference using generated answers and conditional log-likelihood margins.
  • Conclusion: All six CLL-scored models showed the opposite modality-reallocation pattern on ChartQA-Conflict.The reversal persisted after calibration for unimodal accuracy loss and under the chart-to-table control.

Limitations

The study’s limitations concern attribution scope, model coverage, confounded task and representation differences, benchmark accessibility, degradation calibration, and the distinction between source arbitration and factual reliability. The CLL margin agrees with generated source choices on 75.4% of attributable trials, making it useful but imperfect.

  • Attribution scope: The conflict construction targets attributable numeric answers and does not transfer directly to multiple-choice outputs, where option labels may obscure source attribution.Extending the framework to free-form, categorical, and multiple-choice outputs requires alternative attribution procedures.
  • Model coverage: The primary graded analysis covers only open-weight 2B–8B-parameter VLMs because CLL requires token-level log-probabilities, limiting generalization beyond evaluated models.The prompt-framing comparison covers five GSM8K models, while legibility-adjusted regression uses six model clusters.
  • Experimental confounds: The arithmetic and ChartQA-Conflict settings jointly vary task, representation, and conflict construction, preventing identification of a single causal mechanism.The chart-versus-table control rules out chart-specific graphical encoding alone as the explanation for the reversal, but does not eliminate remaining task differences.
  • Benchmark accessibility: ChartQA-Conflict uses one natural-visual benchmark reviewed by two non-blind authors, and its report values may be more directly accessible than chart or table values.The reversal therefore compares sources with potentially different baseline directness rather than equally accessible visual and textual evidence.
  • Degradation calibration: Nominally matched corruption levels are not psychometrically matched, and calibrated reallocation slopes rely on approximate unimodal usability estimates from three degraded levels.Calibration against accuracy loss partially addresses the mismatch but does not fully resolve it.
  • Evaluation scope: 75.4% agreement between CLL margins and generated source choices shows that the metric is useful but imperfect, while the framework does not test contradiction detection, uncertainty, or abstention.The setup forces a single answer and reduces legibility without reducing content correctness, measuring source arbitration rather than factual reliability or conflict awareness.

Ethical Considerations

The work presents limited direct ethical risk, but its findings should not be treated as evidence that evaluated models are safe or reliable for deployment. Its degradation methods are simplified proxies whose practical and cross-linguistic generalizability is limited.

  • Interpretation and limitations: The findings do not establish that any evaluated model is safe or reliable for deployment.The experiments assess source reliance under controlled visual and textual degradation rather than deployment safety or reliability.
  • Interpretation and limitations: Gaussian blur and character corruption simplify reduced legibility and omit the full range of accessibility barriers encountered in practice.These manipulations are controlled experimental proxies, not comprehensive representations of real-world accessibility conditions.
  • Interpretation and limitations: The results may not generalize across other languages, scripts, or tokenizations.The passage explicitly identifies these domains as potential limits on generalizability.

Declaration of Generative AI Assistance

The authors used Claude Code and Codex in a limited, supervised capacity for coding support and language editing, while retaining responsibility for the research and paper’s content.

  • Use of Generative AI: Claude Code and Codex supported software engineering and language editing in a limited, supervised capacity.Coding assistance covered debugging, cluster-job orchestration, and figure or LATEX-table formatting; editing covered clarity, prose, cross-references, and consistency.
  • Author Responsibility: The authors conceived, implemented, and verified the research questions, design, datasets, evaluation, analyses, results, and conclusions.
  • Author Responsibility: The authors reviewed all AI-assisted outputs and take full responsibility for the paper’s entire content, including remaining errors.

A Implementation Details · B Degradation Details · C Agreement Between Generated Choice and CLL Margin

The experiments use deterministic multimodal prompting, degradation, generation, and teacher-forced CLL procedures, with separate handling for open-weight and frontier API models. Across GSM8K and SVAMP, CLL agrees with generated source choice on 75.4% of attributable observations but remains a distinct continuous preference measure.

  • A Implementation Details: Open models use automatic device placement, bfloat16 precision, checkpoint-specific processors, and deterministic greedy decoding with task-specific output limits.InternVL2-8B uses a model-specific chat interface and is excluded from CLL scoring because its interface does not support continuation scoring.
  • A Implementation Details: Arithmetic prompts request step-by-step solutions ending in #### <answer>, whereas ChartQA-Conflict requests exactly one answer-only line in that format.CLL reuses the same task prompt and conversation scaffold, appending candidate answers after the literal #### delimiter.
  • A Implementation Details: Frontier API evaluation covers all 229 ChartQA-Conflict items and 300 GSM8K role-neutral conflicts for GPT-5.6-Luna, using generated answers only.Neither API exposes teacher-forced continuation scores, so neither frontier model enters the CLL analysis.
  • A Implementation Details: The CLL arbitration margin subtracts mean candidate log-probabilities, so positive values favor the text-supported answer and negative values favor the image-supported answer.Both candidates are scored under the same multimodal context, and the clean L0 value is shared across the two degradation arms.
  • B Degradation Details: Image degradation applies blur, noise, and contrast changes, while text degradation independently corrupts non-whitespace characters at probabilities 0.08, 0.18, and 0.35 for L2, L4, and L5.L0 leaves the image unchanged; text corruption deletes selected characters with probability 0.5 and otherwise replaces them with random alphanumeric characters, preserving whitespace.
  • C Agreement Between Generated Choice and CLL Margin: Agreement varies across models and is lower overall in the text-degradation arm, indicating that CLL is behaviorally grounded but not interchangeable with generated choice.The analysis therefore reports CLL as a continuous candidate-preference measure and generated behavior separately.
  • C Agreement Between Generated Choice and CLL Margin: 75.4% of observations show agreement between CLL-margin sign and generated source choice after counting the shared clean endpoint once.The total is 20,281/26,893, with an item-clustered bootstrap 95% CI of 74.4%–76.4%.

D Frontier-Model Behavioral Results … I ChartQA-Conflict Construction

Frontier-model behavior reverses across tasks: GPT-5.6-Luna follows the arithmetic pattern but both frontier models favor the clean chart on ChartQA-Conflict. Accuracy calibration, table-image controls, and normalization sensitivity preserve the ChartQA reversal, while the benchmark construction enforces reviewed, attribution-clean conflicts.

  • D Frontier-Model Behavioral Results: Both frontier models show negative ChartQA-Conflict asymmetry, indicating stronger reallocation when the chart is degraded.GPT-5.6-Luna reallocates almost entirely under chart degradation, while Gemini-3.5-Flash shows a smaller but strongly chart-dominant contrast.
  • D Frontier-Model Behavioral Results: Chart-following remains strong under light chart degradation, but moderate or heavy chart degradation shifts preferences sharply toward the report.Report degradation produces little additional movement toward the already-preferred chart.
  • D Frontier-Model Behavioral Results: GPT-5.6-Luna’s arithmetic contrast is RI = 0.323, RT = 0.614, and A = +0.291, with n = 189 and p < .001.The paired-bootstrap 95% CI is [+0.169, +0.407], matching the predominant open-model arithmetic direction and opposing Luna’s ChartQA result.
  • E Legibility-Adjusted Regression Details: Legibility-adjusted regression fits reallocation slopes per unit of proportional unimodal accuracy loss, with D = bT − bI comparing text- and image-degradation arms.Slopes are fitted through the origin over L0, L2, L4, and L5, with matched-item bootstrap intervals.
  • F The Reversal Survives Accuracy Calibration: D = bT − bI remains negative for all six models, with 95% CIs excluding zero, so the ChartQA reversal is not explained by greater chart information loss.All six models exceed the prespecified 0.10 clean-accuracy floor and are calibrated.
  • G Chart-versus-Table Control: Replacing charts with official source tables rendered as plain images preserves the visual information and degradation ladder for all 229 conflicts.The table supports the chart answer exactly, while rows, labels, and values are retained without editing toward the report-supported answer.
  • H CLL Normalization Sensitivity: Changing the candidate-length normalization exponent rescales asymmetry magnitude but preserves its sign and significance in every model–benchmark cell.All entries remain significant at p < 0.05, and estimates move monotonically toward zero as α increases without crossing zero.
  • I ChartQA-Conflict Construction: ChartQA-Conflict retains 230 numeric-answer items after screening 449 questions, constructs report-supported counterfactuals distinct from chart answers, and reviews every item.Two authors independently verified report entailment, answer validity, and aT ≠ aI after normalization; flagged items were adjudicated.

J ChartQA-Conflict Behavioral Results

Generated-answer contrasts on ChartQA-Conflict corroborate the likelihood-based reversal for five models with usable scores, while InternVL2 shows the opposite behavioral direction. These contrasts use only items decidable at all four endpoints, so their sample sizes are smaller than the 229-item CLL sample.

  • Generated-answer results: Five models with usable generated answers and CLL scores corroborate the CLL reversal on ChartQA-Conflict.The generated-answer contrasts are reported in Table 11.
  • Generated-answer results: Generated-answer contrasts retain only items decidable at all four required endpoints for each model, causing model-specific sample sizes below the 229-item CLL sample.The reported values are complete-case mean changes in source-choice indicators, not CLL margins.
  • Generated-answer results: InternVL2 is behavioral-only because its custom chat interface does not expose continuation scoring, yet it shows the opposite behavioral direction.This model is therefore not included among the models with both generated answers and CLL scores.

K Prompt-Framing Results

On matched GSM8K items, the text-versus-image reallocation asymmetry remains positive under both original and neutral Source A/Source B prompts. Neutral framing changes the magnitude in model-dependent ways, while LLaVA-1.6 lacks matched CLL results for comparison.

  • Framing comparison: Positive A indicates stronger reallocation under text degradation than image degradation, comparing original and role-neutral Source A/Source B prompts on the same GSM8K items.The comparison covers five models with complete matched CLL results.
  • Model-dependent effects: The asymmetry remains positive for every model under both prompts, but neutral framing changes its magnitude in model-dependent directions.Neutral framing reduces the effect for Qwen2.5-VL-7B and Idefics3-8B, increases it for Qwen2-VL-2B and LLaVA-OneVision-7B, and produces no detectable change for Phi-3.5-Vision.
  • Evaluation coverage: LLaVA-1.6 is omitted from the CLL comparison because matched original-prompt CLL results are unavailable, although generated-answer results exist under both framings.Table 12 reports paired-bootstrap 95% confidence intervals and a two-sided Wilcoxon signed-rank test; its neutral-minus-original contrast is computed within item.
Loading 2608.17205v1…