PaperlayerView exact revision on arXiv

Source-linked AI summary

VisualTrust: Video Reasoning under On-Screen Claim Framing

Huachao Zhu, Junjielong Xu, Sicheng Li, Liang Yang, Mingtao Chen, Zhichao Hu, Li Jiang, Pinjia He

arXiv:2610.21159v1cs.CV

TL;DR

Existing benchmarks test video understanding or on-screen text reading without distinguishing a rendered assertion from visual evidence, leaving claim deference unmeasured. VisualTrust varies a rendered claim while holding video, question, and options fixed; DEAR checks for conflicts, selectively blinds evidence, and gives a fresh-context reader the resulting media. Mean accuracy was 90.1% with no claim and 51.9% with a misleading claim.

  • Problem

    Existing benchmarks test video understanding or on-screen text reading without distinguishing a rendered assertion from visual evidence, leaving claim deference unmeasured.

    Source: §1 · p. 2

  • Method

    VisualTrust varies a rendered claim while holding video, question, and options fixed; DEAR checks for conflicts, selectively blinds evidence, and gives a fresh-context reader the resulting media.

    Source: §1 · p. 3, §6.1 · p. 9

  • Results

    Mean accuracy was 90.1% with no claim and 51.9% with a misleading claim.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Visual accuracy alone does not ensure resistance to claims: models can abandon a correct answer for the asserted alternative, so resistance should be evaluated separately.

    Source: §1 · p. 3, §7 · p. 10

    DEAR was evaluated on VisualTrust with three proprietary models and one question per source video; its effectiveness on other benchmarks and open-weight models remains untested.

    Source: H LIMITATIONS · p. 25

Abstract

from arXiv · show

On-screen text can describe what a video shows, or it can frame what the video is supposed to mean. Current benchmarks treat both alike, testing whether a model can read the text rather than whether it should trust it. We study claim deference, multimodal large language models' tendency to let an on-screen claim override conflicting visual evidence. We introduce VisualTrust, a counterfactual benchmark with 5,063 cases across 14 claim-fact classes. Each case pairs a question with three versions of the same video: one with no claim, one with an accurate claim, and one with a misleading claim that alters a single grounded fact. The option the misleading claim supports is the trap answer. Across seven open-weight and four proprietary models, mean accuracy falls from 90.1% without a claim to 51.9% with a misleading claim. Among initially correct responses, 42.8% flip to the trap answer and only 0.6% become other errors. Strong no-claim accuracy does not ensure resistance. We also introduce DEAR (Decoupled Evidence Arbitration and Reading), a training-free agent for video question answering that separates evidence arbitration from reading. An independent check reports conflicting evidence, a selector weighs it without a fixed modality priority and blinds what it withholds, and the reader answers from the resulting video in a fresh context. On 500 videos across three proprietary models, DEAR raises accuracy under misleading claims from 47.2--64.0% to 81.8--85.2% and reduces trap-answer selection from 34.4--50.8% to 11.6--15.0%, while preserving no-claim accuracy.

1 INTRODUCTION

The introduction identifies a gap between reading on-screen text and deciding whether to trust it, then presents VisualTrust and DEAR to study and address claim deference. Across tested models, misleading claims redirect many initially correct answers toward the claim-supported trap option.

1 INTRODUCTION

  • Existing benchmarks test video understanding or on-screen text reading, but do not distinguish a recorded sign from an overlaid assertion about the scene.Unsupported claims can consequently become unsupported model outputs in moderation, retrieval, and summarization.Source: §1 · pp. 1–2
  • Claim deference is distinct from object hallucination and sycophancy because the misleading signal is rendered in the visual input that models must ground against the recorded scene.The introduction notes that models may cite the claim as evidence and explain away conflicting visual evidence.Source: §1 · pp. 2–3
  • 90.1% mean accuracy without a claim fell to 51.9% with a misleading claim; 42.8% of initially correct responses flipped to the trap answer, versus 0.6% becoming other errors.The pattern held across all 14 fact classes, and strong no-claim performance did not ensure resistance.Source: §1 · pp. 2–3
  • DEAR separates evidence arbitration from reading: an independent checker reports conflicts, a selector weighs evidence without fixed modality priority and withholds selected content, and a reader answers in a fresh context.Across 500 videos and three proprietary models, misleading-claim accuracy rose from 47.2–64.0% to 81.8–85.2%, while no-claim accuracy was preserved.Source: §1 · p. 3
  • VisualTrust tests claim deference with counterfactual cases that vary the rendered claim while holding the video, question, and options fixed.Its 5,063 cases span 14 claim-fact classes, and misleading claims alter one visually grounded fact.Source: §1 · pp. 2–3

2 RELATED WORK

Related work evaluates video understanding and text reading, studies text–vision conflicts and conversational claim pressure, and distinguishes those questions from claim verification. VisualTrust measures whether an in-video claim redirects responses away from visual evidence.

2.1 READING TEXT IN VIDEO

  • Video-MME and LongVideoBench assess video understanding, while BOVText and MME-VideoOCR evaluate text recovery and text-based reasoning, not whether accepting an overlay overrides visual evidence.The cited works are Video-MME (Fu et al., 2025), LongVideoBench, BOVText, and MME-VideoOCR (Shi et al., 2025).Source: §2.1 · p. 3

2.2 TEXT–VISION CONFLICT AND CLAIM DEFERENCE

  • Prior studies examine text overriding visual evidence, often in images, or misleading claims through conversational pressure; VisualTrust instead tests rendered video claims without asking models to endorse them.VisualTextTrap also uses controlled overlays but adds a training-time mitigation, whereas VisualTrust adds matched single-point claim pairs and answer-source screening.Source: §2.2 · pp. 3–4

2.3 CLAIM VERIFICATION AND MISINFORMATION

  • Fact-verification work predicts whether content is supported, while DEAR answers the original question and intervenes by changing which evidence remains available during response generation.Chain-of-Verification revises a response using independent checks, and VideoAgent gathers question-relevant evidence through visual tools.Source: §2.3 · p. 4

3 FORMULATING CLAIM DEFERENCE

VisualTrust frames video reasoning as deciding whether to trust recorded visual evidence or a conflicting on-screen claim. Its matched counterfactual setup varies the claim channel while holding the underlying video and question fixed, then measures accuracy and claim-directed flips.

3 FORMULATING CLAIM DEFERENCE

  • Grounded answers to questions about the recorded scene should follow visual evidence rather than a conflicting rendered claim.The paper calls targeted reliance on the rendered claim claim deference.Source: §3 · p. 4

3.1 TASK SETUP

  • The counterfactual design holds each case’s video, question, ordered options, and non-target content fixed while varying only the claim channel.This setup isolates the effect of claim framing within a case.Source: §3.1 · p. 4
  • Cases use no-claim, accurate-claim, and misleading-claim conditions; the latter changes one visually grounded fact while other claim presentation remains unchanged.No-claim retains scene text and other non-target text; accurate claims agree with visual evidence, while misleading claims conflict on the pivoted fact.Source: §3.1 · p. 4
  • Each question targets the video’s pivoted fact, and its correct answer and misleading-claim-supported trap answer are designated from the visual evidence.The question is about the pivoted fact, not what the rendered claim says.Source: §3.1 · p. 4
  • Each case is evaluated across conditions with the same question and ordered answer options.The case contributes a matched triple of model responses.Source: §3.1 · p. 4
  • No-claim comparisons measure the effect of adding an overlay, while accurate-versus-misleading comparisons isolate the claim-content contrast under matched presentation.The two contrasts serve different evaluation purposes.Source: §3.1 · p. 4

3.2 EVALUATION METRICS

  • Accuracy reports correctness separately for each claim condition, and metrics are computed per model over complete three-condition cases.The section defines accuracy as correctness under condition c.Source: §3.2 · pp. 4–5
  • Trap-flip rate measures no-claim-correct responses that switch to the trap answer under a misleading claim, separating these reversals from pre-existing errors.Transitions to other errors are also reported to assess whether new errors concentrate on the trap answer.Source: §3.2 · p. 5

4 VISUALTRUST: CONTROLLED CLAIM INTERVENTIONS

VisualTrust pairs claims over shared video bases to create controlled three-condition cases. Its construction fixes answers before claim writing, screens questions for source ambiguity, and validates that the rendered claims preserve access to visual evidence.

4 VISUALTRUST: CONTROLLED CLAIM INTERVENTIONS

  • VisualTrust contains 5,063 matched three-condition cases from 2,811 source videos across five sources with varied claim provenance, evidence spread, and annotation density.The sources include native creator titles, long-form videos, and short clips with annotations covering actions, objects, relations, and on-screen text.Source: §4 · p. 5

4.1 CLAIM-BLINDED BASES

  • Each video uses one claim-blinded base shared by the no-claim condition and the two rendered-claim conditions.When original creator claims are present, they are erased with UnderEraser and repaired clips are manually reviewed.Source: §4.1 · p. 5

4.2 CASE AND QUESTION CONSTRUCTION

  • Case construction fixes the visually grounded correct answer and conflicting trap answer before deriving the misleading claim by a single-point edit.Questions and options are specified first.Source: §4.2 · p. 6
  • Questions are retained or revised to target the visual referent, excluding unresolved cases where the wording could license the rendered claim as an answer source.All questions are multiple choice, and the final wording and options remain fixed across all three conditions.Source: §4.2 · p. 6

4.3 CLAIM-FACT CLASSES AND VALIDATION

  • The 14 claim-fact classes group cases by the visually verifiable proposition changed, from local text or readouts to whole-video propositions.The classes support tests across different fact types.Source: §4.3 · p. 6
  • Released cases are checked to ensure the pivoted fact holds, paired claims differ only on that fact, and misleading text reads naturally and conflicts with the video.Question screening and claim validation are detailed in Appendix A.Source: §4.3 · p. 6

4.4 RENDERED CLAIM CONDITIONS

  • Accurate and misleading claim cards share a design and exposure timing, differing only in the pivoted fact string.Cards are aligned with evidence that settles the claim, and released cards do not occlude evidence or non-claim text needed to answer.Source: §4.4 · p. 6

5 MEASURING CLAIM DEFERENCE

Section 5 tests whether misleading on-screen claims redirect video models’ answers, whether baseline visual accuracy predicts resistance, and how models justify flipped responses.

5.1 EXPERIMENTAL SETUP

  • The evaluation covers eleven video MLLMs under all three claim conditions, with each model using its officially recommended video preprocessing.Complete three-condition cases are scored; ambiguous responses count as incorrect but not as trap selections.Source: §5.1 · pp. 6–7, Table 1 · p. 7

5.2 MISLEADING CLAIMS REDIRECT CORRECT RESPONSES

  • Mean accuracy falls from 90.1% with no claim to 51.9% with a misleading claim across eleven video models.Figure 3 compares accuracy across the three claim conditions.Source: §5.2 · p. 7, Figure 3 · p. 7
  • Among no-claim-correct responses, 42.8% flip to the trap answer under misleading claims, while only 0.6% become other errors.Figure 3b shows the destinations of responses that were correct without a claim.Source: §5.2 · p. 7, Figure 3 · p. 7
  • Accurate claims raise mean accuracy to 98.5%, contrasting with the effect of matched misleading claims.The accurate and misleading claims are matched in presentation and exposure, differing only in the fact they assert.Source: §5.2 · p. 7

5.3 VISUAL ACCURACY DOES NOT ENSURE RESISTANCE TO MISLEADING CLAIMS

  • Seed 2.1 Pro and Gemini-3.5-Flash differ by 0.9 percentage points in no-claim accuracy but by 20.3 points in trap-flip rate.Across the five Qwen3.5 variants, no-claim accuracy spans 89.1–93.1% while trap-flip rates remain within 44.3–47.7%.Source: §5.3 · p. 8

5.4 CLAIM USE IN REASONING TRACES

  • In 600 traces from three models, 75–100% of flip traces cite the rendered claim as evidence, compared with 4–5% of still-correct traces.The authors call this stated use of a claim as a premise claim anchoring.Source: §5.4 · p. 8
  • Among flip traces acknowledging conflicting evidence, 86% explain the discrepancy away and continue relying on the claim.The authors note that this pattern motivates DEAR’s separation of evidence arbitration from reading.Source: §5.4 · p. 8

6 MITIGATING CLAIM DEFERENCE

Section 6 presents DEAR, which separates evidence arbitration from reading, and evaluates it against prompt-level baselines on VisualTrust.

6.1 DECOUPLED EVIDENCE ARBITRATION AND READING

  • An independent subagent checks the full video and question for conflicting accounts without seeing answer options or a prior response.If no inconsistency is reported, the original video goes directly to the reader.Source: §6.1 · pp. 8–9
  • When accounts disagree, a selector assesses relevance and credibility without fixed modality priority, withholds selected media, and gives the reader a fresh context.The reader does not receive the conflict report, selection rationale, or earlier responses.Source: §6.1 · p. 9

6.2 EVALUATION ON VISUALTRUST

  • The evaluation compares four methods on 500 source videos across three backbones, measuring accuracy by claim condition and trap-answer selection under misleading claims.DEAR uses two model calls per question without intervention and three with intervention.Source: §6.2 · p. 9, Table 2 · p. 9
  • DEAR raises misleading-claim accuracy to 81.8–85.2% from 47.2–64.0% for benchmark-prompt Direct, while reducing trap-answer selection to 11.6–15.0% from 34.4–50.8%.The strongest prompt-level baseline reaches at most 70.0% accuracy; no-claim accuracy rises for every backbone, and no condition falls below benchmark-prompt Direct.Source: §6.2 · p. 9, Table 2 · p. 9

7 CONCLUSION

The conclusion emphasizes evaluating resistance to on-screen framing separately from visual accuracy, while preserving the value of on-screen language and grounding answers in evidence appropriate to the question.

7 CONCLUSION

  • VisualTrust shows that models can recognize a fact yet abandon it when an on-screen claim asserts otherwise, concentrating errors on the asserted alternative.The authors conclude that resistance to such framing should be evaluated separately from visual accuracy.Source: §7 · p. 10

A CASE CONSTRUCTION AND SCREENING

  • Question screening rewrites cases where a rendered claim could be a legitimate answer source and excludes cases that remain ambiguous.Retained questions must exclude the rendered claim as an answer source through their wording or video context.Source: A CASE CONSTRUCTION AND SCREENING · p. 13

B.1 DEFINITIONS AND COMPOSITION

  • Each case is labeled for the proposition changed by the misleading claim, rather than for the string-edit operation.Table 3 gives class definitions, examples, and case counts.Source: §B.1 · p. 13

B.2 CLASS-WISE RESULTS

Trap-answer flips occur across every claim-fact class, with the highest rates in questions requiring reasoning beyond literal reading. The evaluation also specifies matched conditions, response scoring, paired statistics, and a trace audit that distinguishes stated justifications from internal mechanisms.

B.2 CLASS-WISE RESULTS

  • 72.1% of initially correct responses flip to the trap answer in EVT, versus 7.7% in GLO; accurate-claim flip rates remain below 1.1%.Flips occur in all 14 classes; event, comparison, and causal questions have the highest rates, while whole-video propositions have lower rates than text and numeric readouts.Source: §B.2 · p. 13, Figure 5 · p. 14, Table 4 · p. 14

C RENDERING THE MATCHED CONDITIONS

  • Accurate and misleading claims share presentation and exposure, differing only in the fact asserted.The claim conditions use shared layout, typography, placement, and timing, with line reflow allowed for different string lengths; reviewers check that overlays do not obscure relevant evidence.Source: C RENDERING THE MATCHED CONDITIONS · p. 14

D.1 MODEL INPUTS AND GENERATION

  • Models receive the condition video, question, and ordered options, without condition labels, answer keys, or construction annotations.Thinking is enabled, and video preprocessing follows each model’s officially recommended configuration.Source: §D.1 · pp. 14–15, Table 5 · p. 15

D.2 BLIND RESPONSE-TO-OPTION MAPPING

  • A blind mapper assigns each free-form response to an option using only the question, options, and response text; ambiguous mappings count as incorrect, not trap selections.The completed scoring contains 166,780 responses, including 126 ambiguous responses.Source: §D.2 · p. 15

D.3 PAIRED SUMMARIES

  • The equal-model mean net increase in trap-answer selection is positive in all 14 classes, ranging from 7.8 percentage points for GLO to 57.5 for EVT.Each model–class combination has at least 93 no-claim-correct cases in the trap-flip denominator; the increase is also positive for each model within each class.Source: §D.3 · p. 15

E REASONING-TRACE AUDIT

E.1 ILLUSTRATIVE EXCERPTS

Illustrative traces show models acknowledging visual–claim conflicts yet retaining the claim as grounds for their answer. DEAR separates conflict reporting and evidence selection from a reader isolated from the report and selection rationale.

E.1 ILLUSTRATIVE EXCERPTS

  • In one trace, Qwen3.5-Plus reconciles a 07:04 overlay with a 07:05 alarm display by reasoning that the alarm could still be ringing after it started.The trace concludes that the set time is 07:04 despite identifying the one-minute discrepancy.Source: §E.1 · p. 16, Table 8 · p. 17

F.1 INFORMATION AVAILABLE TO EACH STAGE

  • The consistency-checking subagent reports question-relevant disagreements and their sources, but does not decide which account to trust.A nonempty report triggers evidence selection; otherwise, the original video goes to the reader.Source: §F.1 · p. 17

F.2 MEDIA TRANSFORMATION AND READER ISOLATION

  • After blinding, the reader gets the resulting media and original question in a fresh context, without the conflict report, selection rationale, earlier responses, or OCR transcriptions.The reader is instructed to distinguish unavailable evidence from evidence of absence.Source: §F.2 · p. 17, Table 10 · p. 18

F.3 EVALUATION AND ABLATIONS

On a shared 500-video cohort, removing either blinding or reader isolation reduces DEAR’s performance under misleading claims. Case studies illustrate claim deference and evidence blinding in action.

F.3 EVALUATION AND ABLATIONS

  • The evaluation compares three methods on 500 distinct source videos, using the same cohort for Qwen3.5-Plus, Seed 2.1 Pro, and Gemini-3.5-Flash.Each method and model receives all three claim conditions; the comparison includes Direct with benchmark and reader prompts, and Self-Review.Source: §F.3 · p. 18
  • Removing blinding returns misleading-claim accuracy to the prompt-level range, while removing reader isolation lowers accuracy by 8.8 points relative to full DEAR.The ablations compare variants on the same Qwen3.5-Plus cohort; both media intervention and reader isolation contribute to the reported result.Source: §F.3 · p. 18

G CASE STUDIES

  • Case studies show benchmark-prompt traces retaining a claim despite scene disagreement, while DEAR’s blinded reader identifies the green plastic box blocking the flaps.The DEAR selector withholds the rendered claim after the subagent reports the disagreement.Source: G CASE STUDIES · p. 18

G.1 REASSIGNING AN ACTION

The case studies illustrate claim deference in action attribution and object identification. In each, the misleading claim steers the response away from the visible evidence.

G.1 REASSIGNING AN ACTION

  • The misleading claim attributes moving shellfish to the woman, and the model switches to her despite the man holding the tongs throughout the transfer.The model selects the man with no claim and with an accurate claim, but changes under the misleading claim.Source: §G.1 · p. 18, Figure 6 · p. 19

G.2 IDENTIFYING THE BLOCKING OBJECT

  • The misleading claim blames the blue-lid container, and all three direct or self-review responses select it despite the green plastic box visibly blocking the flaps.DEAR reports the disagreement, withholds the rendered claim, and its reader identifies the green plastic box from the blinded video.Source: §G.2 · p. 20, Figure 7 · p. 21

H LIMITATIONS

VisualTrust’s controlled single-fact setup limits what its results say about videos with interacting or partially visible claims; DEAR’s evaluation and evidence-separation requirements also constrain its scope.

H LIMITATIONS

  • Real videos may contain interacting claims, mix accurate and inaccurate statements, or leave relevant events partially visible, unlike VisualTrust’s single-fact counterfactuals.Extending the benchmark to these settings would test how models weigh less localized conflicts and incomplete visual accounts.Source: H LIMITATIONS · p. 24
  • Cross-model differences reflect both model behavior and the visual input each model receives because models use their recommended video preprocessing.Source: H LIMITATIONS · p. 25
  • DEAR is evaluated on VisualTrust with three proprietary models and one question per source video, and its check or blinding steps may miss or remove needed evidence.When conflicting and necessary evidence share a region or interval, blinding can remove both and leave the reader unable to answer.Source: H LIMITATIONS · p. 25

Related papers

Loading 2610.21159v1…