PaperlayerView exact revision on arXiv

Source-linked AI summary

Diagnosing and Correcting Bias in MLLM for Long Video Understanding

Xusheng Liang, Jianqiao Sun, Hao Zhang, Yulei Niu, Hengshuang Zhao, Jiawei Ma

arXiv:2610.36210v1cs.CV

TL;DR

Long-video QA is challenged by sparse access to decisive moments and shortcut cues that can mislead MLLMs even when relevant evidence is available. The paper introduces S-MME to diagnose shortcut-sensitive behavior and CLEAR, which uses restricted counterfactual views to steer full-video reasoning. CLEAR consistently improves accuracy and reduces shortcut-consistent errors across MLLMs, with gains reported on public long-video benchmarks.

  • Problem

    Long-video QA must handle sparsely dispersed decisive frames under low-rate sampling, and models may still follow linguistic or visual shortcuts even when relevant evidence is available.

    Source: §1 · pp. 1–2

  • Method

    S-MME diagnoses shortcut-sensitive cases, while training-free CLEAR compares full-video inputs with question-only and salient-view probes and steers hidden states at inference.

    Source: §1 · p. 2

  • Results

    CLEAR consistently improves accuracy and reduces shortcut-consistent errors across different MLLMs; Qwen3-VL-8B improves by +7.8 on MLVU.

    Source: §5.2 · p. 7, §6 · p. 10

  • Takeaways & Limitations

    S-MME tests whether models use decisive video evidence rather than collapsing to plausible shortcut alternatives.

    Source: §3.1 · p. 3

    Each sample requires auxiliary question-only and salient-view forward passes, adding latency and computation versus a single-pass baseline, especially for long videos and large backbones.

    Source: §A.1 · p. 14

Abstract

from arXiv · show

Long-video question answering is challenging since the answer often hinges on a few decisive moments scattered throughout the video, while memory constraints force Multimodal Large Language Models (MLLMs) to sample frames at extremely low rates, resulting in sparsity that risks omitting key evidence. Though aggressively enlarging the context size is intuitive, we argue that performance remains fundamentally constrained by the bias, \emph{i.e.}, the misleading cues caused by spurious \textit{linguistic} correlations and salient yet irrelevant \textit{visual} observations. In this paper, we first introduce {S-MME}, a benchmark designed to systematically diagnose shortcut-biased behavior in MLLMs. To mitigate this failure, we further propose a training-free framework, Counterfactual Long-video Evidence-Aware Reasoning (CLEAR). By treating the full video-question pair as a complete view, CLEAR estimates bias effects via augmented views by only keeping the question or locally salient visual clips. Then, we intervene hidden states to mitigate the discrepancy at inference such that the correct answer can be predicted confidently. With experimental studies across five benchmarks, we show that our method is generalizable and can be integrated in different MLLMs, yielding consistent gains in prediction and robustness. We hope CLEAR can offer a complementary direction in test-time scaling for long-video understanding.

1 Introduction

Long-video MLLMs face both sparse access to decisive frames and shortcut bias that can persist even when relevant evidence is available. The paper introduces S-MME to diagnose these failures and CLEAR to estimate shortcut effects from restricted views and steer full-video reasoning.

1 Introduction

  • Long-video answers can depend on sparsely dispersed decisive frames, while memory limits force low-rate sampling and scaling context or visual capacity incurs substantial cost.The paper frames these approaches as an efficiency–effectiveness trade-off.Source: §1 · p. 1
  • Even when relevant frames are available, MLLMs can follow spurious linguistic cues or salient but causally irrelevant observations, so sparse sampling alone does not explain biased predictions.Existing frame selection, token pruning, and memory methods assume models will use key observations robustly.Source: §1 · pp. 1–2
  • S-MME tests shortcut sensitivity with question-only and salient-yet-irrelevant views that support distractors, then checks whether decisive clips can recover the correct answer.Its cases contrast a restricted observation that induces an incorrect prediction with evidence frames that permit the correct answer.Source: §1 · p. 2
  • CLEAR treats the full video-question pair as a complete view, compares it with question-only and salient visual views, and applies late-layer residual steering to suppress shortcut signals.The framework uses prediction discrepancies to estimate bias effects and intervenes on hidden states without additional training.Source: §1 · p. 2
  • CLEAR yields consistent improvements across Video-MME, LongVideoBench, MLVU, and S-MME, while supporting different MLLMs in classification and next-token prediction modes.The framework is training-free and uses counterfactual views to estimate bias effects before intervening on hidden states.Source: §1 · p. 2

2 Related Works

Related work improves long-video evidence access through input compression, context extension, selection, memory, and agentic reasoning. This paper focuses instead on whether models use available evidence faithfully when shortcut cues offer plausible answers.

Multimodal LLMs for Long-Video Understanding

  • Long-video methods compress visual tokens, reduce redundancy, or extend context to process more evidence, often at non-trivial memory or computational cost.CLEAR is framed as complementary: it studies faithful use of accessible evidence when shortcut cues provide plausible answers.Source: §2 · pp. 2–3

Shortcut Learning and Bias in Visual Question Answering

  • Static-image VQA studies address answer priors, question templates, and dataset correlations, while long-video reasoning adds salient clips that can miss temporally distributed events or later evidence.The paper extends shortcut diagnosis to language priors and partial-observation shortcuts in long-video reasoning.Source: §2 · p. 3

Training-Free Methods for Long-Video Understanding

  • Training-free long-video methods select or retrieve evidence, maintain memories, or use agents for iterative inference, primarily optimizing what evidence is selected, retained, or queried.The related-work section groups these approaches into selection-based, memory-enhanced, and agent-based methods.Source: §2 · p. 3

3 S-MME: Shortcuts in Multi-Modal Evaluation

S-MME makes biased evidence use diagnosable by contrasting complete video evidence with restricted shortcut cues. Its cases specify the evidence and plausible distractor, and its construction process mines long-video questions, identifies shortcuts, and validates samples.

3.1 Making Evidence Use Diagnosable

  • S-MME targets cases where a model predicts a shortcut-consistent distractor despite decisive evidence in the full-video input.This diagnostic distinguishes shortcut-driven failure from errors caused by missing frames, temporal ambiguity, or other sources.Source: §3.1 · p. 3

3.2 Shortcut-Sensitive Cases

  • Each shortcut-sensitive case pairs a uniquely answerable full video with an insufficient restricted view that makes a specific incorrect answer plausible.The two shortcut sources are linguistic correlations from the question or options and salient, query-related clips that lack broader temporal context; each case also annotates the sufficient evidence span and distractor.Source: §3.2 · pp. 3–4

3.3 Benchmark Construction

  • S-MME construction mines questions requiring long-video evidence, uses restricted views to identify plausible distractors, then retains only cases passing human validation.Validation checks answer uniqueness, restricted-view insufficiency, and distractor plausibility; ambiguous or noisy cases are removed.Source: §3.3 · p. 4, Figure 2 · p. 4

4 CLEAR: Counterfactual Long-video Evidence-Aware Reasoning

CLEAR diagnoses shortcut-biased reasoning by comparing the full-video path with question-only and salient-view probes, then uses residual steering to correct the full-video trajectory. It intervenes in late decoder layers, where the last block performs best.

4.2 Shortcut Probing via Counterfactual Views

  • Question-only and salient-view probes expose linguistic and visual shortcut trajectories using the same query and shared decision-token position.The salient view selects locally relevant clips, which can be prominent yet incomplete and miss temporal context needed for the answer.Source: §4.2 · pp. 5–6

4.3 Residual Contrastive Steering

  • CLEAR contrasts restricted-view hidden states with the full-video state and injects activated residual corrections to steer away from shortcut anchors before decoding.Intervention is gated by trajectory discrepancy; the residual is a directional contrast, not an assumed disentangled shortcut component.Source: §4.3 · p. 6
  • Late-layer intervention performs best, with the last block, layers 28–35, giving the highest accuracy.The authors relate this to answer-oriented representations forming near decoding.Source: §4.3 · p. 6, Figure 4 · p. 6

5 Experiments

CLEAR improves long-video benchmark performance and S-MME shortcut robustness, while tests of context scaling, probe roles, and residual strength examine how those gains arise and where they trade off.

5.1 Experimental Setup

  • Experiments use LLaVA-Video-7B, InternVL3-8B, and Qwen3-VL-8B across four public benchmarks and S-MME; GPT-4o and Gemini-1.5-Pro serve as references.CLEAR is applied to open-source models because their hidden states are accessible.Source: §5.1 · p. 7

5.2 Main Results on Public Long-video Benchmarks

  • CLEAR consistently improves open-source long-video MLLMs under the same final-frame budget and decoding setting.Reported gains include +7.8 for Qwen3-VL-8B on MLVU, +7.2 on Video-MME-Long, and +6.3 for InternVL3-8B on LVBench.Source: §5.2 · p. 7

5.3 Diagnostic Evaluation on S-MME

  • On S-MME, CLEAR raises Qwen3VL accuracy from 20.5% to 26.5% and lowers ShortErr from 44.0% to 36.9%.The reduction appears in both conflict types; humans reach 91.2% accuracy with 3.2% ShortErr, while the strongest closed-source model reaches 52.2% accuracy with 16.1% ShortErr.Source: §5.3 · p. 8, Table 3 · p. 8

5.4 Is More Context Enough?

  • Across frame budgets, CLEAR achieves higher accuracy and lower ShortErr; more frames improve baseline accuracy, but ShortErr decreases more slowly.CLEAR also improves Video-MME, suggesting shortcut correction generalizes beyond S-MME.Source: §5.4 · pp. 8–9, Figure 5 · p. 9

5.5 Do Shortcut Probes Align with Conflict Types?

  • Q-only mainly improves linguistic-conflict cases, S-only is more effective on visual-conflict cases, and dual-branch variants improve overall robustness.The table caption identifies Q-only and S-only as language-prior and salient-view corrections, respectively.Source: Table 4 · p. 8, §5.5 · p. 9

5.6 Efficiency and Intervention Analysis

  • With comparable inference budgets, CLEAR achieves larger accuracy improvements than adding full-video frames, which brings limited gains at noticeably higher cost.The authors suggest the benefit mainly comes from shortcut diagnosis and residual corrections.Source: §5.6 · p. 9, Table 5 · p. 9
  • Moderate residual strength gives the best accuracy–robustness trade-off; at excessive α, ShortErr decreases slightly while accuracy drops.The backbone remains frozen, with probes providing inference-time correction signals.Source: §5.6 · p. 9, Table 5 · p. 9

6 Conclusion

The paper concludes that long-video MLLMs can follow linguistic or visual shortcuts despite decisive evidence, and introduces S-MME to diagnose this behavior. CLEAR uses counterfactual views and late-layer residual steering, with experiments showing improved accuracy and fewer shortcut-consistent errors.

6 Conclusion

  • Long-video MLLMs may follow spurious linguistic correlations or salient misleading visuals despite decisive evidence being present.S-MME tests whether models use full-video evidence or collapse to shortcut-consistent alternatives.Source: §6 · p. 9
  • S-MME diagnoses whether models follow complete video evidence or choose shortcut-consistent alternatives.Source: §6 · p. 9
  • CLEAR is a training-free inference-time framework that uses counterfactual views and late-layer residual steering on the full-video path.Experiments on public long-video benchmarks and S-MME show consistent accuracy gains and fewer shortcut-consistent errors across different MLLMs.Source: §6 · p. 10

A.1 Limitation

CLEAR adds inference cost through auxiliary probes, and improving its efficiency is identified as future work.

A.1 Limitation

  • Each CLEAR sample requires question-only and salient-view probes before full-video decoding, adding latency and computation over a single-pass baseline.The overhead is particularly notable for long videos and large MLLM backbones.Source: §A.1 · p. 14
  • The authors identify efficiency improvement as an important future direction after focusing on validating controlled shortcut probes for evidence-grounded reasoning.Suggested approaches include adaptive probe activation, lightweight approximations, representation caching, shared early-layer computation, and early exit when trajectory discrepancy is small.Source: §A.1 · p. 14
  • Potential efficiency approaches include activating probes adaptively or exiting early when full-video and shortcut-probe trajectories differ only slightly.Other listed options are lightweight probe approximations, representation caching, and shared early-layer computation.Source: §A.1 · p. 14

A.2 Implementation Details

The implementation uses frozen open-source MLLMs, two auxiliary probes, and late-layer residual steering; the section also reports open-ended and qualitative results.

A.2 Implementation Details

  • Experiments use frozen LLaVA-Video-7B, InternVL3-8B, and Qwen3-VL-8B, with 64 uniformly sampled frames for full-video inference unless noted otherwise.CLEAR adds question-only and top-K salient-view probes, using K = 4 by default; probes compute shortcut residuals rather than generate the final answer.Source: §A.2 · p. 14
  • Residual steering targets late decoder layers, with Qwen3-VL-8B using layers 28–35 and other backbones using a corresponding late-layer block.A probe-specific residual activates only above τ = 0.15; default residual strength is α = 0.15, selected from the stated accuracy–shortcut suppression trade-off.Source: §A.2 · p. 14
  • Efficiency is measured as inference time relative to the same backbone's vanilla 64-frame baseline, including probing, residual construction, and final decoding.Experiments use NVIDIA H100 GPUs with batch size one for long-video inference.Source: §A.2 · p. 14

Open-ended video QA

  • VideoEvalPro gains are +1.2 points on LLaVA-Video-7B, +1.5 on InternVL3-8B, and +3.4 on Qwen3-VL-8B with CLEAR.The benchmark evaluates free-form answers without candidate options.Source: Table 6 · p. 14, §A.2 · p. 14

Qualitative analysis

  • Figure 6 presents an open-ended QA example in which CLEAR corrects a shortcut-biased response toward the decisive temporal evidence.The figure is described as a qualitative effect of CLEAR on open-ended video QA.Source: §A.2 · p. 14, Figure 6 · p. 15

A.3 Broader Impact

CLEAR is presented as a complementary robustness technique, not a standalone safety mechanism, because it cannot prevent all erroneous or harmful base-model outputs.

A.3 Broader Impact

  • CLEAR steers long-video MLLMs toward evidence-consistent predictions but cannot prevent the base models from producing erroneous, biased, or harmful responses.Source: §A.3 · p. 15
  • The authors characterize CLEAR as a complementary robustness technique rather than a standalone safety mechanism.Source: §A.3 · p. 15
  • The paper cautions against treating CLEAR as a standalone safety mechanism, especially in high-stakes applications.Source: §A.3 · p. 15

Related papers

Loading 2610.36210v1…