Who Drives the Probability Game of VLMs? A Temporal Causal Drive Evaluation Framework

Shuyao Xiao; Shengling Wang; Haoyu Niu; Ke Chao; Changwei Xu; Xinran Duan; Chaoyong Jiang

Source-linked AI summary

Who Drives the Probability Game of VLMs? A Temporal Causal Drive Evaluation Framework

Shuyao Xiao, Shengling Wang, Haoyu Niu, Ke Chao, Changwei Xu, Xinran Duan, Chaoyong Jiang

arXiv:2609.02000v1Sep 2, 2026cs.CV

TL;DR

VLM evaluation often measures final-answer quality without showing which information sources drive generation. This paper introduces SCM-based interventions and backdoor adjustment to derive step-indexed causal-drive metrics, finding a transition from early question and visual guidance toward greater prefix reliance and improved recovery over observational PMI baselines.

Problem
Conventional outcome metrics and largely static modality analyses provide limited evidence about which sources drive VLM generation and how source dominance evolves over time.
Method
The framework models autoregressive decoding with an SCM and uses interventions and backdoor adjustment to derive reference-free PCD, QCD, and VCD trajectories.
Results
Causal-drive trajectories show a consistent shift from stronger early question and visual guidance toward increasing prefix reliance, while QCD and PCD reduce recovery error over observational PMI baselines by 34.8% and 47.1%, respectively.
Takeaways & Limitations
Causal-drive trajectories complement conventional answer-level evaluation with source-level diagnostics for multimodal generation.
Takeaways & Limitations
Interpretation depends on the assumed SCM, fixed model parameters, absence of unobserved common causes beyond adjusted variables, and the evaluated models, scales, and settings.

Abstract

from arXiv · show

Vision-language models (VLMs) are increasingly evaluated on complex image and video understanding tasks, yet conventional metrics primarily assess final-answer quality and reveal little about how different information sources shape the generation process. We propose a causal and temporal evaluation framework that traces the evolving roles of visual input, question text, and generated prefixes during autoregressive decoding. Grounded in a Structural Causal Model, we use interventions and backdoor adjustment to derive three step-indexed causal-drive metrics---Visual Causal Drive (VCD), Question Causal Drive (QCD), and Prefix Causal Drive (PCD)---for characterizing source-specific generation patterns without requiring reference answers. Experiments on Qwen3-VL-8B-Instruct across MAVIS, LLaVA-Video-178K, and MiraData, together with cross-model validation on InternVL2-8B, reveal a consistent transition from stronger early question and visual guidance toward increasing reliance on generated prefixes. Randomized-intervention validation shows that QCD and PCD reduce recovery error over observational PMI baselines by 34.8\% and 47.1\%, respectively. On VLMBias, the prefix--visual imbalance score achieves 0.767 AUROC and 0.873 AUPRC for distinguishing prior-driven from visually grounded generations. These results show that causal-drive trajectories provide complementary source-level diagnostics for multimodal generation.

1. Introduction

Conventional VLM evaluation emphasizes final-answer quality, while existing process and modality analyses provide limited insight into which sources drive generation and how that reliance changes over time.

1. Introduction: Outcome-oriented metrics compare generated outputs with references or annotations but reveal limited evidence about what supports an answer.Examples include BLEU, ROUGE, METEOR, CIDEr, SPICE, BERTScore, and task accuracy.
1. Introduction: Final-answer correctness does not establish the reliability of the underlying reasoning or reveal the intermediate evidence used.
1. Introduction: Existing process-oriented methods largely analyze textual reasoning chains, step correctness, coherence, efficiency, or confidence trajectories.
1. Introduction: Tracing source dominance can help distinguish weak visual grounding, question misunderstanding, and prefix-induced error accumulation.
1. Introduction: Static modality-contribution methods typically produce global or sample-level importance estimates rather than tracing dominance during autoregressive generation.
1. Introduction: The proposed framework uses an SCM, interventions, and backdoor adjustment to derive step-indexed drives for prefixes, visual input, and question text.It is designed to characterize source-specific generation patterns dynamically.

3. Temporal Causal Framework

The framework models VLM answer generation as a temporal autoregressive process in which visual input and question text influence each generated token, while prior tokens propagate through the prefix.

3. Temporal Causal Framework: Answer generation is represented as a temporal autoregressive process over visual tokens, question tokens, and an answer sequence that grows token by token.Generation continues until an end-of-sequence token or maximum length is reached.
3. Temporal Causal Framework: The causal graph encodes prefix-to-token propagation, continuous question guidance, and visual grounding throughout generation.
3. Temporal Causal Framework: Prefix effects are confounded by shared question and visual causes, while question effects are confounded because visual input affects both the question and answer.
3. Temporal Causal Framework: The framework uses SCM interventions instead of repeated input ablation to separate causal influence from statistical association.Repeated counterfactual decoding is described as costly, unstable, and difficult to scale.
3. Temporal Causal Framework: Backdoor adjustment is applied to prefix and question effects, whereas visual effects are identified through truncated factorization because visual input is a root node.

4. Causal Drive Metrics

The causal-drive framework calibrates source-specific interventional effects against a shared null-source likelihood and tracks prefix, question, and visual influence across decoding steps.

4. Causal Drive Metrics: A shared null-source reference sets a common baseline by evaluating the target sequence with both visual and question sources absent.Causal-drive scores are calibrated relative to this likelihood.
4. Causal Drive Metrics: Prefix Causal Drive (PCD) measures the influence of observed generation history on the current token using an intervention that fixes the historical prefix.Its successive values form a temporal profile of prefix-conditioned generation.
4. Causal Drive Metrics: Question Causal Drive (QCD) quantifies the causal driving effect of the question on the generated answer.
4. Causal Drive Metrics: Visual Causal Drive (VCD) measures visual-evidence contribution while the question is fixed to a null reference.It traces visual support throughout generation under absent question information.
4. Causal Drive Metrics: Together, PCD, QCD, and VCD provide step-indexed diagnostics of how generation history, question information, and visual evidence evolve beyond final-answer evaluation.

5. Experimental Evaluation

Experiments across models and datasets show that causal-drive trajectories expose how question, visual, and prefix influences evolve during generation, while source-removal and intervention tests validate their attribution. These diagnostics complement conventional answer-level metrics by revealing source reliance and generation patterns.

Temporal transition of causal drives: PCD rises rapidly and remains high, whereas QCD and VCD are strongest early and generally decline across decoding.This indicates increasing reliance on generated history alongside earlier question and visual guidance.
Temporal transition of causal drives: Dataset- and model-specific trajectories include faster prefix reliance on MAVIS, longer-lasting question guidance on video QA datasets, and stronger early-to-middle QCD and VCD for InternVL2-8B on MAVIS.These differences show that temporal source reliance varies across evaluation settings.
Clustering generation patterns: Trajectory clustering identifies prefix-driven, visually suppressed, and multi-source coordinated generation patterns across samples.The clustering uses 10 normalized position bins and concatenated QCD, VCD, and PCD trajectories, producing three behavioral groups.
What conventional metrics miss: Answer-level metrics can miss meaningful source support when generated answers differ from references and cannot explain why matching answers are produced.Causal drives and additional queries reveal question and visual reliance beyond answer–reference agreement.
Validating causal attribution: 34.8% and 47.1% reductions in recovery error are achieved by QCD over PMIQ and PCD over PMIP, respectively, under randomized interventions.QCD has lower error on 98% of targets, while PCD does so on all targets.
Complementarity to conventional metrics: Causal drives are weakly correlated with conventional metrics, with PCD nearly uncorrelated, indicating complementary process-level signals rather than replacements.QCD and VCD show weak correlations with BERT-F1, BLEU-4, and TER.
Validating source-level attribution: 0.767 AUROC and 0.873 AUPRC are achieved by combining prefix and visual contributions with SP V for distinguishing two generation patterns.Answer-level VCD alone yields AUROC = 0.747 and AUPRC = 0.842.

6. Conclusion

The framework traces how visual evidence, question text, and generated history shape VLM generation, revealing temporal source-reliance patterns without reference answers. Its causal-drive trajectories complement answer-level metrics, while validation demonstrates improved attribution and a scope-bound interpretation.

Causal-drive metrics PCD, QCD, and VCD trace how visual evidence, question text, and generated history shape VLM generation without reference answers.They are derived by modeling autoregressive decoding with a Structural Causal Model.
34.8% and 47.1%: QCD and PCD reduce recovery error over observational PMI baselines in randomized-intervention validation.
0.767 AUROC and 0.873 AUPRC: the prefix–visual imbalance score distinguishes prior-driven from visually grounded generations on VLMBias.
The drives diagnose source reliance rather than replace correctness metrics, and strong visual drive does not necessarily indicate correct visual understanding.

7. Proofs of Causal Effects

The causal-effect proofs show how interventions and backdoor adjustment identify prefix and question effects, while the visual effect follows from fixing the question to a null prompt under the assumed SCM.

The prefix effect is identified by fixing the generated prefix and blocking backdoor paths through question and visual input.The relevant paths are A<t ←Q →At and A<t ←V →At.
The question effect is identified by intervening on Q and adjusting over V to block the confounding path Q ←V →A≤t.
The visual effect fixes V and the question to the null prompt, reducing the interventional distribution to the model’s conditional generation distribution under the assumed graph.The implementation uses a null model input and teacher-forced scoring to isolate visual support.

8. Proofs of Causal Drives

The causal-drive definitions compare source-specific interventional generation probabilities with a shared null-source likelihood, producing step-indexed measures for prefix, question, and visual influence.

The shared null-source reference is P0(A≤t = a≤t) = P(A≤t = a≤t | V = ∅, Q = ∅).
PCD compares the interventional probability after fixing the observed history with the null-source likelihood, and larger values indicate stronger prefix-conditioned generation.
QCD compares generation probability under the question intervention with the shared null-source likelihood.
VCD measures visual support relative to the shared null-source likelihood while holding the question at the null prompt.It is cumulative over the answer prefix and traces visual support when question information is absent.

9. Computational Complexity and Resource Overhead

The framework’s computational cost depends on sequence length, vocabulary size, and the number of visual or question interventions, with PCD the most expensive and VCD the lightest.

O(|V| · |Q| · T), O(|V| · T), and O(T): PCD, QCD, and VCD have progressively lower per-sample computational complexity.
Memory overhead grows with vocabulary size, sequence length, and the number of visual or question conditions because per-token intervention probabilities must be stored.
Batching, caching, and sampling over large visual or question sets keep GPU memory usage manageable for typical VLM evaluation workloads.
K = 4 support yields Pearson correlations of 1.000 for QCD and 0.997 for PCD against the K = 16 reference.At K = 8, the correlations are 1.000 and 0.999, respectively.

10. Reference and Implementation Sensitivity

The framework tests whether causal-drive trajectories are robust to reference-image and prompt choices, then summarizes generations into trajectory-based clusters. The resulting trajectories remain highly consistent under these implementation changes, while clustering yields three interpretable modes.

Prompt sensitivity: 0.962 and 0.977 sample-wise Pearson correlations were obtained for QCD and PCD after replacing the empty prompt with “Please provide a response.”For VCD, the empty-prompt and matched-prompt versions reached an average sample-wise Pearson correlation of 0.971, with 95% CI [0.965, 0.976].
Trajectory representation: Each sample is divided into 10 relative decoding-position bins, with QCD, VCD, and PCD averaged separately within each bin.The three binned trajectories are concatenated into a representation for each generation.
Generation modes: A 30-dimensional standardized representation is clustered with k-means using k = 3, producing prefix-dominant generation, visual suppression, and multi-source coordination modes.Adjacent k values and different random initializations preserve the main behavior patterns.

12. VLMBias Validation Details

VLMBias validation tests whether source-level measures distinguish prior-driven from visually grounded generations. The prefix–visual imbalance score performs strongly, while cross-scale results preserve polarity but show differing temporal agreement across drives.

Validation set: The binary VLMBias validation uses 170 samples: 112 prior-driven and 58 visually grounded generations.The full set contains 200 valid samples, including 30 other-error generations.
Binary discrimination: 0.767 AUROC and 0.873 AUPRC were achieved by the Prefix-Visual imbalance score SP V = PCD−VCD.The corresponding 95% confidence intervals are [0.691, 0.839] and [0.826, 0.916].
Cross-scale validation: QCD and PCD remain positive while VCD remains negative across all 50 Qwen3-VL-32B-Instruct VLMBias samples.This preserves source-level polarity relative to the evaluated smaller model.
Cross-scale validation: 0.992 correlation for VCD, 0.753 for PCD, and 0.295 for QCD quantify cross-scale agreement in four-step mean trajectories.VCD is highly consistent, PCD shows moderate agreement, and detailed QCD trajectories differ across scales.
Baseline construction: PMI provides observational baselines for QCD and PCD, whose effects require causal adjustment under the assumed SCM.VCD remains interventional and is identified through truncated factorization because V is a root node; the PMI quantities characterize conditional statistical dependence.