Source-linked AI summary
Diagnosing and Mitigating Perception-Decision Misalignment in Omni-LLMs via Modality Subspace Activation
Hongbo Jiang, Jie Li, Yunhang Shen, Tianyu Xie, Pingyang Dai
TL;DR
Omni-LLMs can produce nearly unchanged decisions when indispensable modalities are removed, raising a problem of perceptual-decision misalignment. This paper diagnoses the issue with CausalMSBench, ARR, and LAD, then applies training-free Modality Subspace Activation, which improves causal modality sensitivity and reduces shortcut reliance.
Problem
Omni-LLMs may produce invariant decisions despite missing indispensable modalities, indicating limited faithfulness to multi-modal perception.
Method
The paper combines CausalMSBench, ARR, and LAD with training-free MSA, which balances modality activation in hidden representations.
Results
Across quantitative evaluations, MSA enhances causal modality sensitivity and reduces reliance on shortcut learning.
Takeaways & Limitations
The results support hidden-representation activation balancing as an inference-time approach for restoring modal reliance in Omni-LLMs.
Takeaways & Limitations
CausalMSBench is scoped to samples whose multi-modal dependency is validated by a predefined human-perception gap threshold.
Abstract
from arXiv · showhide
Omni-Large Language Models (Omni-LLMs) power complex multi-modal reasoning in applications like World Action Models and autonomous agents. However, their strong performance often masks a profound Perceptual-Decision Misalignment (PDM), where decisions remain unfaithful to multi-modal perceptions. To diagnose this, we formalize Causal Modality Sensitivity (CMS), operationalized via a dual-lens framework: Answer Retention Rate (ARR) at the macro behavioral level, and Logit Angular Discrepancy (LAD) to track microscopic distribution shifts. We also curate CausalMSBench, a diagnostic dataset isolating language priors. Benchmarking reveals that popular Omni-LLMs exhibit critically low CMS, showing negligible distribution shifts even when key modalities are removed. To rectify this, we propose Modality Subspace Activation (MSA), a training-free inference-time framework that uses Singular Value Decomposition (SVD) to estimate modal activation strengths. MSA dynamically balances modal projections in the last hidden state, effectively restoring CMS across benchmarks.
Introduction
The introduction identifies a perceptual-decision misalignment in Omni-LLMs: decisions can remain nearly unchanged when indispensable modalities are removed. It formalizes Causal Modality Sensitivity (CMS), introduces LAD and ARR for diagnosis, and proposes training-free Modality Subspace Activation (MSA) to realign decisions with perceptual features.
- Motivation: When indispensable modalities are removed, Omni-LLMs show negligible changes in accuracy, outputs, and multi-choice representation distributions, revealing perceptual-decision misalignment.This shortcut-like behavior contrasts with humans’ stronger reliance on necessary modalities and motivates examining both macro behavior and microscopic decision-space shifts.
- Intervention method: MSA is a training-free, inference-time intervention that uses SVD on modality-specific last-hidden-state activations to construct directional subspaces.It projects inference-time hidden states onto these subspaces and uses projection L2 norms to quantify and rank modality activation strength.
- Intervention method: When language priors overshadow essential multimodal cues, MSA adaptively regulates hidden-state geometry by suppressing overactive directions and enhancing underactive modality subspaces.The intervention couples final decisions with perceptual features without parameter retraining.
- Diagnostic perspective: CMS formalizes the alignment gap between perceptual multi-modal inputs and final decisions as a diagnostic evaluation dimension.The framework targets whether decisions remain faithful to the modalities available to the model.
- Metrics and discoveries: LAD detects fine-grained distribution shifts in multi-choice logits, while ARR measures answer retention, together exposing misalignment that accuracy alone can miss.Prior evaluations mainly rely on macro-level metrics, which may overlook models retaining their original answers after modality perturbation.
Related Work
Related work frames Omni-LLM reliability around cross-modal perception–decision consistency, attributing misalignment largely to language priors or modality preferences. Existing training-free remedies include contrastive decoding and attention-based visual-head enhancement, while MSA quantifies modality activation strength using full-modality inputs.
- QUACK (Yuan et al. 2026) audits perception–decision consistency in multimodal agents and reveals a perception–expression disconnect in long-range memory and dynamic confrontation.
- Prior work attributes perception–decision misalignment to language priors or modality preferences, with text-token attention severely dominating other modalities.
- Training-free mitigation methods include contrastive decoding, which contrasts standard-input logits with logits from distorted visual perturbations to calibrate outputs.
- Attention-based methods identify reliable heads from their image-token concentration, augmenting visual contributions while suppressing system prompts and textual priors.
- MSA quantifies each modality’s activation strength from full-modality inputs by leveraging CausalMSBench and microscopic insights from LAD.
Method
The method constructs a dependency-controlled benchmark and diagnoses Perceptual-Decision Misalignment through Causal Modality Sensitivity, combining macro answer retention with microscopic logit shifts. It then introduces training-free Modality Subspace Activation, which uses modality-specific hidden-state subspaces to rebalance inference-time activations.
- Benchmark construction: CausalMSBench enforces Multi-Modal Dependency, requiring indispensable-modality removal to cause a predefined perceptual-performance gap validated by human experts.This constraint ensures that modality omission should materially degrade perception rather than merely perturb redundant inputs.
- CMS diagnosis: CMS diagnoses modality sensitivity by comparing full-modality and omission conditions, with low behavioral divergence indicating insufficient awareness of missing essential modalities.The framework defines low CMS when the expected change in the probability of the correct answer remains within an acceptable bound γ.
- CMS metrics: CMS combines Answer Retention Rate for discrete choice invariance with Logit Angular Discrepancy for microscopic distribution shifts, providing complementary macro- and micro-level diagnostics.ARR measures whether the top-1 choice changes after modality omission, while LAD tracks hidden-state-induced logit discrepancies that macro accuracy can conceal.
- MSA intervention: MSA is a training-free inference-time intervention that builds modality-specific activation subspaces with disjoint anchor samples and SVD, then suppresses stronger and boosts weaker modality projections.During inference, modality activation strengths are estimated from projections of the last hidden state onto truncated singular-vector bases, and a balancing factor λ controls geometric rectification before token prediction.
Experiment · Experimental Settings
The experiments introduce CausalMSBench to evaluate strict multimodal dependence while minimizing language priors, and use disjoint SocialOmni anchor samples for modality-activation estimation. They benchmark five popular Omni-LLMs under modality perturbations, with Table 1 reporting main performance and MSA-based metrics.
- CausalMSBench: CausalMSBench contains 617 samples designed to minimize language priors and enforce multimodal dependency.It combines JointAVBench (Chao et al. 2026), DailyOmni (Zhou et al. 2026), and newly collected and annotated samples.
- CausalMSBench: The dataset construction additionally targets greater diversity while avoiding trivial questions.The construction method is described in the supplementary material.
- Subspace Construction Samples Source: Activation-strength anchor samples are strictly disjoint from evaluation sets to prevent data leakage.This separation is used for modality-subspace construction.
- Subspace Construction Samples Source: The 500 randomly selected SocialOmni samples cover who speaks, when interruptions occur, and how interruptions occur.These three dynamic social-interaction dimensions make SocialOmni suitable for multimodal content understanding.
- Other Settings: Experiments evaluate Qwen2.5-Omni (Xu et al. 2025a), Qwen3-Omni (Xu et al. 2025b), InteractiveOmni (Tong et al. 2025), OmniVinci (Ye et al. 2025), and MiniCPM-4.5 (Yu et al. 2025).These models represent the popular Omni-LLMs selected for evaluation.
- Other Settings: Table 1 reports main performance under three modality-perturbation settings, including Accuracy, MSA-based Accuracy, CMS, and average perturbation.The table uses color rankings for the first-, second-, and third-highest performances in each column.
Overall Results
MSA improves CMS across all evaluated models by increasing the effect of modality perturbations on outputs. Without MSA, models generally show weak sensitivity to audio removal, while CMS and LAD reveal differences hidden by answer accuracy alone.
- Overall Model Comparison: Under perturbation, the consistent CMS pattern is No-Visual&Audio > No-Visual > No-Audio, indicating that audio removal has little output impact.MiniCPM-4.5 shows the highest sensitivity to modality perturbations among the compared models.
- Overall Model Comparison: Qwen3-Omni-30B-A3B-Instruct reaches 57.33% response accuracy normally, but its perturbed-output variation is lower than MiniCPM-4.5’s.InteractiveOmni-4B undergoes an 11.13% accuracy drop after modality perturbation compared with MiniCPM-4.5.
- Modality-Sensitivity Diagnostics: LAD detects internal multi-choice-space shifts even when Interactive-Omni-4B’s textual answers remain correct and unchanged after perturbation.The representative cases have LAD values of 1.34 and 0.85.
- MSA Results: MSA improves CMS for all evaluated models by amplifying modality perturbations’ impact on outputs.This makes models less able to rely solely on the stronger modality when reasoning over full-modality inputs on CausalMSBench.
Case Study
The case study shows that unchanged discrete predictions can conceal substantial continuous logit-distribution shifts after modality perturbations. LAD therefore complements ARR as a diagnostic of modality sensitivity.
- Interactive-Omni-4B produces correct predictions under full modalities and retains ARR = 1.0 for both samples under No-Visual&Audio and No-Visual perturbations.
- LAD reaches 1.34 and 0.85 for samples (a) and (b), respectively, despite unchanged discrete predictions, revealing drastic shifts in continuous logit distributions.
- Figure 4 evaluates how activation sample scale N affects Accuracy (%) and CMS (%), with N = 0 denoting the unsteered baseline without MSA.
The Analysis and Discussion of MSA
MSA is sensitive to both activation-sample size and intervention strength: larger samples stabilize its SVD subspaces, while λ = 0.5 balances accuracy and CMS. At stronger intervention, MSA remains accurate where unguided enhancement produces inflated CMS and degraded accuracy.
- Sample Size Sensitivity: Increasing the activation-matrix sample size to N = 1000 stabilizes representative SVD subspaces, whereas N = 125 produces noisy singular directions and suboptimal performance.The evaluated sample sizes are N ∈ {125, 250, 500, 1000}, compared against the no-MSA baseline N = 0.
- Intervention Strength: λ = 0.5 provides the best reported Pareto trade-off, preserving 40.07% Accuracy while maintaining effective representation enhancement.At λ = 0.1, Accuracy drops to 8.79% despite an inflated CMS score; increasing λ restores reasoning stability.
- Steering Strategy Comparison: At λ = 0.9, Random reaches CMS 61.06 but Accuracy falls to 38.44%, whereas MSA maintains 44.63% Accuracy, exposing unguided enhancement’s weakness.At λ = 0.5, Random appears comparable to MSA because lower intervention strength conceals this drawback.
Conclusion
This work diagnoses Perceptual-Decision Misalignment (PDM) in Omni-LLMs, where predictions remain invariant after indispensable modalities are omitted. It introduces CausalMSBench, evaluates behavior with ARR and LAD, and presents training-free Modality Subspace Activation (MSA) to restore modal reliance.
- Conclusion: The study targets PDM in Omni-LLMs, characterized by invariant predictions when indispensable modalities are omitted.
- Conclusion: CausalMSBench is curated under strict multi-modal dependency to diagnose this misalignment.
- Conclusion: Model behavior is evaluated using Answer Retention Rate (ARR) and Logit Angular Discrepancy (LAD).
- Conclusion: Modality Subspace Activation (MSA) is introduced as a training-free mechanism that geometrically suppresses dominant modality activation to restore modal reliance.