Source-linked AI summary
Taming Visual Neglect: A Variational Information Bottleneck Framework for Adaptive Attention in Multimodal In-Context Learning
Kaito Tanaka, Yuji Nishimura, Keisuke Matsuda, Aya Nakayama
TL;DR
Multimodal ICL can either benefit from visual demonstrations or neglect them, while the conditions governing this behavior remain insufficiently understood. VIB-ICL uses the Information Bottleneck principle and CMIG to characterize visual utility, derive adaptive attention rules, and evaluate the approach across five benchmarks. The method reports up to 4.7% accuracy gains and 35% fewer required demonstrations.
Problem
Existing studies show that visual context sometimes improves multimodal ICL and sometimes becomes redundant, while theory does not explain this cross-modal redundancy or its attention implications.
Method
VIB-ICL estimates Cross-Modal Information Gain with variational bounds, derives Information Bottleneck-based attention reallocation, and selects demonstrations maximizing cross-modal information gain.
Results
Across five multimodal ICL benchmarks, VIB-ICL achieves consistent improvements, including up to 4.7% accuracy gains and 35% reduction in required demonstrations.
Takeaways & Limitations
CMIG determines whether visual context provides non-redundant information, while visual neglect is Information Bottleneck-optimal when visual information is redundant.
Takeaways & Limitations
The theoretical analysis is limited to the linear attention model, motivating extensions to general transformer architectures.
Abstract
from arXiv · showhide
Large vision-language models exhibit strong in-context learning (ICL) capabilities, yet when and why visual context helps multimodal ICL remains poorly understood. Empirical studies show a puzzling dichotomy: models sometimes effectively leverage visual demonstrations, yet often neglect them entirely. We propose VIB-ICL, an information-theoretic framework that resolves this dichotomy through the Information Bottleneck principle. We introduce the Cross-Modal Information Gain (CMIG), which quantifies the additional mutual information that visual context provides about the target beyond textual context. We derive a generalization bound showing that multimodal ICL's excess risk over text-only ICL is governed by the CMIG, proving that multimodal ICL provably outperforms text-only ICL when visual information is non-redundant. We further prove that visual context neglect, often viewed as a failure mode, is the Information Bottleneck-optimal solution when visual information is redundant, yielding a closed-form Attention Reallocation Principle that prescribes how visual attention weights should be adaptively adjusted. We instantiate this principle in the VIB-ICL algorithm, which estimates CMIG via variational bounds and dynamically reallocates attention. Experiments on five benchmarks demonstrate consistent improvements of up to 4.7\% accuracy gains and 35\% reduction in required demonstrations, validating our theoretical predictions.
1 Introduction
VIB-ICL addresses when visual context helps multimodal ICL by measuring its non-redundant information and deriving Information Bottleneck-based attention policies. Its theory and experiments connect CMIG to performance, rational visual neglect, adaptive reallocation, and improved benchmark efficiency.
- Motivation: Visual context benefits multimodal ICL inconsistently across tasks and architectures, motivating a theory of information redundancy and attention allocation.Prior studies report both substantial visual-demonstration gains and effective visual neglect.
- Framework: VIB-ICL introduces Cross-Modal Information Gain, measuring additional target-label information supplied by visual context beyond textual context.CMIG is defined as ΔI = I(V; Y|T, D) − I(V; Y|T).
- Theory: The generalization bound links multimodal ICL’s excess risk over text-only ICL to CMIG, establishing an advantage when ΔI > 0 with sufficiently many demonstrations.The bound is stated in Theorem 7.
- Theory: Visual context neglect is Information Bottleneck-optimal when CMIG falls below the IB regularization threshold, ΔI < λIB.This provides a principled account of neglect as rational behavior when visual information is redundant.
- Algorithm: VIB-ICL estimates CMIG variationally, reallocates visual attention adaptively, and selects demonstrations maximizing cross-modal information gain.Figure 1 presents this pipeline from estimation through attention adjustment and demonstration selection.
- Experiments: Up to 4.7% accuracy gains and 35% fewer required demonstrations were reported across five multimodal ICL benchmarks.The experiments also provide interpretable insights into cross-modal information utilization.
2 Related Work
Related work spans theoretical accounts of text-only ICL, multimodal ICL benchmarks and applications, and methods addressing visual context neglect. These studies motivate VIB-ICL’s focus on cross-modal interaction and dynamic attention allocation.
- Theoretical ICL: Text-only ICL theory includes implicit Bayesian inference and generalization analyses whose sample complexity depends on task-distribution effective dimension.The cited work primarily develops theory for text-only settings.
- Multimodal ICL: Multimodal ICL research includes visual demonstrations, benchmark evaluation, ambiguous spatial reasoning, test-time recognition, and visual robustness applications.These works examine how models learn from or benefit from visual context across tasks.
- Attention and Adaptation: Methods addressing visual context neglect include DARA’s dynamic modality attention, M2IV’s representation engineering, and related efficient adaptation approaches.These methods target limitations in multimodal ICL through attention adjustment or representation-level adaptation.
3 Methodology
The methodology formalizes multimodal ICL through CMIG, which measures visual information about labels beyond textual context, and establishes assumptions and an information decomposition for analyzing its benefit.
- Problem Setup: Multimodal ICL predicts labels from image-text-label demonstrations without updating model parameters.The demonstrations and query are drawn independently from the same task distribution.
- Cross-Modal Information Gain: CMIG measures the additional mutual information that visual context provides about labels beyond textual context, conditioned on demonstrations.When CMIG is positive, visual context contains non-redundant information; when it is approximately zero, text may already suffice.
- Assumptions: The framework assumes conditional independence of visual and textual features given the label, sub-Gaussian features, and bounded label noise below one-half.These assumptions support the information-theoretic and concentration-based analysis.
- Information Decomposition: Under conditional independence, the benefit of visual context is captured by I(V; Y|T), which becomes small when visual and textual features are highly correlated.Conversely, label-relevant visual information absent from text increases the potential benefit of multimodal ICL.
3.4 Generalization Bound for Multimodal ICL
The generalization analysis decomposes multimodal ICL risk into approximation and estimation components, showing that positive CMIG can make multimodal ICL outperform text-only ICL with enough demonstrations.
- Generalization Bound: Theorem 7 bounds multimodal ICL’s excess risk over text-only ICL with probability at least 1 −δ using CMIG, effective dimension, and demonstration count.The bound is stated under Assumptions 3–5.
- Risk Decomposition: The risk analysis decomposes performance into an approximation gap and estimation errors for multimodal and text-only predictors.Multimodal ICL has larger effective dimension, while text-only ICL incurs greater estimation error per dimension because it lacks visual information.
- Information-Theoretic Term: The reduction in estimation variance from conditioning on visual information is proportional to I(V; Y|T) under sub-Gaussian assumptions.This connects conditional mutual information to the benefit of incorporating visual context.
- Main Guarantee: Multimodal ICL provably outperforms text-only ICL when CMIG is positive and the number of demonstrations is sufficiently large.The guarantee holds when the negative information-benefit term dominates the positive estimation term.
3.5 Sample Complexity of Multimodal ICL
The sample-complexity analysis links CMIG to fewer demonstrations and characterizes visual attention as an Information Bottleneck decision: neglect redundant visual context and increase attention when it is informative.
- Sample Complexity: Theorem 8 compares the demonstrations required by multimodal and text-only ICL to achieve excess risk ϵ and defines their sample-complexity reduction ratio.The comparison derives multimodal requirements from Theorem 7 and sets ∆I to zero for text-only ICL.
- Sample Complexity: CMIG reduces the effective target excess risk from ϵ to ϵ + ∆I/(2σ2_y), producing a quadratic sample-complexity reduction.When ∆I is much larger than ϵ, multimodal ICL requires dramatically fewer demonstrations for the same performance.
- Visual Context Neglect as IB Optimal: Visual context neglect is Information Bottleneck-optimal when I(V; Y|T) is below the threshold λIB.The threshold reflects the model’s compression–relevance trade-off: smaller β yields larger λIB and more neglect.
- Attention Reallocation Principle: The Attention Reallocation Principle provides a closed-form rule for assigning visual attention based on conditional mutual information and the IB threshold.It supplies a principled basis for dynamically reallocating attention according to visual informativeness.
- Attention Reallocation Principle: When I(V; Y|T) exceeds λIB, the optimal visual attention weight becomes positive and increases linearly with informativeness.The weight approaches one as visual information becomes highly non-redundant.
3.7 VIB-ICL Algorithm
VIB-ICL operationalizes the theory by estimating CMIG variationally, adapting visual attention layer by layer, and selecting demonstrations with high estimated cross-modal information gain.
- Algorithm Components: VIB-ICL combines variational CMIG estimation, adaptive attention reallocation, and information-gain-guided demonstration selection.These three components instantiate the Attention Reallocation Principle.
- CMIG Estimation: The algorithm estimates intractable CMIG with a variational lower bound using approximate posterior distributions for multimodal and text-only predictions.The distributions are parameterized with LVLM encoder representations and optimized on the demonstration set.
- Adaptive Attention: VIB-ICL adjusts visual-token attention scores in each transformer layer using layer-specific CMIG estimates.This lets the model emphasize visual information at informative layers and neglect it where it is redundant.
- Demonstration Selection: The method selects a subset of demonstrations that maximizes estimated CMIG, using greedy additions based on marginal information gain.Selection is performed from a candidate pool subject to a fixed subset size.
4 Experiments
Experiments across five multimodal ICL benchmarks test VIB-ICL’s theoretical predictions, component contributions, efficiency, scalability, stability, and backbone generalization. VIB-ICL consistently improves performance while adaptively using visual context according to estimated CMIG.
- Main Results: VIB-ICL achieves the best performance on four of five benchmarks and the highest average accuracy, with gains of 4.7% on TrueMICL and 3.9% on CausalVLBench.On VL-ICL Bench, it improves 2.1% over DARA.
- CMIG Validation: CMIG correlates strongly with multimodal accuracy gains, with Pearson r = 0.89 and p < 0.001 across task categories.High-CMIG tasks benefit substantially from visual context, whereas text-heavy VQA shows minimal benefit.
- Visual Neglect: For low-CMIG tasks, forcing visual attention decreases performance relative to visual neglect, while high-CMIG tasks favor positive visual attention.The observed optimum matches the Attention Reallocation Principle.
- Ablation Study: Adaptive attention reallocation contributes +2.3% average improvement, compared with +1.1% from CMIG estimation and +0.8% from demonstration selection.The ablation identifies attention reallocation as the largest individual contributor.
- Sample Efficiency: VIB-ICL matches Vanilla ICL accuracy with approximately 35% fewer demonstrations, with the largest efficiency gains on high-CMIG tasks.This pattern is consistent with the predicted CMIG-related sample-complexity reduction.
- Robustness and Scalability: VIB-ICL adds less than 5% computation-time overhead, exhibits lower variance than all baselines, and improves performance across LLaVA-1.5-13B and Qwen-VL-Chat.Attention reallocation is essentially cost-free because it scales existing attention scores.
5 Conclusion
VIB-ICL uses Cross-Modal Information Gain to determine when visual context benefits multimodal ICL and when neglect is optimal. Its adaptive attention algorithm improves performance across five benchmarks, while the framework remains open to broader architectures, uncertainty modeling, and modalities.
- VIB-ICL identifies CMIG as the fundamental quantity determining the benefit of visual context in multimodal ICL.
- Multimodal ICL provably outperforms text-only ICL when CMIG is positive and sufficient demonstrations are available.
- Visual context neglect is IB-optimal when visual information is redundant with textual content.
- VIB-ICL achieves consistent improvements across five benchmarks through variational CMIG estimation and adaptive attention reallocation.
- Extending the analysis beyond linear attention, adding task-level uncertainty, and applying the approach to audio and video remain future directions.
A Proof of Theorem 7
The proof of Theorem 7 decomposes multimodal and text-only prediction risk, relates their approximation gap to mutual information, and bounds estimation error using PAC-Bayesian arguments.
- The proof analyzes Bayes-optimal multimodal prediction under squared loss.The loss is ℓ(ŷ, y) = (ŷ − y)^2.
- It separately characterizes the risk of the text-only Bayes predictor for comparison.
- The approximation gap is derived by comparing the multimodal and text-only risks.
- The law of total variance connects variance reduction to mutual information in the proof.
- PAC-Bayesian arguments bound estimation error for the transformer-implementable multimodal ICL predictor class.
- The posterior-prior KL term is bounded by C · d_eff, where d_eff is the predictor class’s effective dimension.
B Proof of Theorem 9
Theorem 9’s proof analyzes a linear multimodal representation under Gaussian noise, bounds information terms, and evaluates the IB objective at zero visual attention to establish neglect optimality below a threshold.
- The proof expands I(Z; V, T) for a linear representation combining visual and textual features with Gaussian noise.The representation is Z = α_vϕ(V) + α_tψ(T) + ϵ, with ϵ distributed as N(0, σ²I).
- It expands I(Z; Y) using a data processing inequality chain under sub-Gaussian assumptions.
- The proof evaluates the derivative of the Information Bottleneck objective at α_v = 0.
- When I(V; Y|T) < λ_IB, α*_v = 0 is optimal, so visual attention is neglected below the IB threshold.The derivation also states the equivalent condition I(V; Y|T) < λ_IB/β = 1/β² before simplifying to λ_IB = 1/β.
C Variational CMIG Estimation Details
The variational CMIG estimation procedure parameterizes separate multimodal and text-only variational distributions, trains their estimator, and applies the same procedure layer by layer.
- The procedure parameterizes variational distributions for estimating CMIG.
- The multimodal representation network uses concatenated visual and textual representations as input.
- The text-only variational parameters use a separate two-layer MLP that receives only the textual representation.
- The variational CMIG estimator is trained by maximizing a specified training objective.
- Optimization uses Adam with learning rate 10^-4 for 1000 steps on the demonstration set.
- Layer-wise CMIG estimation applies the same variational procedure to intermediate representations at each transformer layer.