Source-linked AI summary
Dr. SHAP-AV: Decoding Relative Modality Contributions via Shapley Attribution in Audio-Visual Speech Recognition
Umberto Cappellazzo, Stavros Petridis, Maja Pantic
TL;DR
AVSR models combine audio and visual speech cues, but how they balance these modalities remains unclear. Dr. SHAP-AV uses Shapley attribution to analyze modality contributions across conditions and decoding stages, finding that models shift toward visual reliance under noise while retaining substantial audio contributions even under severe degradation.
Problem
How AVSR models balance acoustic and visual information remains unclear, motivating a principled analysis of modality contributions.
Method
Dr. SHAP-AV applies Shapley attribution to analyze AVSR modality contributions across acoustic conditions, architectures, decoding stages, and temporal alignment.
Results
Models shift toward visual reliance as audio quality deteriorates but retain 38-46% audio contributions at −10 dB SNR, while modality balance evolves during generation and temporal alignment persists.
Takeaways & Limitations
SNR is the dominant factor driving modality balance, supporting explicit mechanisms that modulate modality reliance based on input quality.
Takeaways & Limitations
The analysis excludes prompt and special tokens.
Abstract
from arXiv · showhide
Audio-Visual Speech Recognition (AVSR) leverages both acoustic and visual information for robust recognition under noise. However, how models balance these modalities remains unclear. We present Dr. SHAP-AV, a framework using Shapley values to analyze modality contributions in AVSR. Through experiments on six models across two benchmarks and varying SNR levels, we introduce three analyses: Global SHAP for overall modality balance, Generative SHAP for contribution dynamics during decoding, and Temporal Alignment SHAP for input-output correspondence. Our findings reveal that models shift toward visual reliance under noise yet maintain high audio contributions even under severe degradation. Modality balance evolves during generation, temporal alignment holds under noise, and SNR is the dominant factor driving modality weighting. These findings expose a persistent audio bias, motivating ad-hoc modality-weighting mechanisms and Shapley-based attribution as a standard AVSR diagnostic.
1. Introduction
The introduction frames AVSR modality balance as an underexplored problem and presents Dr. SHAP-AV, a Shapley-based framework for analyzing contributions across acoustic conditions, decoding, and temporal alignment. Experiments reveal adaptive visual reliance under degradation alongside persistent audio bias, motivating explicit modality weighting and Shapley attribution as an AVSR diagnostic.
- Motivation: AVSR supplements noise-vulnerable acoustic signals with visually informative lip movements, but visual contributions are less evident in clean conditions because audio is temporally richer and visual cues are ambiguous.Audio directly encodes the speech signal at high temporal resolution, whereas lower-rate lip movements leave many phonemes visually indistinguishable.
- Motivation: Explicit VSR supervision strengthens visual-only performance: AV-HuBERT and Omni-AVSR outperform four AVSR-task-only models that collapse when audio is absent.The four task-only models are Llama-AVSR, Llama-SMoP, Whisper-Flamingo, and Auto-AVSR.
- Approach: Dr. SHAP-AV uses Shapley values to quantify modality contributions through global weighting, generation-time dynamics, and temporal alignment across six AVSR models.The framework extends analysis across SNR levels from clean speech to −10 dB and includes cross-attention encoder-decoder architectures.
- Key findings: 38-46% audio contributions persist at −10 dB SNR, although models shift toward visual reliance as audio quality deteriorates.Different noise types produce varying visual reliance, with more challenging conditions causing larger shifts toward vision.
- Key findings: Modality contribution changes during decoding: Whisper-Flamingo and Omni-AVSR increase audio reliance, whereas AV-HuBERT maintains stable balance.Both modalities also remain temporally aligned with output tokens under acoustic degradation.
- Key findings: Acoustic conditions dominate modality balance, while recognition difficulty has minimal within-SNR impact; utterance-duration effects vary by architecture.These persistent biases motivate explicit modality-weighting mechanisms and Shapley-based attribution as a standard AVSR diagnostic.
2. Methodology
Dr. SHAP-AV adapts Shapley-value attribution to autoregressive AVSR by measuring how audio and visual features affect each generated token’s log-probability. Its Shapley matrix supports complementary global, generative, and temporal-alignment analyses of modality contributions.
- Shapley-value background: Shapley values quantify each feature’s contribution through weighted marginal contributions across all possible coalitions.They satisfy efficiency, symmetry, linearity, and null-player axioms, while attributing to the model’s predictive distribution rather than ground-truth correctness.
- Shapley-value background: Permutation SHAP and Sampling SHAP approximate otherwise intractable coalition evaluations, using M = 2000 sampled coalitions for stable modality attribution.Both methods provide unbiased estimates that converge to the true Shapley values and are implemented with the shap library.
- Autoregressive AVSR adaptation: For each generated token, Dr. SHAP-AV computes feature Shapley values from expected conditional log-probability while masking unobserved audio or visual features.Prompt and special tokens are excluded; audio and visual features correspond to modality representations or projected tokens depending on the model architecture.
- Autoregressive AVSR adaptation: The resulting Shapley matrix Φ ∈ R^N×T records each input feature’s contribution to every generated token and underpins all subsequent analyses.The matrix entries are ϕ_i,t for input feature i and output token t.
- SHAP-based analyses: Three complementary analyses operate on Φ: Global SHAP measures overall modality balance, Generative SHAP tracks contribution dynamics during generation, and Temporal Alignment SHAP tests input-output temporal correspondence.Temporal Alignment SHAP asks whether early input features contribute more to early tokens and late features to late tokens, probing preservation of sequential speech structure.
3. Experimental Setup
The experiments evaluate six AVSR models on the LRS2 and LRS3 benchmarks, spanning LLM-based and cross-attention-based architectures. Modality attribution uses consistent zero masking across models and conditions to compare relative utilization.
- Datasets: Experiments use LRS2 and LRS3, containing 225 and 433 hours of English video, respectively.LRS2 footage comes from BBC programs, while LRS3 consists of TED talk clips.
- Model families: Six state-of-the-art AVSR models are grouped into LLM-based and cross-attention-based families.LLM-based models process modality-specific encoder outputs alongside prompt tokens, whereas cross-attention-based decoders process audio-visual embeddings through cross-attention.
- Model families: The LLM-based group includes Llama-AVSR and Llama-SMoP, which use modality-specific encoders and projectors or sparse mixture-of-experts modules.The analysis focuses on Llama-SMoP’s Joint-Experts, Joint-Router configuration.
- Model families: The cross-attention-based group includes AV-HuBERT and Auto-AVSR, using masked prediction or fused ResNet- and Conformer-based streams with Transformer decoding.AV-HuBERT uses a noise-augmented model pretrained on LRS3 and VoxCeleb2 and fine-tuned on LRS3.
- Attribution setup: Excluded features and tokens are masked by setting them to zero, consistently across models and conditions to compare relative modality attributions.For LLM-based models, audio and video tokens are masked after the projection layers.
4. Results
Across benchmarks and analyses, AVSR models adapt modality weighting to acoustic conditions, shifting toward visual input under noise while retaining substantial audio reliance. Modality balance also varies during generation and with architecture, whereas temporal alignment remains robust and SNR outweighs recognition difficulty as a driver of weighting.
- Global SHAP: At −10 dB, most models reduce audio contributions to 39–46%, then increase them to 63–73% in clean speech, revealing visual adaptation with persistent audio reliance.Permutation and Sampling SHAP produce similar results, supporting stable attributions with M = 2000 coalitions; subsequent analyses use the faster Permutation SHAP.
- Global SHAP: Whisper-Flamingo and AV-HuBERT show the widest adaptation ranges at approximately 30–34 points, while LRS2 reveals stronger Whisper-Flamingo audio bias at 88–95%.On LRS2, Llama-AVSR and Omni-AVSR decrease from 65.4% and 59.9% in clean conditions to 46.0% and 42.7% at −10 dB, whereas Auto-AVSR remains near 56%.
- Generative SHAP: During generation, Whisper-Flamingo and Omni-AVSR increase clean-condition audio reliance from 65% to 71% and 63% to 72%, while noisy decoding produces U-shaped trajectories.AV-HuBERT remains stable in clean and noisy conditions, with less than 3 points of variation throughout generation.
- Temporal Alignment SHAP: Audio and visual features preserve temporal correspondence with output tokens under noise, with diagonal alignment scores of 2.90 in clean speech and 1.70 when noisy.Early, middle, and late feature groups peak at corresponding generation stages for both modalities.
- Noise-Dependent Adaptation: Noise type and severity govern visual shifting: music and sound noise yield WERs of 2.6–7.8, versus 26.3–40.9 for babble noise at −10 dB.These noise types induce a smaller visual shift than babble noise, while Omni-AVSR remains reasonably robust despite training only on babble noise.
- Duration and Difficulty Effects: Modality balance is architecture-dependent across duration and remains stable across WER bins within each SNR condition, indicating acoustic conditions rather than recognition difficulty determine weighting.Whisper-Flamingo’s audio contribution falls from 75% to 69% clean and 42% to 33% noisy; Omni-AVSR remains around 45–46% at −10 dB and 60–65% at 2.5 dB.
5. Conclusion
Dr. SHAP-AV uses Shapley values to analyze modality contributions in AVSR across six state-of-the-art models. The findings show increased visual reliance under noise, persistent audio contributions, evolving modality balance during generation, architecture-dependent duration sensitivity, and preserved temporal alignment.
- Framework: Dr. SHAP-AV analyzes AVSR modality contributions using Shapley values across six state-of-the-art models.The framework is designed to characterize how models use acoustic and visual information.
- Modality contributions: Models shift toward visual reliance under noise while maintaining surprisingly high audio contributions under severe degradation.This reveals that visual reliance increases without eliminating substantial audio dependence.
- Contribution dynamics: Modality balance evolves during token generation, with architecture-dependent sensitivity to utterance duration.The contribution dynamics vary according to both decoding stage and model architecture.
- Temporal alignment: Models preserve temporal alignment between inputs and outputs.This alignment remains part of the reported cross-model findings.