PaperlayerView exact revision on arXiv

Source-linked AI summary

MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition

Umberto Cappellazzo, Minsu Kim, Pingchuan Ma, Honglie Chen, Xubo Liu, Stavros Petridis, Maja Pantic

arXiv:2510.04136v1eess.AScs.CVcs.SD

TL;DR

Fixed-rate token compression limits inference-time tradeoffs between information fidelity and compute in AVSR. MoME combines sparse experts with Matryoshka representations in a frozen LLM, and outperforms prior models across AVSR, ASR, and VSR while using fewer parameters and training hours.

  • Problem

    Fixed-rate token compression yields one sequence and prevents dynamically balancing information fidelity and computational efficiency at inference time.

    Source: §1 · p. 2

  • Method

    MoME adds top-k routed and shared experts to a frozen LLM within the Matryoshka framework, sharing a router across scales and modalities.

    Source: §1 · p. 2

  • Results

    MoME outperforms prior Matryoshka-based and fixed-scale models across AVSR, ASR, and VSR on LRS2 and LRS3, while using significantly fewer parameters and training data hours.

    Source: §5 · p. 10

  • Takeaways & Limitations

    MoME supports resource-adaptive inference across token granularities while maintaining strong performance at high compression and in noisy conditions.

    Source: §5 · p. 10

    All experts must remain in memory, and inference cost is slightly higher than Llama-AVSR and Llama-MSTK.

    Source: G Limitations · p. 20

Abstract

from arXiv · show

Large language models (LLMs) have recently shown strong potential in audio-visual speech recognition (AVSR), but their high computational demands and sensitivity to token granularity limit their practicality in resource-constrained settings. Token compression methods can reduce inference cost, but they require fixing a compression rate in advance and produce a single fixed-length output, offering no flexibility to balance information density and efficiency at inference time. Matryoshka representation learning (MRL) addresses this by enabling a single model to operate across multiple token granularities, allowing compression rates to be adjusted dynamically. However, current MRL-based methods treat each scale independently during training, limiting cross-scale generalization, robustness at high compression, and interpretability. To overcome these limitations, we propose MoME (Mixture of Matryoshka Experts), a novel framework that integrates sparse Mixture-of-Experts (MoE) into MRL-based LLMs for AVSR. MoME augments a frozen LLM with top-k routed and shared experts, allowing dynamic capacity allocation across scales and modalities. A shared router promotes consistent expert activation across granularities, enabling compressed sequences to benefit from representations learned at lower compression. Experiments on LRS2 and LRS3 demonstrate that MoME achieves state-of-the-art performance across AVSR, ASR, and VSR tasks, while requiring significantly fewer parameters and maintaining robustness under noise. MoME unifies the adaptability of MRL with the efficiency of MoE, offering a scalable and interpretable solution for resource-aware speech recognition.

1 Introduction

AVSR can use visual speech cues to address acoustic noise, but long audio-visual inputs make LLM inference costly, and fixed-rate compression limits deployment flexibility. MoME combines sparse experts with Matryoshka representations to support scale-aware inference, with reported gains across recognition tasks and reduced resource use.

1 Introduction

  • AVSR incorporates visual cues such as lip movements, which remain unaffected by acoustic noise, to improve robustness and recognition accuracy in noisy conditions.This motivation is framed against ASR degradation in acoustically challenging environments.Source: §1 · p. 1
  • Fixed-rate compression produces one token sequence and cannot dynamically balance information fidelity against computational efficiency across inputs or deployment settings.Audio-visual speech inputs are long and temporally dense, increasing token counts and LLM compute demands.Source: §1 · p. 2
  • Existing Matryoshka models train each scale independently, limiting cross-scale interaction, high-compression performance, and analysis of audio-visual token interactions.The section frames these gaps as constraints on generality, specialization, interpretability, and adaptability.Source: §1 · p. 2
  • MoME adds top-k routed and shared experts to a frozen LLM, with a shared router that aligns expert activation across scales and modalities.This lets more-compressed sequences use expert pathways shaped by lower-compression sequences.Source: §1 · p. 2
  • MoME outperforms prior Matryoshka-based and independently trained models across AVSR, ASR, and VSR; Figure 1 reports parity or better with fewer activated parameters and training hours on LRS3.The framework also reports strong robustness in noisy scenarios.Source: §1 · p. 2, Figure 1 · p. 2

2 Related Work

Related work covers sparse expert routing, multigranular representations, and LLM-based speech recognition. MoME brings these strands together by combining Matryoshka representations with sparse experts for AVSR.

2.1 Mixture of Experts

  • MoE layers replace a standard FFN with multiple experts and sparse routing, activating only a subset per input; prior work also uses MoE for parameter-efficient fine-tuning of frozen backbones.Audio-visual MoE work has explored scaling capacity and processing audio and visual data.Source: §2.1 · p. 3

2.2 Matryoshka Representation Learning

  • MRL encodes information at multiple granularities in one representation, enabling token-level adaptive deployment under varying computational constraints.Prior vision-language and audio-visual LLMs extend this hierarchy to token sequences.Source: §2.2 · p. 3

2.3 Audio-Visual Speech Recognition

  • Prior LLM-based AVSR methods use fixed-rate compression, MRL, or speaking-rate-based adaptive compression, while MoME combines MRL with sparse MoE layers in a frozen LLM.Llama-AVSR and MMS-Llama require retraining for specific token rates; MoME reuses expert activations across granularities and adds shared experts.Source: §2.3 · p. 3
  • Figure 2 depicts multiscale audio-visual tokens entering MoME, where a top-k router selects routed experts and shared experts capture scale-invariant knowledge.The module can be placed parallel to multiple components within an LLM layer.Source: Figure 2 · p. 4

3 Methodology

The method forms audio-visual token sequences at multiple compression rates, then processes them through MoME modules alongside a frozen LLM. Shared routing and experts support cross-scale reuse, while training averages transcription losses across scales and includes a load-balancing term.

3.1 Audio-Visual Matryoshka Token Sequences

  • Audio and video encoders produce modality-specific tokens, which are projected and combined at every audio-video compression-rate pairing to form multigranular sequences.Text instructions and ground-truth tokens are appended to each sequence; inference granularity can be adjusted to requirements.Source: §3.1 · p. 4

3.2 Mixture of Matryoshka Experts (MoME) Module

  • MoME uses bottleneck experts in parallel with a frozen LLM, sharing experts and the router across scales so compressed inputs can reuse patterns learned from richer representations.Routed experts specialize while always-active shared experts capture global, cross-scale knowledge.Source: §3.2 · p. 5
  • Each token is processed by shared experts and only its top-K affinity-scored routed experts, keeping the routed computation sparse.The router scores token-to-expert affinities; only K out of Nr routed experts receive nonzero gate values.Source: §3.2 · p. 5
  • MoME can be inserted parallel to the MHSA module, FFN, or whole Transformer layer; the described formulation places it parallel to the FFN.In that placement, the layer combines FFN and MoME outputs with the residual hidden state.Source: §3.2 · p. 5
  • Training averages autoregressive next-token prediction losses across audio-visual scales and adds a load-balancing loss to reduce router collapse.The scale weights cij are set to 1, and the load-balancing loss is scaled by 0.01.Source: §3.2 · p. 6

4 Experiments

Experiments compare MoME across AVSR, ASR, and VSR settings, then examine expert choices, cross-scale behavior, and computational cost. MoME is reported to outperform comparison methods while supporting inference at different token granularities.

4.1 Experiment Settings

  • Experiments use LRS2 and LRS3 for AVSR, and LRS3 for ASR and VSR, with coarse-to-fine compression rates and average pooling.LRS2 contains 225 hours of video clips and LRS3 contains 433 hours of transcribed English video clips.Source: §4.1 · p. 6

4.2 Main Results

  • MoME variants outperform the reported AVSR baselines across compression rates on LRS2 and LRS3, with sparse activation reducing active parameters.The comparison includes fixed-rate and Matryoshka methods; the table’s active parameter count excludes shared projectors.Source: Table 1 · p. 7, §4.2 · p. 7
  • On LRS3, MoME achieves competitive WER at two audio-visual token-rate settings while activating fewer parameters and requiring fewer training-data hours under one suite of weights.The comparison uses Llama 3.1-8B and Whisper-medium.Source: §4.2 · p. 7
  • For VSR on LRS3, FFN and LAYER variants show consistent gains across scales; the study also evaluates ASR with single-modality inputs.Figure 3 presents MoME-23/4 results for both VSR and ASR.Source: §4.2 · pp. 7–8, Figure 3 · p. 7

4.3 Ablation Studies

  • Adding a shared expert consistently reduces WER across compression rates, while a second shared expert can degrade performance at rate (16,5) and increase active parameters.The authors use one shared expert in the main experiments.Source: §4.3 · p. 8, Table 3 · p. 8
  • Increasing top-k yields moderate gains, especially at higher compression, but increases computation; k of 1 or 2 is the stated efficiency–performance trade-off.The ablation fixes eight routed experts and uses one shared expert.Source: §4.3 · p. 8, Table 4 · p. 10

4.4 Visualization Analysis

  • Across scales, tokens show strong similarity, with each highly compressed token correlating mainly with about two to three less-compressed tokens; expert activation also aligns across scales but varies by layer.The visualizations cover token similarity and layer-wise expert activations.Source: Figure 4 · p. 8, §4.4 · p. 9, Figure 5 · p. 9

4.5 Computation Cost Analysis

  • At rates (16,5), MoME reduces processing from 1,673 to 184 tokens and TFLOPs by up to 8x with modest WER increase; inference time falls to 6.74 seconds from 12.75 seconds for 23 seconds of speech.The inference-time comparison uses MoME-15/3-MHSA on an NVIDIA L40 46GB GPU.Source: §4.5 · p. 10, Table 5 · p. 10

5 Conclusion

MoME combines Matryoshka representation learning with Mixture-of-Experts to support resource-adaptive inference across token granularities. The conclusion reports strong performance across speech-recognition tasks, parameter efficiency, noise robustness, and cross-scale and cross-modal analyses.

5 Conclusion

  • MoME outperforms prior Matryoshka-based and fixed-scale models on AVSR, ASR, and VSR across LRS2 and LRS3 while using fewer parameters and training data hours.The conclusion also reports robustness in noisy conditions.Source: §5 · p. 10
  • A shared router and shared experts promote expert alignment across scales, letting compressed sequences benefit from richer representations.The framework combines MRL with MoE for resource-adaptive inference across token granularities.Source: §5 · p. 10
  • MoME supports extremely parameter-efficient fine-tuning and demonstrates robustness in noisy conditions.Source: §5 · p. 10
  • Ablation and visualization analyses support MoME’s capacity to capture cross-scale and cross-modal interactions.Source: §5 · p. 10
  • The authors expect MoME to extend to other multimodal domains, including vision-language tasks, with minimal adaptation.Source: §5 · p. 10

A.1 Data Licenses

The appendix states the usage licenses for LRS2 and LRS3 and describes the visual and audio preprocessing applied to the datasets.

A.1 Data Licenses

  • LRS3 is licensed under CC BY-NC-ND 4.0, while LRS2 permits academic, non-commercial research.Source: §A.1 · p. 17

A.2 Pre-Processing

  • Video preprocessing crops 96 × 96 mouth-region ROIs and normalizes each frame using the training-set mean and standard deviation.Source: §A.2 · p. 17
  • Audio preprocessing applies z-normalization separately to each utterance.Source: §A.2 · p. 17

A.3 Training Details

The appendix describes training augmentations and optimization, decoding settings, and three possible insertion locations for MoME within the LLM.

A.3 Training Details

  • Training uses AdamW for 10 epochs with cosine annealing, weight decay 0.1, and NVIDIA H200 GPUs.The learning rate is 1e-3 for ASR and AVSR and 5e-4 for VSR.Source: §A.3 · p. 17
  • Training adds NOISEX bubble noise sampled uniformly from [-5, 0, 5, 10, 15, 20, ∞] dB and uses prompts for speech, video, or both.The prompt template is “Transcribe {task_prompt} to text.”Source: §A.3 · p. 17

A.4 Inference Details

  • Decoding uses beam search with beam width 15 and temperature 0.6.Source: §A.4 · p. 17

B MoME’s Insertion Variants

C Additional AVSR Experiments

Additional experiments compare MoME configurations across compression scales on LRS2 and LRS3; the reported results show consistent advantages for configurations with 14.1M parameters.

C Additional AVSR Experiments

C.1 Additional Ablations on MoME

The ablations favor a shared router and uniform scale weights. Additional LRS2 visualizations show cross-scale, cross-modal token correlations and consistent expert selection across granularities.

C.1 Additional Ablations on MoME

  • Separate scale-specific routers degrade performance more at higher compression, while the shared router promotes cross-scale alignment.Table 7 covers the disjoint-router ablation on LRS2 and LRS3 for MoME-15/3-LAYER.Source: §C.1 · p. 18, Table 7 · p. 19
  • Non-uniform weights do not improve lower-scale performance and degrade results at higher scales, so the main experiments use uniform weights.The tested NUW coefficients are [1, 1, 1.5, 2]; uniform weighting sets c_i,j = 1 across scales.Source: §C.1 · p. 18, Table 7 · p. 19

D Additional Visualization Analyses

D.1 Additional Cross-Modal Matryoshka Token Analyses

  • For “it’s a long way from home” on LRS2, MoME-15/3 shows strong linear correlations across token granularities and modalities.The reported cosine similarities are high along the diagonal; shared routing and experts encourage lower-granularity tokens to attend to specific higher-granularity sequence portions.Source: §D.1 · pp. 18–19, Figure 8 · p. 19

D.2 Additional Expert Activation Analysis

MoME activates similar experts across scales at a given layer while varying expert pools across layers. Its sparse routing entails modest inference-time overhead in the reported comparison.

D.2 Additional Expert Activation Analysis

  • MoME-23/4-FFN activates the same experts at a given layer across four scales, but different expert pools from layer to layer.The LRS2 analysis suggests this pattern also holds across configurations, including 15 versus 23 experts and MHSA versus FFN.Source: §D.2 · p. 19, Figure 9 · p. 20

E Inference Cost Comparison

  • MoME-15/3-MHSA has around 1.25x inference time and lower TPS, yet significantly outperforms Llama-AVSR and Llama-MTSK in the reported comparison.The comparison averages four speech inputs at audio/visual compression rate (4,2); router overhead and expert dispatching latency are identified as the main sources of increased time.Source: E Inference Cost Comparison · p. 19, Table 8 · p. 20
  • The authors prioritize memory reduction at high compression, noise robustness, and interpretability over faster inference.They regard the moderate inference overhead as justified for memory-constrained or multi-resolution scenarios, while noting router execution as an optimization avenue.Source: E Inference Cost Comparison · pp. 19–20

F MoME’s Extension to Other Multimodal Tasks

MoME’s extension to other multimodal tasks depends on a frozen pretrained backbone and inputs whose token granularity affects performance and efficiency. The authors describe image–text retrieval as one possible application.

F MoME’s Extension to Other Multimodal Tasks

  • Applying MoME beyond AVSR requires a pretrained encoder or LLM backbone that can remain frozen during fine-tuning.The other stated condition is that input token granularity must affect performance and efficiency, allowing compression.Source: F MoME’s Extension to Other Multimodal… · p. 20
  • For image–text retrieval, the authors propose using CLIP image tokens compressed by average pooling or another method, with a MoME layer parallel to transformer layers.This design is described as enabling retrieval at different resource constraints.Source: F MoME’s Extension to Other Multimodal… · p. 20
  • The proposed extension trains across token compression rates using the task-specific loss, such as contrastive loss.This follows the paper’s use of multiple audio-visual granularities in AVSR.Source: F MoME’s Extension to Other Multimodal… · p. 20

G Limitations

MoME retains the memory overhead of keeping all MoE experts resident and has slightly higher inference cost than the named baselines. The paper focuses on audio-visual models, and its use of pretrained LLMs carries potential accuracy and bias concerns.

G Limitations

  • All MoME experts must reside in memory despite sparse activation, increasing memory usage; bottleneck-designed experts keep the overall footprint modest.The authors also report slightly higher inference cost than Llama-AVSR and Llama-MSTK.Source: G Limitations · p. 20
  • The paper evaluates audio-visual models; effectiveness in broader domains such as vision-language modeling remains future work.The authors describe integration into other LLMs and tasks as feasible, but call for exploring effectiveness in those domains.Source: G Limitations · pp. 20–21

H Societal Impacts

  • Because MoME uses pretrained LLMs, it may generate inaccurate or biased outputs; the authors recommend safety and fairness evaluations before deployment.Source: H Societal Impacts · p. 21
Loading 2510.04136v1…