Source-linked AI summary
UniAE-MoE: A Unified Audio Encoder via Mixture of Experts
Shengbo Cai, Zhisheng Zhang, Zichao Nie, Jing Peng, Jingran Xie, Zhiyong Wu
TL;DR
A single audio encoder can struggle to preserve both fine-grained acoustic detail and high-level semantics across diverse tasks. UniAE-MoE fuses Qwen2-Audio and Audio-Flamingo 3 with a shared-expert MoE and task-specific training, achieving a 0.802 XARES-LLM score and top-tier performance in the Interspeech 2026 challenge.
Problem
A single encoder can struggle to preserve fine-grained acoustic details and high-level semantics across speech recognition, audio classification, and captioning.
Source: §2.2 · p. 3Method
UniAE-MoE fuses Qwen2-Audio and Audio-Flamingo 3 with a SwiGLU MoE and shared expert, then uses two-stage instruction tuning and task-specific data scaling.
Source: §1 · p. 2Results
UniAE-MoE achieves a 0.802 XARES-LLM score and state-of-the-art performance, with top-tier results in the Interspeech 2026 challenge.
Source: §1 · p. 2, §4 · p. 11Takeaways & Limitations
The reported challenge results support robust generalization across diverse audio tasks.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Large Audio Language Models (LALMs) rely on effective audio encoders for multi-task performance. We introduce UniAE-MoE, a unified audio encoder designed to model cross-domain audio representations and achieve outstanding downstream understanding performance via a Mixture-of-Experts (MoE) architecture. Specifically, we explore mainstream audio encoders and integrate those from Qwen2-Audio and Audio-Flamingo 3, which demonstrate superior downstream capabilities. To facilitate effective model fusion, we improve our encoder using SwiGLU with shared experts to decouple encoder networks, and we further introduce a two-stage instruction-tuning strategy to better adapt the model to diverse downstream tasks. Moreover, we propose the task-specific data scaling (TSDS) technique to enhance UniAE-MoE's understanding capabilities. On the XARES-LLM benchmark, UniAE-MoE attains a score of 0.802, achieving state-of-the-art performance. It also delivers top-tier performance in the official Interspeech 2026 Audio Encoder Capability Challenge, further demonstrating robust generalization across diverse audio tasks. Together, these results validate the effectiveness of UniAE-MoE for unified audio understanding across speech, music, and general audio domains.
1 Introduction
The introduction frames unified audio encoding as a response to heterogeneous tasks that strain monolithic encoders and straightforward fusion. UniAE-MoE combines two complementary encoders through MoE, unified instruction training, and task-specific data scaling, and reports leading benchmark results.
1 Introduction
- Monolithic encoders and dense shared adapters must model heterogeneous acoustic cues, while straightforward multi-encoder fusion can introduce redundancy and speech-recognition trade-offs.General-purpose perception encoders are not inherently aligned with generative audio-to-language inference and instruction following.Source: §1 · p. 2
- The method casts 20 downstream tasks as instruction-response generation and uses two-stage instruction tuning to support multi-task generalization.The training strategy moves from modality alignment to multi-task instruction fine-tuning.Source: §1 · p. 2
- UniAE-MoE fuses Qwen2-Audio and Audio-Flamingo 3 with a SwiGLU-based MoE and shared expert to decouple and combine cross-domain representations.The framework uses a two-stage strategy progressing from semantic alignment to multi-task instruction fine-tuning.Source: §1 · p. 2
- Task-Specific Data Scaling addresses cross-task data imbalance in the unified training setting.The authors introduce it because task data are unbalanced.Source: §1 · p. 2
- 0.802 overall score on XARES-LLM establishes state-of-the-art performance, while Interspeech 2026 results are top-tier across classification and understanding tasks.The paper also reports top-tier performance in the official Audio Encoder Capability Challenge.Source: §1 · p. 2
2 Methodology
UniAE-MoE combines complementary audio encoders with token-routed and shared experts, then trains the fused representations through a two-stage instruction-following setup. Task-specific data scaling selectively expands underrepresented tasks to address imbalances in the training data.
2.1 High-Performance Dual-Model Backbone
- Audio-Flamingo 3 and Qwen2-Audio score 0.767 and 0.737, respectively, and are selected as complementary backbones for unified audio modeling.Audio-Flamingo 3 is described as more balanced for long-form audio and environmental sounds, complementing Qwen2-Audio’s speech and music emphasis.Source: §2.1 · p. 3
2.2 Universal Audio Representation with MoE Fusion
- The MoE aligns and combines the two encoders’ features, then routes each token to top-K SwiGLU experts while a shared expert processes every token.The shared expert targets invariant acoustic features; the authors say it helps preserve model complementarity and mitigate expert collapse.Source: §2.2 · pp. 4–5
2.3 Multi-task Instruction Tuning
- Twenty downstream tasks are cast as autoregressive instruction-following; two training stages first align audio features, then fine-tune remaining components on diverse task instructions.Stage one uses ASR and audio captioning supervision while freezing the audio encoders and LLM; stage two keeps the audio encoders frozen.Source: §2.3 · pp. 5–6
2.4 Task-Specific Data Scaling for Training
- Task-Specific Data Scaling selectively augments underrepresented tasks with relevant public datasets to address cross-task data imbalance.Additions include captioning datasets and datasets for genre and emotion recognition; AudioSetCaps provides approximately 1.73M audio–text pairs.Source: §2.4 · pp. 6–7
3 Experimental Results and Analyses
The experiments compare audio encoders, fusion strategies, ablations, and data scaling, then assess challenge performance. UniAE-MoE scores 0.802 on XARES-LLM and shows strong, though not uniform, results across challenge tracks.
3.1 Experimental Setups
- Evaluation uses 20 public XARES-LLM tasks and macro-averages scores across datasets; challenge evaluation additionally includes hidden datasets.Preprocessing truncates clips to 30 seconds; training has two stages of 30,000 and 80,000 steps.Source: §3.1 · p. 7
3.2 Benchmarking Audio Encoder
- Audio-Flamingo 3 and Qwen2-Audio achieve the highest encoder scores, 0.767 and 0.737, respectively, motivating their complementary use.The benchmark compares representative encoders from speech, music, and general-audio domains.Source: §3.2 · p. 8, Table 3 · p. 8
3.3 MoE Makes Better Fusion
- With the optimal dual-model backbone, UniAE-MoE reaches 0.802, improving on the best Concat baseline of 0.778 on XARES-LLM and capturing synergies between complementary audio encoders.Source: §3.3 · pp. 8–9
3.4 Ablation Study
- Removing SwiGLU drops the score from 0.802 to 0.794; UniAE-MoE-Light uses 88.5 M parameters and scores 0.797.The ablations assess SwiGLU's contribution and parameter efficiency.Source: Table 5 · p. 9, §3.4 · p. 9
3.5 Data Scaling Strategies
- TSDS raises standard MoE from 0.759 to 0.794 and MoE-SwiGLU from 0.799 to 0.802 overall, while Free Music Archive rises from 0.848 to 0.979.Broad data scaling raises standard MoE from 0.759 to 0.782 but gives no corresponding gain for MoE-SwiGLU.Source: §3.5 · pp. 9–11, Table 6 · p. 10
3.6 Official Challenge Evaluation
- Track A results are top-tier on eight tasks; Track B is competitive overall, but lower hidden-test performance on AISHELL-6 and LibriHeavy remains a challenge.Track A covers classification and Track B audio understanding.Source: Table 7 · p. 10, §3.6 · p. 11
4 Conclusion
UniAE-MoE fuses Qwen2-Audio and Audio-Flamingo 3 with MoE, SwiGLU, and shared experts, and uses two-stage training with task-specific data scaling. It scores 0.802 on XARES-LLM and achieves top-tier Track A and strong Track B challenge performance.
4 Conclusion
- UniAE-MoE combines Qwen2-Audio and Audio-Flamingo 3 through MoE fusion, with SwiGLU, shared experts, two-stage training, and Task-Specific Data Scaling.The paper reports a 0.802 XARES-LLM score and top-tier overall Track A performance, including top-tier results on eight datasets.Source: §4 · p. 11