Source-linked AI summary
MI-Fuse: Label Fusion for Unsupervised Domain Adaptation with Closed-Source Large-Audio Language Model
Hsiao-Ying Huang, Yi-Cheng Lin, Hung-yi Lee
TL;DR
SER can degrade under domain mismatch when source data are unavailable and a strong LALM is accessible only through an API. MI-Fuse fuses predictions from an API-based LALM and a source-trained classifier to adapt a student, which outperforms the LALM and baselines across cross-domain transfers.
Problem
The paper asks whether a student can be adapted to outperform an API-only LALM using unlabeled target audio, when domain mismatch degrades SER and source data are unavailable.
Source: §1 · p. 1Method
MI-Fuse fuses multiple stochastic predictions from an API-based LALM and a source-trained SER classifier, weighting mean distributions by mutual-information-based uncertainty and stabilizing training with an EMA-updated teacher.
Source: §1 · p. 1Results
Across six transfer settings, MI-Fuse averages 58.38% unweighted accuracy, 3.9% higher than the best-performing baseline (LALM SFUDA), and student models outperform closed-source LALMs.
Source: §4.2 · p. 3, §6 · p. 4Takeaways & Limitations
MI-Fuse is presented as a practical recipe for deploying emotion-aware speech systems under realistic constraints.
Source: §6 · p. 4MI-Fuse assumes meaningful LALM emotion probabilities and a fixed discrete emotion-category set, while API cost, latency, and proprietary access may hinder deployment and differing label spaces may hinder applicability.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Large audio-language models (LALMs) show strong zero-shot ability on speech tasks, suggesting promise for speech emotion recognition (SER). However, SER in real-world deployments often fails under domain mismatch, where source data are unavailable and powerful LALMs are accessible only through an API. We ask: given only unlabeled target-domain audio and an API-only LALM, can a student model be adapted to outperform the LALM in the target domain? To this end, we propose MI-Fuse, a denoised label fusion framework that supplements the LALM with a source-domain trained SER classifier as an auxiliary teacher. The framework draws multiple stochastic predictions from both teachers, weights their mean distributions by mutual-information-based uncertainty, and stabilizes training with an exponential moving average teacher. Experiments across three public emotion datasets and six cross-domain transfers show consistent gains, with the student surpassing the LALM and outperforming the strongest baseline by 3.9%. This approach strengthens emotion-aware speech systems without sharing source data, enabling realistic adaptation.
1. INTRODUCTION
SER performance can degrade under domain mismatch, while source data may be unavailable and LALMs may be accessible only through closed APIs. MI-Fuse addresses this SFUDA setting by fusing LALM and source-classifier predictions to train a target student.
1. INTRODUCTION
- SER performance often degrades when training and deployment differ in corpus, speakers, channel or noise, or language.The paper frames this mismatch as a challenge for real-world SER deployment.Source: §1 · p. 1
- SFUDA adapts to an unlabeled target domain without source data, and the studied protocol further restricts the source LALM to API queries.The LALM's weights cannot be fine-tuned or inspected.Source: §1 · p. 1
- The central question is whether unlabeled target audio and an API-only LALM can support a student that outperforms the LALM in the target domain.The paper also aims to surpass the domain-specific teacher.Source: §1 · p. 1
- MI-Fuse fuses multiple stochastic predictions from the LALM and a source-trained SER classifier, weighting their mean distributions by mutual-information-based uncertainty.The classifier teacher is updated with an EMA of the student to smooth supervision across iterations.Source: §1 · p. 1
- The paper formalizes API-only LALM SFUDA, proposes denoised label fusion with an EMA-updated teacher, and reports better adaptation performance than existing SFUDA methods.The experiments span multiple SER datasets.Source: §1 · p. 1
2. RELATED WORK
Prior SFUDA work uses pseudo-labeling, consistency regularization, and clustering-based training to adapt with unlabeled target data. MI-Fuse adds general-domain LALM guidance alongside a source-domain teacher.
2. RELATED WORK
- SFUDA methods commonly adapt using pseudo-labeling, consistency regularization, or clustering-based training on unlabeled target data.Pseudo-label sources include class-centroid clustering, neighborhood aggregation, and label ensembling; methods may also use entropy minimization or information maximization.Source: §2 · pp. 1–2
- Unlike approaches relying solely on traditional SSL or self-training, MI-Fuse combines pseudo-labels from a general-domain LALM and a source-domain teacher for complementary supervision.This introduces LALM guidance during source-free target adaptation.Source: §2 · p. 2
3. METHODOLOGY
MI-Fuse adapts a student using unlabeled target audio and fused predictions from a source-trained classifier and an API-only LALM. It weights teacher predictions by uncertainty and stabilizes training with diversity regularization and an EMA teacher.
3.1. Problem formulation
- SFUDA adapts a student on shifted, unlabeled target audio without source data, where source-model pseudo-labels may be noisy.The source-trained classifier remains available, but the source dataset and target labels do not.Source: §3.1 · p. 2
3.2. Model uncertainty estimation
- Mutual information estimates epistemic uncertainty from disagreement across stochastic teacher predictions; high-MI teachers receive lower e^-MI weight.Stable, low-MI predictions are treated as more trustworthy.Source: §3.2 · p. 2
3.3. Label fusion
- Classifier and prompted LALM predictions are averaged across stochastic passes, then fused by MI-based weighted averaging to supervise a student initialized from the classifier.Experiments use 8 classifier passes and 5 LALM queries.Source: §3.3 · pp. 2–3
3.4. Diversity Loss and EMA Teacher Update
- To counter class collapse and unstable supervision, training combines cross-entropy to fused soft labels with batch-diversity regularization and an EMA-updated classifier teacher.The EMA momentum is α = 0.999.Source: §3.4 · p. 3
4. EXPERIMENTS
Experiments evaluate MI-Fuse across emotion datasets and cross-domain transfers, comparing it with zero-shot and adaptation baselines. Ablations and training curves examine fusion choices and training stability.
4.1. Setup
- The study evaluates MSP-Podcast, IMPROV, and IEMOCAP across four emotion categories using unweighted accuracy.Gemini 2.5 Flash serves as the LALM; WavLM base+ is used for the classifier teacher and student.Source: §4.1 · p. 3
4.2. Compare with other methods
- 58.38% unweighted accuracy averaged across six transfer settings, 3.9% higher than the best-performing baseline, LALM SFUDA.MI-Fuse achieved the highest accuracy in four of six transfer directions.Source: §4.2 · p. 3, Table 1 · p. 4
4.3. Ablation study
- On both IEMOCAP transfer directions, MI-Fuse with Multi, Direct Fusion, and MI achieves the best accuracy among the tested fusion alternatives.KL gating underperforms direct fusion, single-teacher training is weaker, and entropy weighting lags behind MI.Source: Table 2 · p. 4, §4.3 · p. 4
4.4. Training stability analysis
- On IMP → IEM fold 1, MI-Fuse has higher final development accuracy and more stable training than the classifier-teacher and LALM-teacher baselines.The classifier teacher declines after about 400 steps, attributed to overfitting on early EMA-teacher pseudo-labels; MI-Fuse improves throughout training.Source: Figure 2 · p. 4, §4.4 · p. 4
5. LIMITATION
MI-Fuse has practical and label-space constraints that bound where it can be applied.
5. LIMITATION
- MI-Fuse depends on LALMs that produce meaningful probabilistic predictions over emotion categories.Proprietary API reliance, inference cost, and latency may hinder use in resource-constrained or privacy-sensitive settings.Source: §5 · p. 4
- The fusion scheme assumes a fixed set of discrete emotion categories across datasets.Different target-domain taxonomies can hinder direct applicability.Source: §5 · p. 4
6. CONCLUSION
MI-Fuse is a label-fusion framework for source-free SER adaptation under closed-source LALM constraints. Experiments across multiple datasets report improved cross-domain performance and students outperforming closed-source LALMs on target domains.
6. CONCLUSION
- MI-Fuse addresses source-free SER adaptation under realistic closed-source LALM constraints.The framework integrates mutual-information-aware fusion, a diversity loss, and an EMA-updated teacher to produce stable pseudo-labels.Source: §6 · p. 4
- The framework balances the strengths of general-purpose LALMs and domain-trained classifiers.The conclusion attributes stable pseudo-labels to its combination of mutual-information-aware fusion, diversity loss, and an EMA-updated teacher.Source: §6 · p. 4
- Across multiple datasets, MI-Fuse consistently improves cross-domain performance, surpasses strong baselines, and enables students to outperform closed-source LALMs on target domains.Source: §6 · p. 4