PaperlayerView exact revision on arXiv

Source-linked AI summary

AMRD: Adaptive Multi-Teacher Relational Distillation for Lightweight Speech Emotion Recognition

Yuqi Li, Yi-Cheng Lin, Xianglong Wang, Kuo Yang, Xiaoqin Feng, Yixuan Wang, Huiran Duan, Yingli Tian

arXiv:2607.25289v1cs.LG

TL;DR

Lightweight speech emotion recognition needs distillation methods that handle changing teacher reliability and preserve relationships among samples. AMRD adaptively weights multiple teachers and distills relational structure, matching or exceeding the best single-teacher baseline in seven of eight student–dataset settings, with gains up to 2.9% accuracy.

  • Problem

    Existing multi-teacher distillation for speech emotion recognition does not adequately address batch-varying teacher reliability or the loss of inter-sample relational structure in logit-level transfer.

  • Method

    AMRD combines one-class-SVM-based per-batch teacher weighting with relational similarity matrix distillation between teacher and student representations.

  • Results

    AMRD matches or exceeds the best single-teacher baseline in seven of eight student–dataset settings across four architectures, with gains of up to 2.9% accuracy.

  • Takeaways & Limitations

    Adaptive multi-teacher distillation supports lightweight on-device emotion recognition under extreme compression, with both mechanisms operating during training without added inference cost.

  • Takeaways & Limitations

    Evaluation is limited to in-corpus, audio-only experiments, leaving cross-corpus, cross-language, and multimodal generalization unvalidated.

Abstract

from arXiv · show

On-device speech emotion recognition (SER) is critical for real-time applications, yet large self-supervised models that excel at SER are too costly for edge devices. Multi-teacher knowledge distillation can compress them into a lightweight student, but two challenges remain: teacher reliability varies across batches, and logit-level distillation ignores inter-sample relational structure. We propose Adaptive Multi-teacher Relational Distillation (AMRD) to address both. A one-class SVM on each teacher's logit similarity matrix assigns per-batch weights favoring more coherent teachers. A relational distillation loss aligns teacher and student similarity matrices, capturing structure that logit matching misses. On IEMOCAP and CREMA-D datasets across four student architectures, AMRD outperforms single-teacher distillation baselines in most settings, and ablations confirm both components yield complementary gains.

I. INTRODUCTION

AMRD addresses the deployment and modeling challenges of multi-teacher SER distillation by adapting teacher weights per batch and aligning inter-sample relational structure. Across IEMOCAP and CREMA-D, it matches or exceeds the best single-teacher baseline in seven of eight settings, with gains up to 2.9% accuracy.

  • Motivation: SSL models such as WavLM and data2vec improve SER but their approximately 94 M parameters make edge deployment impractical.Knowledge distillation trains a compact student to mimic a large teacher, while multi-teacher distillation combines complementary teacher knowledge.
  • Challenges: Teacher reliability varies across samples and batches as speaker identity, emotional intensity, and recording environment change, limiting uniform or fixed aggregation weights.The challenge arises from differences in teacher architecture and pre-training objective, alongside SER’s subjective and variable data conditions.
  • Method: AMRD uses a one-class SVM to estimate each teacher’s per-batch reliability from pairwise logit structure and assign higher weights to more coherent predictions.This adaptive aggregation targets reliability shifts caused by emotion subjectivity and speaker diversity.
  • Method: AMRD introduces RSMD, a feature-level relational loss that transfers inter-sample similarity structure from multiple teachers to the student.The loss complements logit-level distillation by preserving relational information among samples.
  • Results: Seven of eight student–dataset settings match or exceed the best single-teacher baseline, with gains of up to 2.9% accuracy across IEMOCAP and CREMA-D.The evaluation spans four student architectures.

II. RELATED WORK · A. Feature-based and relational KD. · B. Multi-teacher KD for SER.

Prior knowledge-distillation work progresses from logit and point-wise feature matching toward relational objectives, while multi-teacher SER methods still face limitations in adapting teacher contributions to batch-level reliability.

  • A. Feature-based and relational KD.: DKD and CKD refine soft-target transfer at the logit level but do not capture inter-sample structure.
  • A. Feature-based and relational KD.: FitNets introduced point-wise intermediate feature matching, which also ignores relational structure.
  • A. Feature-based and relational KD.: SPKD, RKD, PKT, and CCKD transfer relational information through Gram-matrix, distance-angle, similarity-distribution, and correlation objectives.The passage specifies SPKD matching activation Gram matrices, RKD penalizing distance and angle discrepancies, PKT matching pairwise-similarity distributions via KL divergence, and CCKD aligning correlation structure.
  • B. Multi-teacher KD for SER.: Multi-teacher KD aggregates complementary knowledge from several teachers, but most methods use uniform or fixed weights.
  • B. Multi-teacher KD for SER.: Uniform or fixed teacher weights cannot adapt to per-batch reliability shifts.
  • B. Multi-teacher KD for SER.: Bijoy et al. proposed cosine-similarity routing for multilingual emotion recognition, weighting each teacher per sample by cosine similarity between its logits and the student’s.
  • B. Multi-teacher KD for SER.: Because routing selects the teacher already aligned with the student, it may reinforce existing predictions instead of prioritizing the most reliable source.

III. PROPOSED METHOD · A. SVM-Based Dynamic Teacher Weighting

AMRD distills knowledge from two frozen self-supervised teachers into a lightweight student using batch-adaptive teacher weighting. It evaluates teacher coherence through batch logit structure with a one-class SVM, then uses the resulting quality scores for aggregation.

  • III. PROPOSED METHOD: AMRD distills knowledge from M=2 frozen SSL teachers, data2vec and WavLM, into a lightweight student with trainable parameters θ.Each teacher produces classification logits and pre-classifier features for every utterance.
  • A. SVM-Based Dynamic Teacher Weighting: Teacher reliability shifts across batches, so AMRD uses adaptive aggregation rather than fixed teacher weights.Per-sample confidence measures cannot capture inconsistent logit patterns across samples in the same batch.
  • A. SVM-Based Dynamic Teacher Weighting: AMRD ℓ2-normalizes each teacher’s batch logits and computes a cosine similarity matrix to assess pairwise prediction structure.The resulting matrix has dimensions B×B for a batch of B utterances.
  • A. SVM-Based Dynamic Teacher Weighting: A one-class SVM uses each teacher’s similarity matrix G_m as a precomputed kernel and converts mean signed distance into a scalar quality score q_m.The SVM learns a compact boundary around the majority of data points.
  • A. SVM-Based Dynamic Teacher Weighting: Coherent teacher logits form a tight cluster and yield higher q_m, whereas scattered or inconsistent predictions lower q_m.The hyperparameter ν controls the upper bound on the outlier fraction and is fixed at ν=0.1.
  • A. SVM-Based Dynamic Teacher Weighting: Teacher aggregation weights are obtained by applying a softmax to the quality scores q_m.This produces normalized weights that favor teachers with higher batch-level quality scores.
  • A. SVM-Based Dynamic Teacher Weighting: Each teacher’s logits are softened with a shared temperature T before forming the aggregated soft-target distribution.This step combines the dynamically weighted teacher predictions for distillation.
  • A. SVM-Based Dynamic Teacher Weighting: With B=16, fitting the B×B SVM matrix adds negligible training overhead and is unnecessary during inference.The weighting procedure is therefore applied during training without inference-time SVM computation.

B. Relational Similarity Matrix Distillation

Relational Similarity Matrix Distillation preserves inter-sample structure that logit-level distillation misses by matching teacher and student pairwise similarity matrices. It uses normalized features and uniformly averages relational losses across teachers, rather than reusing logit-based SVM weights.

  • Relational Similarity Matrix Construction: RSMD matches teacher and student pairwise cosine similarity matrices to preserve inter-sample feature-space relationships beyond per-sample class distributions.The loss computes a B×B relational similarity matrix for each batch from teacher and student features.
  • Relational Similarity Matrix Construction: RSM construction avoids direct feature-vector comparison when teacher and student dimensions and distributions differ.Pairwise similarities capture sample proximity regardless of the underlying feature dimension.
  • Relational Similarity Matrix Construction: Z-score normalization followed by ℓ2-normalization makes similarity values comparable despite differences in feature magnitude and distribution.The z-score step centers and scales each sample’s feature vector across its dimensions before ℓ2-normalization.
  • RSMD Loss: Minimizing the per-teacher MSE aligns student and teacher RSMs while penalizing uniformly high pairwise similarities among student representations.The Frobenius-norm expansion shows alignment with teacher similarity patterns and regularization of student similarities.
  • Teacher Averaging: Relational losses use uniform teacher averaging rather than SVM-based logit weights because logit coherence does not necessarily indicate feature-level relational quality.The paper reports that uniform averaging yields stable results across all settings and decouples the two weighting mechanisms.

C. Training Objective

The training objective combines hard-label cross-entropy, temperature-scaled KL divergence on aggregated soft targets, and relational similarity-matrix distillation. Fixed coefficients and frozen teachers preserve SVM-based per-batch adaptivity while only student parameters are updated.

  • Loss composition: The overall loss combines cross-entropy on hard labels, temperature-scaled KL divergence on aggregated soft targets, and RSMD.L = αce LCE(s, y) + αkd T^2 KL(log σ(s/T) || pT) + LRSMD.
  • Loss composition: The T^2 factor compensates for gradient-magnitude reduction caused by temperature scaling.σ denotes softmax in the objective.
  • Parameter updates: The coefficients αce and αkd remain fixed because SVM-based weights provide per-batch adaptivity at the soft-target level.This avoids introducing additional coefficient adaptation during training.
  • Parameter updates: Only student parameters θ are updated, while all teachers remain frozen.The aggregation weights wm are fixed during backpropagation because SVM fitting is non-differentiable.

IV. EXPERIMENTS · A. Setup

The experiments use IEMOCAP and CREMA-D to evaluate SER models under established cross-validation protocols, with two fully fine-tuned self-supervised teachers and four lightweight student architectures.

  • A. Setup: The evaluation uses two widely used speech emotion recognition benchmarks: IEMOCAP and CREMA-D.IEMOCAP contains approximately 12 hours of audiovisual recordings, while CREMA-D consists of 7,442 clips from 91 actors.
  • A. Setup: IEMOCAP covers 10 speakers across 5 sessions and approximately 12 hours of audiovisual recordings.The evaluation selects angry, happy, neutral, and sad, merging excited into happy.
  • A. Setup: IEMOCAP evaluation follows leave-one-session-out 5-fold evaluation, with each session serving as the test set in turn.This protocol follows prior work cited in the supplied passage.
  • A. Setup: Two teacher SER models append a single linear classifier to data2vec Large and WavLM Base+.Both teachers are fully fine-tuned per fold, and the best checkpoint is selected by validation unweighted accuracy (UA).

B. Main Results

AMRD achieves the strongest overall results across lightweight students, leading on all IEMOCAP students and three of four CREMA-D students. Its gains persist under extreme compression, although benefits vary by student, dataset, and teacher.

  • The reported comparisons use WavLM or data2vec as single teachers, while AMRD combines both teachers.Tables I and II report WA and UA on IEMOCAP and CREMA-D, respectively.
  • AMRD achieves the highest WA and UA for all four students on IEMOCAP.For MNv3, AMRD reaches 47.21% UA, 2.9 points above WavLM+DKD.
  • On CREMA-D, AMRD leads for three of four students, with LSP+ reaching 56.37% UA, 1.7 points above D2V+KD.For MNv3, AMRD reaches 36.09% UA, below the no-distillation baseline of 36.39%.
  • Teacher effectiveness varies by student and dataset: data2vec is stronger for IEMOCAP LSP+, whereas WavLM is stronger for MNv3; data2vec generally leads on CREMA-D.AMRD still surpasses the best data2vec-based result for three of four CREMA-D students, indicating a contribution from the secondary teacher.
  • LSP+ has 0.78 M parameters, over 120× fewer than WavLM Base+, yet AMRD reaches 52.30% UA on IEMOCAP and 56.37% on CREMA-D.These results improve over no distillation by 2.8 and 3.4 points, respectively.

C. Ablation Study

The ablation study shows that SVM-based teacher weighting and RSMD each independently improve the uniform-weight, no-RSMD baseline, with larger gains on CREMA-D than IEMOCAP.

  • Component Contributions: Each AMRD component independently improves the uniform-weight, no-RSMD baseline across all reported settings.The study evaluates SVM-based weighting and RSMD separately using LSP+ and EB0 as representative student capacities.
  • IEMOCAP Ablation: 0.4–0.7 points: SVM weighting alone raises UA on IEMOCAP.RSMD alone adds 0.2–0.3 points on IEMOCAP.
  • CREMA-D Ablation: 1.8 points: SVM weighting boosts EB0 UA on CREMA-D from 39.75% to 41.51%.SVM weighting also raises LSP+ UA by 1.2 points on CREMA-D.

D. Hyperparameter Sensitivity · V. CONCLUSION

AMRD’s relational supervision is beneficial and stable across moderate geometric-loss weights, leading to fixed defaults for the main experiments. The framework combines adaptive teacher weighting with relational distillation, adds no inference cost, and matches or exceeds the better single-teacher baseline in seven of eight settings while motivating further robustness, transfer, and fairness work.

  • D. Hyperparameter Sensitivity: Performance remains stable for moderate relational-loss weights αgeom ∈[0.05, 0.2].Removing the relational term (αgeom=0) reduces UA in three of four settings.
  • D. Hyperparameter Sensitivity: Removing relational supervision reduces UA in three of four settings, confirming complementary supervision beyond logit-level distillation.LSP+ on IEMOCAP slightly favors αgeom=0.05.
  • D. Hyperparameter Sensitivity: The main experiments use αgeom=0.1, T=4, and ν=0.1, with performance stable around the defaults T=4 and ν=0.1.These defaults follow the reported sensitivity analysis.
  • V. CONCLUSION: AMRD combines SVM-based dynamic teacher weighting with RSMD, aligning pairwise similarity structure between teacher and student feature spaces.Both mechanisms operate only during training.
  • V. CONCLUSION: Both AMRD mechanisms add no inference cost, preserving the framework’s lightweight deployment objective.The added operations are training-only.
  • V. CONCLUSION: Seven of eight student–dataset settings match or exceed the better single-teacher baseline on IEMOCAP and CREMA-D.This is the aggregate conclusion reported for AMRD’s evaluation.
  • V. CONCLUSION: Future work includes scaling beyond two teachers, noise-robust training, and cross-corpus transfer evaluation.The conclusion also calls for systematic fairness evaluation across speaker demographics because distillation can amplify or suppress teacher biases.

VI. LIMITATIONS · VII. GENERATIVE AI USE DISCLOSURE

The study’s limitations concern its two-teacher design and audio-only input, while its generative AI disclosure restricts AI assistance to linguistic polishing rather than substantive scientific work.

  • VI. LIMITATIONS: All experiments use two teachers, although SVM weighting and RSMD are defined for arbitrary M.Scaling beyond two teachers introduces additional hyperparameter choices.
  • VI. LIMITATIONS: More teachers require choices for teacher selection and per-teacher loss weighting.These added choices complicate scaling the method to larger teacher pools.
  • VI. LIMITATIONS: Training cost increases linearly in M as the number of teachers grows.The stated scaling concern applies when extending beyond the two-teacher experiments.
  • VI. LIMITATIONS: Validating AMRD with a larger and more diverse teacher pool remains future work.The paper does not report such validation.
  • VI. LIMITATIONS: AMRD processes speech only, despite emotion also being conveyed through facial expression and linguistic content.Audio-only processing can leave some emotional ambiguities unresolved.
  • VI. LIMITATIONS: Multimodal fusion could resolve ambiguities that are irrecoverable from audio alone.This identifies a potential direction beyond the current audio-only framework.
  • VI. LIMITATIONS: Extending AMRD to multimodal inputs requires aligned multi-channel corpora with per-modality teachers.The paper characterizes this extension as a different research program.
  • VII. GENERATIVE AI USE DISCLOSURE: Generative AI tools assisted only with linguistic polishing of the manuscript.The authors retain sole responsibility for the research design, experiments, analysis, and reported results, and AI tools did not contribute to substantive scientific content.
Loading 2607.25289v1…