Source-linked AI summary
ACR-Net: Mitigating Semantic Dominance via Contrastive Acoustic-Semantic Decoupling
Mengke Zhang, Yanda Shao, Tianhe Wu, Kai Feng
TL;DR
SER and Audio LLMs can fail when vocal tone contradicts text, and training on congruent data can foster rigid semantic priors. We introduce ASPIRE and diagnostic metrics, then ACR-Net explicitly decouples conflicting features; it achieves state-of-the-art ACC across adversarial and congruent datasets.
Problem
SER and multimodal models assume acoustic-semantic congruence, while standard datasets filter incongruent samples and congruent-data training can reinforce rigid semantic priors.
Source: §1 · p. 1Method
The paper introduces ASPIRE with SOP and LDD diagnostics, and ACR-Net uses Cross-Modal Attention and Contrastive Decoupling Loss to separate conflicting representations.
Source: §1 · pp. 1–2Results
ACR-Net achieves state-of-the-art Acoustic Accuracy across adversarial and congruent datasets while preserving acoustic fidelity and neutralizing semantic overconfidence.
Source: §5 · p. 4Takeaways & Limitations
ASPIRE, SOP, and LDD provide a benchmark and diagnostics for exposing semantic dominance and probing representation decoupling.
Source: §1 · pp. 1–2
Abstract
from arXiv · showhide
Speech Emotion Recognition (SER) models and Audio LLMs fail when vocal tone contradicts textual semantics. Extensive training on congruent data engenders a rigid semantic prior, causing "Semantic Dominance", a severe text bias during cross-modal conflicts. We introduce the ASPIRE benchmark of adversarial audio-text pairs across four conflict types, alongside two metrics: Semantic Overconfidence Penalty (SOP) and Latent Decoupling Degree (LDD). Unlike fusion methods that merely re-weight collapsed features, our Acoustic Conflict Resolution Network (ACR-Net) uses Cross-Modal Attention and Contrastive Decoupling Loss to disentangle contradictory representations into orthogonal latent spaces. Experiments show existing models fail under polarity conflicts due to extreme semantic overconfidence. Conversely, ACR-Net preserves acoustic fidelity, maintaining high LDD to drive SOP near zero, achieving superior Acoustic Accuracy (ACC) in both adversarial and standard scenarios.
1. Introduction
The introduction frames cross-modal emotional conflict as a failure of models that privilege text over contradictory vocal cues. It presents ASPIRE and diagnostic metrics alongside ACR-Net, which explicitly decouples conflicting representations.
1. Introduction
- SER and multimodal models assume acoustic-semantic congruence, while standard datasets filter incongruent samples and congruent-data training can reinforce rigid semantic priors.Prior fusion calibration can shift decision boundaries without disentangling the underlying representations.Source: §1 · p. 1
- ASPIRE probes adversarial acoustic-semantic contradictions, while SOP and LDD quantify semantic overconfidence and latent feature entanglement.Evaluations report high SOP and low LDD for multimodal models under implicit discrepancy.Source: §1 · pp. 1–2
- ACR-Net uses a structurally isolated dual-stream architecture, Cross-Modal Attention, and Contrastive Decoupling Loss to separate conflicting representations.Its design includes a frozen Whisper backbone with Low-Rank Adaptation.Source: §1 · p. 1
- Unlike fusion-level mitigation, ACR-Net explicitly decouples representations to address rigid semantic priors.The introduction distinguishes its approach from concurrent FAS work.Source: §1 · p. 2
- ACR-Net uses explicit feature decoupling to achieve state-of-the-art Acoustic Accuracy under severe cross-modal interference.The contribution contrasts this approach with fusion methods that re-weight collapsed features.Source: §1 · p. 2
2. The ASPIRE Benchmark
The ASPIRE benchmark uses adversarially synthesized emotional conflicts and a screened dataset to test models under controlled acoustic-semantic mismatch. SOP measures text’s probability advantage over audio, while LDD probes pre-fusion representation orthogonality.
2.1. Dataset
- ASPIRE addresses the difficulty of finding real-world conflict data with isolated emotional boundaries by synthesizing samples with controllable emotional expression.The benchmark uses the 1.5B-parameter EmotiVoice model for fine-grained emotional control.Source: §2.1 · p. 2
2.1.1. Synthesis Pipeline
- The four-step pipeline generated, standardized, acoustically filtered, and expert-screened samples; the final dataset contains 1,200 samples.The final set comprises 800 conflict and 400 normal samples.Source: §2.1.1 · p. 2, §2.1.2 · p. 2
2.1.2. Dataset Structure
- ASPIRE covers 19 adversarial pairings across four target acoustic emotions and seven textual emotions, grouped into four structural conflict types.Five of 24 possible incongruent combinations were discarded because of persistent TTS failure.Source: §2.1.2 · p. 2, Table 1 · p. 2
2.2. Evaluation Metrics
- SOP quantifies the probability advantage of text’s apparent emotion over audio’s true emotion, with higher values indicating semantic dominance.The authors motivate SOP as a diagnostic beyond classification accuracy alone.Source: §2.2 · p. 2
- LDD measures the pre-fusion orthogonality of pooled acoustic and semantic representations; higher LDD indicates greater decoupling.The metric is intended to distinguish representation-level conflict resolution from fusion-level weight shifts.Source: §2.2 · pp. 2–3
3. Methodology
ACR-Net explicitly decouples acoustic and semantic features rather than seeking a single fused representation. Its dual-stream encoders, conflict-detecting attention, and contrastive loss support this design.
3. Methodology
- ACR-Net uses structurally isolated streams to preserve acoustic and semantic feature spaces before resolving conflicts.The audio stream uses frozen Whisper-Medium weights with LoRA layers in every self-attention block, while the transcript uses a frozen Whisper text encoder.Source: §3 · p. 3, §3.1 · p. 3
3.2. Cross-Modal Attention as Conflict Detector
- Cross-Modal Attention treats acoustic features as queries that scan semantic tokens for contradictions with vocal tone.A residual connection and mean pooling produce temporally aligned representations for spatial decoupling.Source: §3.2 · p. 3
3.3. Contrastive Decoupling Loss
- The hybrid objective pairs cross-entropy with a decoupling loss that acts as a spatial separator for acoustic and semantic representations.The loss uses cosine similarity between pooled acoustic and semantic vectors and penalizes similarity for conflict samples, separating them by margin m.Source: §3.3 · p. 3
- On ASPIRE, ACR-Net achieves the highest ACC while minimizing SOP among the compared representation strategies.Table 2 also reports hyperparameter analysis indicating the necessity of balanced decoupling.Source: Table 2 · p. 3
4. Experiments
ACR-Net is evaluated on standard congruent speech and ASPIRE conflict scenes, with results highlighting representation preservation under contradiction and competitive performance on ordinary speech. The experiments also compare zero-shot baselines and examine differences among conflict types.
4.1. Experimental Setup
- The evaluation groups zero-shot baselines into audio-only SSL, multimodal task models, Audio LLMs, and recent modality interventions.Standard-scene checks use five datasets spanning German, Chinese, British English, English, and North American English.Source: §4.1 · p. 3, §4.2 · pp. 3–4
4.2. Results on Normal Scenes
- On standard congruent datasets, ACR-Net remains robust and competitive, while Qwen2-Audio's scores vary across datasets and drop on cross-lingual EmoDB.The authors note that Qwen2-Audio's high RAVDESS and CASIA scores may reflect instruction-tuning exposure.Source: Table 3 · p. 4, §4.2 · p. 4
4.3. Deep Diagnostics on Conflict Scenes (ASPIRE)
- Fusion models under polarity conflict show SOP > 0.5 and ACC < 45%, as reweighting cannot recover collapsed acoustic information.Polarity Opposite is described as extreme semantic dominance, with linguistic features overpowering acoustic cues during extraction.Source: §4.3 · p. 4
Findings 2. Feature Decoupling is Essential for Modality
- Arousal Opposite is easiest for all models, but fusion models regress on Emotion Masking while ACR-Net maintains diagnostic accuracy.Low-arousal text, such as neutral transcripts, exerts a weaker semantic pull, allowing fusion models to use strong acoustic cues partially.Source: Findings 2. Feature Decoupling is… · p. 4
- LDD stays above 0.8 across all four conflict types, while ACR-Net drives SOP near zero and achieves state-of-the-art ACC.The table covers the four structural conflict types and reports LDD, SOP, and ACC.Source: Table 4 · p. 4, Findings 2. Feature Decoupling is… · p. 4
5. Conclusion
The conclusion presents ASPIRE and SOP/LDD as tools for diagnosing Semantic Dominance, then describes ACR-Net's decoupling approach and reported performance. It frames the work as an architectural blueprint for speech models handling indirect intent.
5. Conclusion
- The paper uses ASPIRE, SOP, and LDD to expose Semantic Dominance and feature entanglement under acoustic-semantic conflict.Traditional fusion is reported to collapse under polarity conflicts as dominant text overwrites subtle paralinguistic cues.Source: §5 · p. 4
- ACR-Net uses Contrastive Decoupling Loss to project contradictory features into orthogonal latent spaces and preserve acoustic fidelity.Source: §5 · p. 4
- The conclusion reports state-of-the-art ACC across adversarial and congruent datasets and proposes a blueprint for interpreting irony and emotion suppression.Source: §5 · p. 4
7. Generative AI Use Disclosure
Generative AI tools assisted with structural refinement, language polishing, and syntax optimization. The authors state that they reviewed and edited the outputs and take accountability for the final manuscript.
7. Generative AI Use Disclosure
- Generative AI tools assisted with structural refinement, language polishing, and syntax optimization.Source: §7 · p. 5
- The authors carefully reviewed and edited generated outputs to align with their intended scientific meaning.Source: §7 · p. 5
- The authors take full accountability for the final content, experimental results, and overall accuracy.Source: §7 · p. 5