Source-linked AI summary
Finding Emotions Where They Belong: Rethinking Audio Emotion Recognition through Masked Temporal Affective Grounding
Abdelrahman Mohamed, Lars Kai Hansen, Zheng-Hua Tan
TL;DR
Audio emotion recognition usually assigns one label to an entire recording, leaving its temporal scope ambiguous when speakers or affective events vary. The paper introduces span-level Temporal Affective Grounding and M-TAG, trained on temporally annotated data. EMO-TAG improves grounding and emotion recognition over compared baselines, while the authors note limitations in data balance and tone-description evaluation.
Problem
Utterance-level emotion labels leave unclear which speech span expresses the emotion, especially in recordings with multiple speakers, overlapping speech, or non-speech cues.
Source: §1 · pp. 1–2Method
The paper constructs temporally annotated single- and multi-span datasets and trains TAG with M-TAG, combining full-sequence language modeling with masked emotion and distance-weighted timestamp losses.
Source: §1 · p. 2, §4.2 · p. 5Results
Takeaways & Limitations
The results support temporal affective grounding as a practical formulation of emotion recognition that can improve temporal grounding without harming recognition performance.
Source: §7 · p. 9The data remain imbalanced, tone descriptions were not manually verified or evaluated, and the study focuses on eight basic emotions.
Source: F LIMITATIONS · p. 39
Abstract
from arXiv · showhide
Audio emotion recognition (AER) typically assigns a single label to an entire recording, leaving the temporal scope of that label ambiguous when multiple speakers and affective events are present. We address this limitation by reformulating AER as a Temporal Affective Grounding (TAG) task that associates emotions with temporally bounded speech spans and vocal tone descriptions. To support this formulation, we curate temporally annotated versions of existing emotion recognition datasets and construct recordings containing two to four affective speech spans, including overlapping speech. Training in this longer format with a standard language-modeling objective can degrade both emotion recognition and temporal grounding performance, while tone descriptions can provide shortcuts for emotion prediction. To address these challenges, we introduce Masked Temporal Affective Grounding (M-TAG), a supervised training objective that combines full-sequence language modeling with emotion and timestamp cross-entropy losses under attention masking. The masking varies the context visible to emotion-label tokens to reduce reliance on shortcuts and improve generalization, while the timestamp loss incorporates a distance-aware weight to penalize larger temporal errors. We evaluate EMO-TAG, a model fine-tuned using our dataset and objective, on emotion recognition and affective temporal-grounding against three AER and audio-language baselines: Flamingo-Next, Audio-Reasoner, and AffectGPT. Our results show that existing models achieve limited affective temporal-grounding despite competitive emotion recognition performance.
1 INTRODUCTION
The paper reframes audio emotion recognition as grounding emotions in bounded speech spans, addressing the ambiguity of assigning one label to recordings that may contain multiple speakers or affective events. It introduces annotated datasets and M-TAG, and reports stronger emotion recognition and temporal grounding than compared baselines.
1 INTRODUCTION
- TAG associates each temporally bounded speech span with a transcript, tone description, and emotion label; its boundaries mark the speech span, not the precise onset and offset of emotion.This reframing addresses recordings containing multiple speakers, overlapping speech, laughter, noise, or empty audio, where a single label makes evaluation ambiguous.Source: §1 · pp. 1–2
- The curated data include refined single-label sets, multi-span examples with two to four spans, and a challenging subset drawn primarily from noisy samples.The data are temporally annotated and derived from existing public emotion-recognition datasets.Source: §1 · p. 2
- Standard language-model training can underweight emotion labels and timestamps, while preceding tone descriptions can offer a shortcut and categorical timestamp loss ignores error distance.These issues can weaken emotion and grounding supervision when the target sequence also contains transcription and tone tokens.Source: §1 · p. 2
- M-TAG combines full-sequence language modeling with masked emotion and timestamp losses, varying context for emotion predictions and weighting timestamp loss by temporal distance.The objective adds supervision to emotion labels and boundaries while retaining full-context supervision of the structured output.Source: §1 · p. 2
- EMO-TAG beats the strongest compared baseline on Emotion-Duration F1 by 23.12 percentage points on MELD and 35.17 percentage points on IEMOCAP.The paper also reports improved recognition on original and refined single-span benchmarks and the best emotion-recognition accuracy on MME-Emotion.Source: §1 · p. 2
2 RELATED WORK
Related work extends audio emotion recognition with natural-language descriptions, reasoning, and acoustic cues, while this paper trains models to generate structured affective spans. Prior temporal grounding work localizes audio events or emotion frames, whereas this formulation assigns emotion to bounded speech spans.
2 RELATED WORK
- Audio-language emotion models add descriptions, rationales, or textualized acoustic cues, while this paper trains the model to generate additional cues as structured output.AffectGPT uses descriptive annotations, while SpeechCueLLM and VowelPrompt provide vocal-characteristic descriptions as model inputs.Source: §2 · pp. 2–3
- Emotion-aware training work uses reinforcement-learning rewards, whereas this approach emphasizes supervised learning with a designed loss function.The cited approaches include GRPO with emotion-wheel rewards and rewards for prosody reasoning or explanation consistency.Source: §2 · p. 3
- Prior temporal audio work grounds reasoning or captions, and Speech Emotion Diarization labels emotion frames; TAG instead aligns an emotion label to a bounded speech span.Its boundaries do not represent the precise onset and offset of emotional expression.Source: §2 · p. 3
3 DATASET CREATION
The dataset pipeline filters and annotates existing emotion-recognition recordings, adds normalized tone descriptions, and constructs multi-span examples with controlled gaps and overlap.
3 DATASET CREATION
- The dataset is built from IEMOCAP, MELD, and Emov-DB, with a pipeline to check audio, refine temporal boundaries, and reject inconsistent samples.The construction includes temporally annotated single-span and multi-span data.Source: §3 · p. 3
3.1 AUDIO PREPROCESSING
- VAD identifies empty audio and speech regions; preprocessing removes high-energy non-speech such as laughter and filters irrelevant or inconsistent speech segments using transcription and forced alignment.Samples are rejected when remaining transcript errors exceed specified insertion or replacement thresholds.Source: §3.1 · p. 4
3.2 TONE ANNOTATION
- Tone descriptions are checked against transcripts and ground-truth emotion, then normalized using segment-level acoustic features and speaker-specific reference statistics.The pipeline retains samples only when transcription similarity reaches 60%; normalized descriptions draw on features such as pitch, intensity, and speaking pace.Source: §3.2 · p. 4
3.3 MULTI-SPAN CONSTRUCTION
- Multi-span examples concatenate cleaned recordings, with consecutive-span gaps or overlaps sampled uniformly between 0.5 and 2.0 seconds.Gaps use silence regions from the same dataset to preserve dataset-specific background characteristics.Source: §3.3 · p. 4
4 METHOD
The method formulates emotion recognition as structured prediction over speech spans and trains with full-context language modeling plus masked emotion and temporal supervision. A three-stage curriculum moves from format learning to masked grounding and multi-span prediction.
4.1 TASK FORMULATION
- TAG predicts one to four speech spans, each containing temporal boundaries, a transcript, a vocal-tone description, and an emotion label.The boundaries identify the speech span to which the emotion label applies.Source: §4.1 · p. 4
4.2 MASKED TEMPORAL AFFECTIVE GROUNDING
- The full-context language-modeling loss supervises target tokens across the structured output, while auxiliary emotion loss emphasizes each span's first emotion-label token.The auxiliary emotion prediction uses a sampled attention mask; selected tokens remain in the sequence but attention from the label to them is blocked.Source: §4.2 · p. 5
- Temporal supervision targets each span's start and end timestamp tokens, using distance-dependent weights for boundary errors and the same masked forward pass as emotion supervision.The overall objective adds emotion and boundary losses to full-sequence language modeling.Source: §4.2 · pp. 5–6
4.3 TRAINING CURRICULUM
- Training progresses from single-span format learning, to masked grounding, and then to multi-span temporal learning.The final stage introduces recordings with two, three, and four spans; noise or silence of 0.5 to 1 seconds is added during masked grounding.Source: §4.3 · p. 6
5 EXPERIMENTS
The experiments evaluate emotion recognition and temporal affective grounding with separate metrics for emotion and timing. EMO-TAG outperforms the strongest compared baseline on multi-span Emotion-Duration F1, while context ablations show that full M-TAG also exceeds emotion-only training.
5 EXPERIMENTS
- The evaluation compares FlamingoNext, Audio-Reasoner, and AffectGPT; AffectGPT is assessed only for emotion recognition, while temporal baselines are prompted for timestamps and labels.Qwen3 extracts predicted emotions and temporal boundaries from the temporal models' outputs.Source: §5 · p. 6
5.1 EVALUATION METRICS
- The evaluation distinguishes utterance-level segment accuracy, any-span emotion hit accuracy, and Emotion-Duration F1, which requires matching both emotion and temporal region.For multi-span recordings, hit accuracy checks each ground-truth span independently; duration F1 uses 0.25-second frames and combines duration precision and recall.Source: §5.1 · pp. 6–7
5.2 RESULTS
- EMO-TAG had the highest IEMOCAP Emotion Segment Accuracy on Original, Refined, and Noisy sets (65.62%, 65.32%, 63.51%) and MELD Original and Refined (54.65%, 57.91%).On MELD Noisy, EMO-TAG achieved 44.12% segment accuracy and 51.47% Emo-Hit Accuracy, compared with Audio-Reasoner's 39.71% and 49.37%.Source: §5.2 · pp. 7–8, Table 1 · p. 8
- On MME-Emotion, EMO-TAG's mean Emotion Segment Accuracy was 58.20% on Noise, 33.56% on Wild, and 45.27% on Lab, exceeding FlamingoNext by 7.43, 1.06, and 7.64 points.EMO-TAG outperformed Audio-Reasoner on all three subsets despite using audio alone while Audio-Reasoner received audio and transcript.Source: §5.2 · p. 8
- 54.55% vs 31.43% on MELD and 63.72% vs 28.55% on IEMOCAP Emotion-Duration F1; gains were 23.12 and 35.17 percentage points.EMO-TAG also achieved the highest Emo-Hit Accuracy on both datasets: 57.09% on MELD and 61.22% on IEMOCAP.Source: §5.2 · p. 8
- Full M-TAG exceeded emotion-only training by 1.4 points in Emo-Hit accuracy and 2.0 points in Emo-Duration F1; full-context SFT with original generated tone performed worst.Emotion-only training averaged 58.8% Emo-Hit accuracy and 57.5% Emo-Duration F1; Figure 2 compares generated and refined tone under full-context training with partial-context SFT settings.Source: Figure 2 · p. 7, §5.2 · p. 8
6 DISCUSSION
The discussion highlights a gap between recognizing emotions and grounding them in the correct speech spans, alongside trade-offs in training context and objective. It also reports preliminary limits of GRPO for this concise audio task.
6 DISCUSSION
- Existing models often identify relevant emotions without grounding them in the correct spans, while EMO-TAG improves grounding and recognition across the evaluated datasets.The comparison covers MELD and IEMOCAP multi-span sets and MME-Emotion benchmarks; EMO-TAG uses audio while some baselines also receive transcripts.Source: §6 · pp. 8–9, Table 2 · p. 9
- Emotion-only training performed strongly, but full-context SFT performed worst overall; refined, more consistent tone descriptions performed better than initial descriptions.The authors hypothesize that detailed intermediate reasoning may require more training data to be learned reliably.Source: §6 · p. 9
- In preliminary experiments, GRPO after SFT without M-TAG yielded modest improvements, while longer generations reduced attention to the audio in this concise-generation task.The authors argue that GRPO should not be treated as the default approach for improving a metric within structured output.Source: §6 · p. 9
7 CONCLUSION
The conclusion presents temporal affective grounding as a feasible formulation of audio emotion recognition, supported by annotated datasets and a masked training objective. The paper also describes plans for reproducibility resources and its uses of AI tools.
7 CONCLUSION
- The authors report that temporal affective grounding with their masked objective improves temporal grounding and emotion recognition against widely used baselines without harming recognition performance.Their formulation uses temporally annotated datasets with varied span settings.Source: §7 · p. 9
REPRODUCIBILITY STATEMENT
- The authors plan to release training and dataset-creation code, datasets if copyright allows, model checkpoints, and W&B artifacts with training and evaluation details.They state that public availability is intended to encourage transparency and reproducibility.Source: REPRODUCIBILITY STATEMENT · p. 10
AI USE STATEMENT
- ChatGPT assisted with coding and manuscript refinement, while FlamingoNext, Qwen3, and ChatGPT were used in data annotation and output structuring.The authors state that AI tools did not formulate the research gap, methodology, research ideas, or scientific contributions.Source: AI USE STATEMENT · p. 10
APPENDICES
The appendices detail model training, audio preprocessing, target annotation, and multi-span construction. They specify the adaptation setup, filtering and alignment procedures, tone-description refinement, and span timing conventions.
A.1 MODEL
- Qwen2-Audio-7B-Instruct is adapted with LoRA on the language model and audio encoder, a fine-tuned projection layer, and trainable timestamp-token rows.LoRA uses rank 16, scaling parameter α = 32, and dropout 0.05; training runs for 50 epochs with AdamW at a learning rate of 10^-5.Source: §A.1 · p. 14
A.2 AUDIO PREPROCESSING
- Audio preprocessing uses WhisperX transcription and alignment, acoustic measures for candidate gaps, and transcription checks when trimming clips.For MAFW and DEFW, samples are filtered by language and clarity, and are discarded when FlamingoNext and Whisper transcriptions have less than 60% alignment.Source: §A.2 · p. 14
A.3 TARGET ANNOTATIONS
- The annotation pipeline standardizes labels into eight emotion categories and refines tone descriptions using speaker-specific acoustic references and segment-level features.Qwen3 extracts speech and tone fields from FlamingoNext output; ChatGPT uses five segments of openSMILE features to refine descriptions of pitch, intensity, pace, and voice quality.Source: §A.3 · pp. 14–15
A.4 MULTI-SPAN DESIGN CHOICES
- Multi-span recordings retain sampled utterances as separate spans, inspect inserted silence intervals, and update span timestamps for each utterance's offset.Speaker numbering follows first appearance within each recording; offsets account for inserted silence and overlap.Source: §A.4 · p. 15
B TRAINING ABLATION
The ablations examine training context, loss components, masking probability, and trainable parameters. They report benefits from context masking and projection-layer training, with p = 1 selected for the main experiments.
B.1 CONTEXT INFORMATION AND MASKING
- The ablation varies training context and loss components, with single-span Emo-Hit Accuracy and temporal Span F1 and mIoU reported.Span F1 uses a 0.25-second boundary tolerance; mIoU measures average overlap between predicted and reference spans.Source: Table 3 · p. 16, §B.1 · p. 16
- Without context masking, auxiliary emotion and timestamp losses yield modest, inconsistent improvements over SFT; introducing masking considerably improves performance.Removing emotion loss mainly hurts emotion recognition, while removing timestamp loss reduces temporal accuracy.Source: §B.1 · p. 16, Table 3 · p. 16
B.2 MASKING PROBABILITY
- The masking effect remains consistent across the evaluated probabilities, and the main experiments use p = 1.Figure 3 evaluates p values of 0, 0.2, 0.5, 0.7, and 1; p = 0 disables context masking.Source: §B.2 · p. 16, Figure 3 · p. 17
B.3 TRAINABLE COMPONENTS
- Training the projection layer alongside the other components gives the best results across all four ablation settings.The configurations add trainable components cumulatively and use the full objective with p = 1.Source: §B.3 · p. 16, Table 4 · p. 17
C.1 EMOTION-ONLY DATA
Emotion-only supervision on added DEFW and MAFW data maintains competitive performance and sometimes surpasses full supervision, though the added data slightly harms IEMOCAP and MELD results. The appendix also reports a modest GRPO improvement over SFT and notes that aggressive filtering loses many good, especially short, samples.
C.1 EMOTION-ONLY DATA
- Added DEFW and MAFW data improve their respective test sets and some MME-Emotion subsets, but slightly harm IEMOCAP and MELD.With emotion-only supervision, performance on the MME-Emotion Noise subset also improves; the authors suggest dataset differences may explain the declines.Source: §C.1 · p. 18, Table 5 · p. 18
- Emotion-only supervision on the additional datasets maintains competitive performance and sometimes surpasses full supervision.For this setting, only the first token of each emotion label receives the masked affective loss; other tokens contribute zero loss.Source: §C.1 · p. 18
C.2 GRPO FINE-TUNING
- GRPO yields a modest improvement over SFT, although early experiments occasionally switch to Chinese during generation.The preliminary experiment initializes from the SFT model and retains the speech, timestamp, tone, and emotion output format.Source: §C.2 · pp. 18–19
D DATASETS
- The filtering pipeline is aggressive and consequently discards many good samples, especially very short ones.Source: D DATASETS · p. 20
D.1 DATA STATISTICS
The data-statistics materials describe dataset composition and filtering, while the tone-description comparisons show a trade-off: refined descriptions use a more consistent acoustic structure but can lose delivery details.
D.1 DATA STATISTICS
- Table 7 covers training and test dataset composition, overlapping-span samples, and filtering statistics for the original datasets.Its caption distinguishes MELD as a more natural setting from the controlled and acted settings of EmoV-DB and IEMOCAP.Source: Table 7 · p. 20, §D.1 · p. 20
D.2 TONE DESCRIPTIONS
- Refined tone descriptions use a more consistent acoustic structure but sometimes lose original details about word emphasis and changes in delivery.Table 8 presents selected comparisons between original and refined descriptions.Source: §D.2 · p. 20, Table 8 · p. 21
D.1 DATA STATISTICS
- Figure 4 summarizes the processing pipeline from speech and tone validation through construction of cleaned single- and multi-span datasets.Source: §D.1 · p. 20, Figure 4 · p. 21
E QUALITATIVE RESULTS
The qualitative results examine how Emo-TAG and baseline models assign emotions to speech spans and, in some cases, describe vocal tone. Examples show matches to emotion sequences and span boundaries as well as errors in labels, span coverage, segmentation, and timing, with outputs differing across models and some apparent annotation errors.
E QUALITATIVE RESULTS
- In one MELD example, all models predict natural delivery despite a surprise annotation; in another, all agree that the ground-truth transcription omits part of the audio.Source: E QUALITATIVE RESULTS · p. 23
E.1 EXAMPLE 1: EMOTION RECOGNITION AND TEMPORAL LOCALIZATION
- Emo-TAG identifies both emotions and boundaries correctly, while Audio-Reasoner recognizes the emotions but misplaces the boundaries.Source: §E.1 · p. 23
E.2 EXAMPLE 2: EMOTION RECOGNITION WITH TRANSCRIPTION DIFFERENCES
- Despite transcript errors, Emo-TAG gets both emotions and boundaries right; FlamingoNext transcribes “How neat is that?” and predicts happiness instead of reference disgust for “how needy is that?”.Source: §E.2 · p. 24
E.3 EXAMPLE 3: RECOVERING THREE EMOTION SPANS
- Emo-TAG recovers all three emotion spans but adds transcript words; FlamingoNext splits the first utterance, while Audio-Reasoner misclassifies the first and last emotions.Source: §E.3 · p. 25
E.4 EXAMPLE 4: CORRECT BOUNDARIES BUT INCORRECT EMOTIONS
- All three models label both utterances neutral, missing surprise and sadness, although Emo-TAG places both boundaries correctly.Source: §E.4 · p. 26
E.5 EXAMPLE 5: MERGING UTTERANCES WITH DIFFERENT EMOTIONS
- Emo-TAG merges the last two utterances into one surprise span, missing the neutral change and including the gap; baselines produce duplicate or extra spans, and all models misclassify the first emotion.Source: §E.5 · p. 27
Audio-Reasoner prediction
- Audio-Reasoner labels the 0.00–5.50-second span neutral, then alternates surprise and neutral across five successive spans from 5.50 to 8.00 seconds, beginning with surprise.Source: Audio-Reasoner prediction · p. 28
E.6 EXAMPLE 6: INCOMPLETE TEMPORAL COVERAGE
- Emo-TAG gets the emotion sequence right but ends its second span 2.25 seconds early; both baselines miss initial anger and add spans beyond the final reference utterance.The reference and Emo-TAG both mark the first anger span as 0.00–4.75 seconds; Emo-TAG’s second span is neutral from 5.50–8.00 seconds.Source: §E.6 · p. 29, Reference annotation · p. 29, Emo-TAG prediction · p. 29
E.7 EXAMPLE 7: EMOTION AND TEMPORAL ACCURACY
- Emo-TAG matches both emotions and their boundaries, while Audio-Reasoner shifts the boundary and FlamingoNext splits the first utterance.Source: §E.7 · p. 31
E.8 EXAMPLE 8: SHORT EMOTIONAL UTTERANCES
- Emo-TAG correctly identifies frustration followed by sadness and matches both spans; the baselines predict sadness followed by neutral.Source: §E.8 · p. 32, Emo-TAG · p. 32, FlamingoNext · p. 32
E.9 EXAMPLE 9: THREE-EMOTION SEQUENCE
- Emo-TAG follows the frustration-to-happiness-to-sadness sequence and matches all three spans despite transcription errors.Source: §E.9 · p. 33, Emo-TAG · p. 33
Audio-Reasoner
- Audio-Reasoner outputs frustration, surprise, and neutral across three spans in one example, but labels both the late-fee and crying utterances neutral in another.Source: Audio-Reasoner · pp. 34–35
E.10 EXAMPLE 10: EMOTION CONFUSION
- Emo-TAG labels a neutral utterance as frustration despite matching both spans, while FlamingoNext identifies both emotions but misplaces their boundaries.This example highlights that emotion-label correctness and temporal-boundary accuracy can diverge.Source: §E.10 · p. 34
FlamingoNext
- In the supplied FlamingoNext annotations, speaker1 is labeled neutral over 0.00–3.00 seconds, while speaker2 is labeled sadness over 4.00–5.00 seconds.Source: FlamingoNext · p. 35
E.11 EXAMPLE 11: MERGED UTTERANCES
- Emo-TAG combines the first two utterances into one frustration span, missing the change from neutral to frustration.Source: §E.11 · p. 35
E.12 EXAMPLE 12: PREMATURE ONSET
- Emo-TAG starts the happiness span 1.75 seconds early, creating a 1.25-second overlap with the preceding neutral span.Source: §E.12 · p. 37
Emo-TAG
- Emo-TAG pairs its frustration and sadness spans with detailed vocal-tone descriptions, and describes a happiness span as bright, high-pitched, fast, and enthusiastic.The examples include slightly raised pitch and quickened tempo for mild frustration, and low pitch and slow pace for quiet sorrow.Source: Emo-TAG · pp. 34, 37
F LIMITATIONS
The authors report imbalanced emotion data, unverified tone descriptions, and an eight-emotion scope; the appendix details prompts for training, evaluation, tone extraction, and refinement.
F LIMITATIONS
- Training data are imbalanced, tone descriptions were not manually verified or evaluated, and the study focuses on eight basic emotions.The authors identify finer-grained labels as future work.Source: F LIMITATIONS · p. 39
G PROMPTS
- Training prompts vary whether emotion is inferred alone, after speech, after tone, or after both speech and tone, while retaining timestamp instructions.The appendix states that full-context training includes speech, timestamps, tone, and emotion, and that partial-context settings vary speech and tone inclusion.Source: G PROMPTS · p. 40, §G.1.1 · p. 40, §G.1.2 · p. 40
G.2 MODEL EVALUATION
- For model evaluation, the authors retain each model’s system prompt and use Qwen3 to extract structured answers without guessing unstated fields.The extractor returns speech, emotion, and timestamps, using null when information is absent and multiple segments when applicable.Source: §G.2 · p. 42, §G.2.1 · p. 42
G.3.1 FLAMINGO PROMPT
- The Flamingo prompt requests timestamped speech and vocal-tone descriptions while explicitly instructing the model not to infer an emotion label.It asks for audible delivery cues and cautions against adding unsupported details.Source: §G.3.1 · p. 43
G.3.2 QWEN3 TONE EXTRACTOR
- The Qwen3 tone extractor concatenates quoted speech spans and summarizes vocal delivery rather than semantic content.Its instructions focus on acoustic cues and allow “wrong tone” when the tone conflicts with the given emotion.Source: §G.3.2 · p. 43
G.4 TONE REFINEMENT
- Tone refinement uses speaker-specific reference values and acoustic features from five temporal segments to produce one concrete tone sentence.The instructions require cues grounded in acoustic features and interpreted relative to speaker reference values.Source: §G.4 · p. 44