Source-linked AI summary
Multimodal Temporal Modeling for Continuous Group Emotion Recognition in Multi-party Dialogues
Soma Iwata, Koji Inoue, Muyun Wu, Taiga Mori, Divesh Lala, Tatsuya Kawahara
TL;DR
Continuous group-emotion recognition in multi-party dialogue remains limited by utterance-level or coarse-window approaches that miss fine temporal dynamics. This study builds one-second labels, including Mixed annotations for emotional divergence, and evaluates multimodal temporal models. The Transformer outperforms baselines and the LLM on key continuous metrics, audio-video input improves on either modality alone, and errors are larger in highly Mixed intervals.
Problem
Prior work often addresses group emotion at the utterance level or in coarse time windows, leaving continuous one-second recognition and explicit emotional-divergence analysis unaddressed.
Source: §1 · pp. 1–2Method
The study constructs one-second soft labels for Group Emotion and Mixed states from continuous TEIDAN annotations, then evaluates multimodal temporal models using audio and video.
Source: §1 · p. 2, §3.1 · p. 5Results
Takeaways & Limitations
Recognition errors were larger in HighMixed intervals, associating emotional divergence with greater difficulty when group emotion is represented by a single value.
Source: §5 · p. 7The dialogue data come from a specific cultural group, so the model may contain cultural biases and requires care across diverse cultural backgrounds.
Source: Safe and Responsible Innovation… · p. 8
Abstract
from arXiv · showhide
To realize natural behavior in dialogue agents in multi-party dialogue scenarios, it is important to understand group emotion such as valence and arousal as a whole. Most prior work addressed this task at the utterance level or using a coarse-grained time window, which is not sufficient to capture emotional dynamics. In this study, we formulate continuous recognition of the Group Emotion at a one-second resolution. Moreover, we also introduce the Mixed state, which captures the emotional divergence among participants in the group. We constructed a dataset with frame-level soft labels based on the TEIDAN corpus and propose a multimodal temporal framework that integrates audio and video information using a sliding-window context. Experimental results demonstrate that the temporal Transformer outperforms simple baselines and shows stronger temporal agreement with the ground-truth labels than the LLM-based model. The effect of context length is limited, whereas audio-visual input outperforms either unimodal input on the continuous-label metrics. Additionally, our analysis shows larger Group Emotion recognition errors in intervals with high Mixed values, exposing emotional divergence as a key challenge for group emotion recognition.
1 Introduction
Recognizing group emotion in multi-party dialogue requires tracking interactions and emotional change over time, including divergence among participants. Prior work leaves a gap in continuous group-level recognition, which this study addresses with one-second labels and multimodal temporal modeling.
1 Introduction
- Group emotion emerges through participant interactions and is not merely the sum of individual emotions; its components include convergence, divergence, emotional culture, and dynamic process.Emotions may also change within utterances and during silence, making multi-party emotional dynamics temporally complex.Source: §1 · p. 1
- Table 1 compares studies by participant scale, recognition target, temporal resolution, non-speaker consideration, and explicit treatment of emotional divergence.The supplied text identifies these comparison dimensions but does not provide the table's cell values.Source: §1 · p. 1, Table 1 · p. 2
- Many multi-party dialogue datasets still predict individual speaker or listener emotion rather than the collective emotional state of the group.The cited continuous-emotion work focuses on individual emotion in dyadic interactions.Source: §1 · p. 2
- Earlier group-level datasets use independent clips or coarse windows, while one-second recognition over full dialogues with explicit divergence analysis was identified as an unaddressed combination.The cited prior temporal analysis used 15-second windows, too coarse to locate changes within utterances or between turns.Source: §1 · p. 2
- The study targets continuous Group Emotion recognition across multi-party dialogues, representing group state with Valence and Arousal and divergence with Mixed labels.It constructs one-second soft labels and compares a temporal Transformer with an LLM-based model using participant-specific audio and visual features.Source: §1 · p. 2
2 Dataset
The study uses the Japanese portion of TEIDAN, drawing on spontaneous discussions among triads to create a dataset of multi-party dialogues.
2 Dataset
- The Dataset section describes TEIDAN, the annotation design, and construction of model ground-truth labels.It also analyzes label distributions, annotator agreement, and relationships between dialogue context and emotion labels.Source: §2 · p. 2
2.1 TEIDAN Corpus
- The dataset comprises 36 spontaneous discussions from 12 groups, with dialogues lasting approximately six minutes on average.The recordings include frontal video of each participant and audio from individual lavalier microphones.Source: §2.1 · p. 2
- The dialogues come from the Japanese portion of TEIDAN and feature groups of three members from the same laboratory, primarily undergraduate and graduate students.Participants discussed given topics in spontaneous discussions or attentive-listening sessions; this study used the spontaneous discussions.Source: §2.1 · p. 2
2.2 Annotation Design
Five annotators continuously label group Arousal and Valence and mark Mixed segments where participants' emotions diverge, capturing changes both within utterances and during silence.
2.2 Annotation Design
- Arousal and Valence represent the group's emotional state, while Mixed Arousal and Mixed Valence mark segments where participants' emotions diverge substantially.The four annotation types were assigned to each dialogue by five annotators.Source: §2.2 · pp. 2–3
- Continuous annotation without fixed units captures emotional shifts within utterances and emotions during silences that utterance-level annotation typically excludes.The design supports long-dialogue context while predicting Group Emotion at fine temporal resolution.Source: §2.2 · p. 3
- Group Emotion extends Russell’s circumplex model from individuals to groups, with Arousal and Valence each classified into five levels from +2 to -2.Mixed labels are assigned to segments where participants' emotions diverge.Source: §2.2 · p. 3
- Annotators judged split-screen video and mixed participant audio holistically, using only information available up to the current moment.They freely marked non-neutral or Mixed segment boundaries, and unmarked periods were treated as 0 or Not Mixed.Source: §2.2 · p. 3
- Figure 2 illustrates five annotators’ continuous Valence and Mixed Valence series, with reasonable agreement in the timing of Valence peaks.The upper plot shows Valence classes and the lower bars show segments each annotator marked Mixed.Source: §2.2 · p. 3, Figure 2 · p. 3
2.3 Label Construction
The study selects annotation data and aggregates continuous labels into one-second frames, using soft labels that preserve class occupancy and Mixed-state duration.
2.3.1 Data Selection.
- Mixed annotations were screened for outlier total labeled durations, excluding an annotator independently by dialogue and Mixed dimension when the deviation reached 1.5 standard deviations or more.One annotator was excluded in all 36 dialogues for Mixed Arousal and in 27 dialogues for Mixed Valence.Source: §2.3.1 · p. 3
2.3.2 Aggregation into Frames.
- For each one-second frame, the soft Group Emotion label is the average time occupancy of each class across all five annotators.This aggregates continuous annotations into frame-level training and evaluation targets.Source: §2.3.2 · p. 4
- Soft labels preserve within-frame class transitions and annotator boundary variation; classification uses the argmax of those labels as the hard label.The study evaluates with regression-based and classification-based metrics.Source: §2.3.2 · p. 4
2.3 Label Construction
- Mixed labels are aggregated into one-second frames as values from 0 to 1, representing the fraction of each frame occupied by the Mixed state.The construction procedure creates ground-truth labels from continuous annotations for model training and evaluation.Source: §2.3 · p. 3, §2.3.2 · p. 4
2.4 Label Distribution and Agreement
This section reports label distributions and annotator agreement for the continuous group-emotion labels. Neutral and Not Mixed labels dominate, while agreement remains low for Mixed Valence.
2.4 Label Distribution and Agreement
- The 12 groups were split 8:2:2 into 24/6/6 dialogues, with no frames, dialogues, or groups shared across splits.Some participants appeared in multiple splits because each took part in two groups.Source: §2.4.1 · p. 4
2.4.1 Label Distribution.
- Class 0 accounts for more than 85% of Group Emotion labels, and Not Mixed exceeds 92% of Mixed labels.Classes -2 and -1 are rare, and label proportions are broadly similar across splits.Source: §2.4.1 · p. 4, Table 6 · p. 4
2.4.2 Inter-annotator Agreement.
- After data selection, agreement increased for both Mixed labels but remained low for Mixed Valence.Group Emotion agreement was relatively modest, although annotators broadly agreed on the timing of emotional peaks.Source: §2.4.2 · p. 4, Table 7 · p. 4
2.5 Relationship between Dialogue Context and Emotion Labels
The section examines associations between dialogue context and continuous emotion labels. Group Arousal and Valence rise with laughter and turn-initiation frequency, while silence is associated with lower values.
2.5.1 Laughter and Emotion Labels.
- Each additional laughing participant was associated with increases of 0.245 in Arousal and 0.323 in Valence.The respective 95% CIs were [0.206, 0.277] and [0.278, 0.363].Source: §2.5.1 · p. 4, Table 8 · p. 5
- Mixed Arousal peaked at 0.32 when two people laughed, while Mixed Valence was 0.16 with one or two laughing participants and 0.06 with all three.These patterns suggest greater divergence when one or two participants laugh, although simultaneous laughter by two or three was relatively rare.Source: §2.5.1 · pp. 4–5
2.5.2 Dialogue Activity and Emotion Labels.
- Excluding sparsely represented counts 4 and 5, Arousal rose from 0.054 to 0.246 and Valence from 0.111 to 0.312 as turn-initiation frequency increased from 0 to 3.Source: §2.5.2 · p. 5
- Silence had lower Arousal (difference = −0.184) and Valence (difference = −0.195) than non-silence.The respective 95% CIs were [−0.218, −0.145] and [−0.230, −0.156].Source: §2.5.2 · p. 5, Table 10 · p. 5
3 Proposed Model
The proposed models estimate one-second group Arousal and Valence from participant-specific audio and video features, using sliding-window context. The approach also accounts for ordinal labels and class imbalance during training.
3.1 Model Architecture
- The temporal Transformer processes concatenated audio-visual features with six encoder layers and predicts Arousal and Valence distributions from the final one-second interval.The LLM-based comparison model sends multimodal features through a Q-Former to Qwen3-1.7B; both variants share participant-specific SigLIP 2 visual and Whisper audio feature extraction.Source: §3.1 · pp. 5–6
- Each W-second window, including past context, predicts the emotional state in its final one-second frame; training and evaluation strides are 3 and 1 seconds, respectively.Audio and video features are aligned at 30 Hz before being concatenated across three participants.Source: §3.1 · pp. 5–6
3.2 Loss Function
- Squared Earth Mover’s Distance uses predicted and target cumulative distributions, penalizing errors between neighboring ordinal classes less than distant-class errors.Arousal and Valence losses are combined with weights both set to 1.Source: §3.2 · p. 6
3.3 Mitigation of Class Imbalance
- To counter the high proportion of Class 0, training adjusts loss weights using class durations and adds sampling weighted by reciprocal class frequency.A smoothing factor adjusts the raw weights before they are normalized; frame weights are determined from the target label distribution and class weights.Source: §3.3 · p. 6
4 Experiments
Experiments compare model type, context length, and input modality on continuous group-emotion predictions, with uncertainty estimated through dialogue-level bootstrap intervals. The temporal Transformer AV(20s) leads on key continuous-label metrics, while audio-visual input improves on unimodal alternatives.
4.1 Setup
- The evaluation reports CCC and MAE on expected frame-level outputs, Pos F1 for positive classes, and paired dialogue-level bootstrap confidence intervals based on six test dialogues.The conditions compared model type under AV(20s), context length for the Transformer, and audio, video, or combined input; each condition was trained up to 10 epochs.Source: §4.1 · pp. 6–7, §4 · p. 6
4.2 Results
- Transformer AV(20s) achieved the highest CCC and lowest MAE for both dimensions; versus the LLM, CCC was higher by 0.059 for Arousal and 0.018 for Valence, and Arousal MAE was lower by 0.023.All three difference confidence intervals excluded zero; the Valence MAE and Pos F1 differences did not show a statistically clear advantage.Source: Table 11 · p. 7, §4.2 · p. 7
- Audio-visual input improved CCC and MAE over audio-only for both dimensions and all reported metrics over video-only, with confidence intervals excluding zero.Audio-only outperformed video-only on all metrics, though confidence intervals excluded zero for only Arousal CCC, Valence CCC, Valence MAE, and Arousal Pos F1.Source: §4.2 · p. 7
- Only the Arousal MAE comparison showed a confidence interval excluding zero in favor of 20-second rather than 5-second context.The other context-length metric differences had confidence intervals including zero.Source: §4.2 · p. 7
5 Discussion
The Transformer outperformed the LLM on selected continuous-estimation metrics, while the results did not establish a general LLM limitation. Longer context showed limited evidence of benefit, and high Mixed intervals were associated with larger errors.
5 Discussion
- The Transformer had higher CCC for both dimensions and lower Arousal MAE than the LLM, but the comparison does not establish a general limitation of LLMs.The multimodal interface and training-data amount may not fully exploit LLM capacity; language-based reasoning and explanation remain promising uses for this task.Source: §5 · p. 7
- Only Arousal MAE showed a clear benefit from longer context, whereas audio-visual input improved CCC and MAE over both unimodal models.Audio appeared to be the stronger unimodal signal in this setting, with video providing complementary information; associations with laughter and dialogue activity may partly explain audio’s importance.Source: §5 · p. 7
- HighMixed intervals had significantly higher MAE for both dimensions, consistent with greater difficulty when one value represents emotional divergence.Possible explanations include salient emotions from some participants or less reliable group-level labels in Mixed segments; limited HighMixed frames and inconclusive Pos F1 differences qualify the finding.Source: §5 · pp. 7–8
- Evaluation measures generalization to unseen group configurations, not unseen individuals, and six test dialogues from two groups may yield bootstrap intervals narrower than group-level intervals.Each condition was trained once on one split, and annotations lack participant-level emotion labels.Source: §5 · p. 8
6 Conclusion
The temporal Transformer outperformed distribution-based baselines and the LLM-based model on key metrics; audio-video input improved results over either modality alone, while longer context helped only slightly. Errors were larger during Mixed intervals, and the authors identify annotation, visual-cue, and streaming directions for future work.
6 Conclusion
- The temporal Transformer outperformed distribution-based baselines and the LLM-based model in CCC for both dimensions and in Arousal MAE.The reported dimensions are Arousal and Valence.Source: §6 · p. 8
- Audio-video input improved CCC and MAE over either modality alone, with audio providing the stronger unimodal signal; 20-second context offered only limited improvement over 5-second context.Source: §6 · p. 8
- Recognition errors were larger during Mixed intervals, with dialogue-level bootstrap confidence intervals excluding zero for these differences.The authors connect this pattern to the difficulty of representing divergent participant emotions with a single group-level value.Source: §6 · p. 8
- Future work includes participant- and group-level state annotations, identifying reliable visual cues complementary to audio, and efficient streaming architectures for real-time operation.The sliding-window inference used here is computationally expensive.Source: §6 · p. 8
Safe and Responsible Innovation Statement
The authors flag group surveillance as a potential misuse, report strict dataset management and informed consent, and note possible cultural bias because participants came from a specific cultural group.
Safe and Responsible Innovation Statement
- The authors flag group surveillance as a misuse concern, report informed consent and strict dataset management, and note possible cultural bias because participants came from one cultural group.Source: Safe and Responsible Innovation… · p. 8