Source-linked AI summary
From Expression to Reaction: Role-aware Visual Transfer and Stimulus-guided Reasoning for Interlocutor Emotion Recognition
Wei Wang, Zhaowu Li, Jianjie Luo, Fu Lee Wang, Lap-Kei Lee, Zhenguo Yang
TL;DR
Listener emotion recognition lacks labeled listener training examples, and speaker labels may not provide reliable visual supervision. RASG builds listener-centered visual supervision and uses speaker context only for ambiguous visual predictions. It achieves 76.25% WAF on MER-Cross, 17.37 percentage points above the official CLIP-large baseline.
Problem
Listener emotion recognition lacks labeled listener examples, and speaker labels may reflect audio or text despite weak facial evidence.
Source: §1 · p. 1Method
RASG filters visually reliable speaker supervision, bootstraps listener pseudo-labels, and applies constrained speaker-context reasoning only to uncertain visual predictions.
Source: §1 · p. 2Results
Takeaways & Limitations
The results show complementary value from role-aware visual transfer and constrained stimulus reasoning.
Source: §5 · p. 6The setup assumes an unlabeled conversational pool whose roles and emotion labels are unknown.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
In this paper, we propose a Role-aware Stimulus-guided (RASG) framework for interlocutor emotion recognition, which predicts listener emotions from listener-only videos and speaker-only audios. RASG consists of Role-aware Visual Transfer (RVT) and Stimulus-guided Boundary Reasoning (SBR) modules, which address supervision mismatch due to the lack of labeled listener data and ambiguity among visually similar listener reactions whose interpretation depends on speaker context, respectively. More specifically, RVT selects speaker samples whose facial expressions support their emotion labels. It then filters listener tracks and uses reliable pseudo-labels to train a listener-centric visual expert. SBR uses a two-class language reasoner only when the visual model is uncertain. It treats speaker audio and text as context rather than direct emotion evidence to distinguish similar listener reactions. Experiments conducted on MER-Cross dataset shows that RASG achieves 76.25\% on MER-Cross and improves the performance of the baseline over 17\%. Our team ranks second in Track 1 (MER-Cross) of the MER Grand Challenge at ACM MM 2026.
1 Introduction
Interlocutor emotion recognition predicts listener emotion from listener video and speaker audio and text, but speaker labels do not always provide reliable visual supervision. RASG builds listener-centered visual supervision and uses speaker context to resolve ambiguous visual predictions.
1 Introduction
- Listener emotion recognition must address absent labeled listener examples, visually unreliable speaker labels, and speaker audio and text that provide eliciting context rather than direct listener evidence.The task infers listener emotion from listener video, speaker audio, and the corresponding transcript.Source: §1 · p. 1
- Role-aware Visual Transfer filters speaker samples whose facial evidence conflicts with their emotion labels, then bootstraps listener supervision from unlabeled non-speaking tracks.Accepted listener pseudo-labels are constrained by confidence, temporal, and cross-view consistency.Source: §1 · p. 2
- Stimulus-guided Boundary Reasoning uses speaker audio and text only for low-margin visual predictions, restricting reasoning to the two most plausible visual classes.The speaker content is treated as eliciting context, not as direct evidence of the listener's emotion.Source: §1 · p. 2
- In extensive experiments on MER-Cross, RASG achieves 76.25% WAF and outperforms the official CLIP-large baseline by 17.37 percentage points.Source: §1 · p. 2
2 Related Work
Prior work models conversational context, pseudo-supervision, and language-based emotion reasoning, but its assumptions leave listener emotion recognition distinct from speaker-centered prediction. The subsections review these strands and their limits.
2.1 Conversational Emotion Recognition and Listener Modeling
- Conversational models encode speaker context and relations, while listener-centered prior work focuses on synthesizing reactions rather than recognizing emotions in authentic listener videos.The cited listener-motion approaches require paired speaker-listener data.Source: §2.1 · p. 2
2.2 Visual Transfer with Reliable Pseudo-supervision
- Existing pseudo-labeling methods use confidence, consistency, class balance, or model agreement, but do not verify visual observability or whether supervision matches the listener role.Speaking status alone cannot establish that a non-speaking track is a valid listener reaction or reliable emotion supervision.Source: §2.2 · p. 3
2.3 Contextual Emotion Reasoning with Large Language Models
- Language-model emotion methods generally reason across modalities as though they describe one subject, while role-specific constraints are needed to separate listener evidence from speaker context.Prompt-bias calibration does not impose that role distinction.Source: §2.3 · p. 3
3.1 Problem Definition
The task predicts a listener's emotion from listener video and the corresponding speaker audio and transcript. The setup distinguishes their roles and includes unlabeled conversational samples.
3.1 Problem Definition
- The training setup includes labeled speaker samples and an unlabeled conversational pool whose roles and emotion labels are unknown.The speaker samples comprise video, audio, transcript, and speaker-emotion labels.Source: §3.1 · p. 3
- At inference, listener video is paired with the corresponding speaker audio and transcript to predict the listener's emotion.The modalities describe two participants rather than one.Source: §3.1 · p. 3
- The video directly captures the listener's reaction, whereas speaker audio and text characterize the preceding stimulus.The label set is neutral, angry, happy, sad, worried, and surprise.Source: §3.1 · p. 3
3.2 Overview of the Framework
RASG addresses missing listener labels and ambiguous visual reactions by separating visual transfer from context-guided boundary reasoning. RVT builds listener-oriented visual supervision; SBR uses speaker context for ambiguous predictions.
3.2 Overview of the Framework
- RVT selects visually reliable speaker supervision and progressively transfers it to listeners, while SBR uses speaker audio and text as eliciting stimulus for ambiguous visual predictions.SBR resolves context-dependent boundaries without overriding confident visual evidence.Source: §3.2 · pp. 3–4
3.3 Role-aware Visual Transfer
Role-aware Visual Transfer addresses the mismatch between speaker labels and listener-visual supervision by filtering speaker examples, bootstrapping labels on non-speaking listener tracks, and training a listener-focused visual expert.
3.3 Role-aware Visual Transfer
- RVT selects speaker labels supported by facial behavior, trains an initial visual teacher, and uses it to annotate strictly filtered listener tracks for listener-domain adaptation.The retained speaker labels are intended to be inferable without audio or text.Source: §3.3 · p. 4, §3.3.1 · p. 4
3.3.1 Visually Aligned Speaker Supervision.
- A heterogeneous vision-language committee tests whether each original speaker label is recoverable from visual evidence alone, retaining visually aligned supervision.The agreement threshold is based on consensus across committee models.Source: §3.3.1 · p. 4
3.3.2 Listener-domain Bootstrapping.
- TalkNet detects speaking activity in 124,802 unlabeled videos, and face tracks that remain non-speaking throughout the speaker utterance are retained as listener samples.Role filtering and confidence filtering together create listener-domain supervision.Source: §3.3.2 · p. 4
3.3.3 Listener Visual Expert Training.
- The listener visual expert combines a CLIP-LoRA branch with an OpenFace behavior branch, aggregating their predictions at the probability level.Confidence-weighted adaptation downweights uncertain pseudo-labels, while probability-level combination preserves confidence and competing candidates for reasoning.Source: §3.3.3 · p. 4
3.4 Stimulus-guided Boundary Reasoning
Stimulus-guided Boundary Reasoning uses speaker audio and text as eliciting context, not direct evidence of listener emotion, and invokes constrained reasoning only for ambiguous visual predictions.
3.4 Stimulus-guided Boundary Reasoning
- SBR reserves speaker audio and text for uncertain visual predictions, restricting reasoning to the two most plausible visual classes.This is intended to resolve boundary ambiguity without replacing the visual model with unrestricted language-model inference.Source: §3.4 · p. 4
3.4.1 Low-margin Uncertainty Routing.
- A small decision margin signals a boundary case, and the DeepSeek reasoner is activated only when the margin falls below the routing threshold.A large margin indicates a clear visual preference that needs no contextual interpretation.Source: §3.4.1 · pp. 4–5
3.4.2 Interaction Evidence Construction.
- For routed samples, the evidence combines the listener-visual distribution and Top-2 candidates with the speaker-audio prediction, confidence, and transcript.Role labels distinguish listener-target evidence from speaker-stimulus context.Source: §3.4.2 · p. 5
3.4.3 Top-2 Pair-constrained Reasoning.
- The reasoner binds evidence by role, checks stimulus-reaction consistency, and selects only from the visual Top-2 candidates.Unrouted samples keep the visual Top-1 class, while routed samples remain restricted to the two visually supported candidates.Source: §3.4.3 · p. 5
3.5 Training Objectives
RASG uses standard cross-entropy objectives for its trainable visual components and introduces no task-specific loss; the visual objectives are optimized sequentially.
3.5 Training Objectives
- RASG adds no task-specific loss, and its inference-only DeepSeek reasoner introduces no training loss.All trainable visual components use standard cross-entropy objectives.Source: §3.5 · p. 5
- The visual teacher is trained on aligned speaker samples, followed by confidence-weighted pseudo-label supervision for listener-domain CLIP and the same listener supervision for the OpenFace behavior classifier.The CLIP and behavior objectives are joined in the complete visual training objective.Source: §3.5 · p. 5
- Although joined in the complete objective, the visual training objectives are optimized sequentially.The paper states that optimization follows the sequence shown in Figure 2(a).Source: §3.5 · p. 5
4 Experiments
Experiments evaluate RASG on MER-Cross, where listener video is compared with speaker-modality and multimodal baselines, and the full framework reaches 76.25% WAF. Ablations examine how role-aware visual transfer and selective stimulus reasoning contribute to that result.
4.1 Datasets
- The track provides 9,395 labeled speaker training samples and 574 listener test samples; 124,802 unlabeled MER videos support listener-domain bootstrapping without test-label use.Active-speaker detection retains non-speaking face tracks as candidate listener reactions.Source: §4.1 · p. 5
4.3 Quantitative Performance of the Approaches
- 76.25% WAF puts RASG 17.37 points above CLIP-large and 18.81 points above Top1; the team placed second in Track 1.The official comparison also reports CLIP-large at 58.88%, HuBERT-base at 35.74%, MacBERT-large at 33.68%, Top1 at 57.44%, and Top2 at 55.24%.Source: Table 1 · p. 6, §4.3 · p. 6, Table 2 · p. 6
4.2 Baselines
- Baselines span speaker audio, speaker text, listener video, multimodal fusion, and an OpenFace-plus-CLIP visual reference.The reference combines holistic CLIP features with OpenFace facial-behavior cues.Source: §4.2 · p. 5, Table 1 · p. 6
4.2.1 Evaluation Metrics.
- WAF is the official primary ranking metric, while ACC is secondary.The metric description identifies class-level precision, recall, and sample counts.Source: §4.2.1 · p. 6
4.2.2 Implementation Details.
- Listener faces are resized to 224 × 224; CLIP ViT-L/14 is adapted with LoRA, OpenFace stays fixed, and TalkNet finds non-speaking tracks.For uncertain visual predictions, DeepSeek receives the visual distribution, Top-2 labels, decision margin, speaker-audio prediction, and transcript, and is restricted to the Top-2 pair.Source: §4.2.2 · p. 6
4.4 Ablation Studies
- 64.28% to 73.42% WAF follows adding RVT to the OpenFace-plus-CLIP reference; adding SBR raises WAF to 76.25%, a total 11.97-point gain.The ablation adds RVT and SBR progressively.Source: Table 3 · p. 6, §4.4 · p. 6
4.5 Effectiveness of RVT and SBR with the CLIP-large Baseline
- 5.40 points come from OpenFace, 9.14 from RVT, and 2.83 from SBR in the 17.37-point gain over CLIP-large.RVT supplies the largest gain; SBR addresses ambiguous low-margin cases without overriding confident visual predictions.Source: Table 4 · p. 6, §4.5 · p. 6
- RVT addresses role-conditioned supervision before listener prediction, while SBR adds a smaller gain on remaining low-margin cases.SBR is restricted to the visual Top-2 pair, complementing rather than replacing the listener-oriented visual expert.Source: §4.5 · p. 6
5 Conclusion
RASG targets interlocutor emotion recognition without labeled listener data by combining listener-focused visual transfer with stimulus reasoning for uncertain predictions. On MER-Cross, it achieves 76.25% WAF, exceeds the official CLIP-large baseline by 17.37 percentage points, and places second in Track 1.
5 Conclusion
- RASG addresses interlocutor emotion recognition without labeled listener data.The framework combines role-aware visual transfer with constrained stimulus reasoning.Source: §5 · p. 6
- RVT filters visually consistent speaker samples and bootstraps pseudo-labeled listener tracks; SBR uses speaker context only for uncertain visual predictions.Source: §5 · p. 6
- 76.25% WAF on MER-Cross exceeds the official CLIP-large baseline by 17.37 percentage points, and RASG places second in Track 1.Source: §5 · p. 6