PaperlayerView exact revision on arXiv

Source-linked AI summary

OneEmo: A Unified Multimodal Reasoning Model for Emotion Perception, Understanding, and Interaction

Jiahao Huang, Zheng Lian, Jingyi Zhang, Zhide Chen, Xiaojiang Peng, Shaonan Wang

arXiv:2608.06013v1cs.HC

TL;DR

Existing emotion models largely specialize by task rather than jointly optimizing perception, understanding, and interaction. OneEmo addresses this with the EmoWorld-130K dataset and Emo-Chord multi-task reinforcement learning, achieving state-of-the-art results against similarly sized baselines across most benchmarks.

  • Problem

    Existing emotion approaches largely specialize by task instead of jointly optimizing perception, understanding, and interaction.

  • Method

    OneEmo combines EmoWorld-130K, an eight-task reasoning dataset, with Emo-Chord, a unified multi-task reinforcement learning strategy.

  • Results

    OneEmo achieves state-of-the-art results against similarly sized baselines across most benchmarks and highly competitive interaction performance.

  • Takeaways & Limitations

    The results support unified affective reasoning and inter-task synergy as a viable direction for multimodal emotional intelligence.

  • Takeaways & Limitations

    The framework relies partly on scripted cinematic data and does not yet cover broader capabilities such as continuous emotion forecasting or long-term empathetic companionship.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in emotional intelligence. However, prevailing research predominantly focuses on task-specific specialization, often neglecting inter-task synergy and leaving latent reasoning potential underexplored. To bridge this gap, we introduce OneEmo, a unified affective generalist capable of mastering emotion perception, comprehension, and interaction. For this purpose, we first construct EmoWorld-130K, a comprehensive dataset that distills specialized affective knowledge into explicit reasoning trajectories via a human-in-the-loop workflow. Supervised fine-tuning on this corpus reveals significant mutual benefits derived from multi-task learning. Second, to fully unlock the latent reasoning potential, we propose Emo-Chord, a novel reinforcement learning strategy that stabilizes optimization through unified multi-task reward allocation. Extensive experiments demonstrate that OneEmo achieves state-of-the-art performance against similarly sized baselines across most benchmarks. Notably, despite having significantly fewer parameters than commercial models, OneEmo delivers highly competitive results. This paper paves the way for more reliable and interpretable affective computing. The code is available at https://github.com/waHAHJIAHAO/OneEmo.

I. Introduction … B. Reinforcement Learning in MELMs

OneEmo addresses task fragmentation and limited latent reasoning in multimodal emotion models by unifying eight affective tasks through EmoWorld-130K and Emo-Chord. The proposed framework combines psychology-informed reasoning data with multi-task reinforcement learning to improve reasoning quality, cross-task synergy, interpretability, and benchmark performance.

  • I. Introduction: Affective computing applications span education, healthcare, and conversational agents, with MLLMs advancing emotion perception, understanding, and interaction.
  • I. Introduction: Early MELM approaches primarily use supervised fine-tuning on instruction-following datasets that map video inputs to emotion labels.
  • I. Introduction: AffectGPT and Emotion-LLaMA demonstrate supervised multimodal emotion modeling, but supervised paradigms remain limited in eliciting MLLMs’ latent reasoning potential.
  • I. Introduction: Existing approaches often specialize in individual emotion tasks, overlooking inter-task synergy despite the integration of perception, understanding, and interaction in human-computer interaction.
  • I. Introduction: OneEmo unifies eight affective tasks spanning emotion perception, understanding, and multi-turn emotion interaction.EmoWorld-130K provides reasoning trajectories across sentiment analysis, basic and fine-grained emotion recognition, intent recognition, humor detection, sarcasm detection, empathetic response generation, and emotional support conversation.
  • I. Introduction: EmoWorld-130K contains psychology-informed reasoning trajectories, while Emo-Chord combines on-policy exploration with off-policy imitation through unified multi-task reinforcement learning.Fine-grained reasoning rewards support high-quality inference and stabilize cross-task synergy.
  • B. Reinforcement Learning in MELMs: Reinforcement learning enhances MELM reasoning through emotion-perception optimization, rationale verification, and rewards based on empathy, credibility, and semantic similarity.
  • B. Reinforcement Learning in MELMs: Emo-Chord extends reinforcement learning to joint multi-task optimization, and together with EmoWorld-130K enhances MELM performance and interpretability.

III. Preliminary · IV. Methodology

The paper frames MLLM post-training through supervised fine-tuning and reinforcement learning, then applies these principles in OneEmo’s EmoWorld-130K training and Emo-Chord optimization pipeline. The preliminary formulation covers likelihood training, GRPO-based reward normalization, KL regularization, importance sampling, and relative-advantage computation.

  • III. Preliminary: SFT learns from curated domain-specific instruction–response pairs by minimizing token-level negative log-likelihood.The objective matches the target response distribution over multimodal contexts and target token sequences.
  • III. Preliminary: The multimodal input context is denoted by x, while y denotes the target token sequence.
  • III. Preliminary: RL optimizes a policy model through feedback from a reward-based credit-assignment system.OneEmo adopts GRPO, which estimates advantages by normalizing rewards across G sampled outputs for each input.
  • III. Preliminary: A KL-divergence penalty regularizes the updated policy πθ toward the reference policy πref to mitigate policy drift.The probability ratio r_i,t is computed by importance sampling between πθ and πref.
  • III. Preliminary: The intra-group relative advantage Â_i is obtained by normalizing raw reward R_i using the reward group’s mean and standard deviation.
  • IV. Methodology: OneEmo’s overall pipeline uses EmoWorld-130K as its training corpus and Emo-Chord as its optimization strategy.The methodology section elaborates on dataset construction and the proposed training pipeline.

A. EmoWorld-130K: Theory-driven Reasoning Dataset … 1) Training Strategy:

The paper builds EmoWorld-130K through theory-guided trajectory distillation, closed-loop verification, and expert-assisted revision to address fragmented affective datasets and underexplored multimodal reasoning. It then trains Emo-Chord with multi-task curriculum learning, auxiliary SFT stabilization, offline cold-start, online optimization, and structured reward components.

  • A. EmoWorld-130K: Theory-driven Reasoning Dataset: EmoWorld-130K addresses isolated affective datasets by modeling synergistic relationships across tasks and making MLLM reasoning more interpretable.Conventional datasets target isolated affective dimensions, neglect cross-task synergy, and leave latent reasoning potential underexplored.
  • 1) Theory-based Trajectory Distillation:: The dataset sources established multimodal emotion datasets and uses original annotations as anchors for structured reasoning trajectories.Seed-2.0-Lite generates trajectories under explicit prompt templates derived from psychological and cognitive frameworks.
  • 2) Automated Reverse Verification:: A closed-loop validation protocol tests whether gold labels can be deduced solely from distilled reasoning trajectories and flags inconsistencies with violation types and causal rationales.Flagged samples are prioritized for human annotation.
  • 3) AI-assisted Human Expert Re-annotation:: Three psychology postgraduates revised problematic trajectories using AI-generated repair suggestions, while random 10% task-level audits triggered iterative re-annotation when quality criteria were unmet.Sampling inspections were repeated until the entire dataset satisfied the quality standards.
  • B. Emo-Chord: Multi-task Collaborative Optimization: Emo-Chord combines training strategy and credit assignment with Task-Aware Linear Decay Gating to regulate reasoning length and mitigate hallucinations.These are identified as Emo-Chord’s two core modules plus an additional gating mechanism.
  • B. Emo-Chord: Multi-task Collaborative Optimization: Two-stage curriculum learning first jointly fine-tunes diverse multimodal tasks, then adds ESC and increases ERG data to strengthen multi-turn dialogue analysis.The curriculum is designed to validate whether synergistic learning across multi-level affective tasks improves overall performance.
  • 1) Training Strategy:: Emo-Chord integrates an auxiliary SFT loss to mitigate policy collapse and reward instability during reinforcement-learning optimization.A dynamic coefficient µ controls the relative contribution of the loss components.
  • 1) Training Strategy:: Training begins with offline cold-start and proceeds to online policy optimization on EmoWorld-130K, while rewards aggregate format, thought, and answer components.The hybrid pipeline is intended to preserve exploratory diversity, improve training stability, and enhance final performance; coefficients γf, γt, and γa aggregate the reward components.

2) Credit Assignment:

Emo-Chord assigns unified rewards across format, task-dependent reasoning, and final-answer quality for perception, understanding, and interaction. It combines rubric-based evaluation with task-aware answer scoring and length regularization to improve interpretability, accuracy, and robustness.

  • Credit Assignment: Task-dependent Rthought and Ranswer rewards distinguish emotion inference and intent modeling from user-state modeling and empathetic response generation.Perception and understanding target emotional states, dialogue styles, and latent intents, whereas interaction targets situational context, emotional state, causal analysis, and supportive responses.
  • Perception & Understanding: Perception and understanding score reasoning factuality against extracted visual facts on a discrete 1–5 scale and evaluate reasoning coherence with detailed LLM-judge rubrics.The factuality check is designed to prevent hallucinated or fabricated visual details from reducing response accuracy.
  • Interaction: Interaction rewards user-state fidelity and strategy alignment against gold references, with Rstate scored 1–5 by an LLM judge.Open-ended answer quality is measured by Sentence-BERT similarity between predicted and gold responses.
  • Credit Assignment: Emo-Chord decomposes credit assignment into format, thought, and answer rewards across perception, understanding, and interaction.Rformat enforces reasoning inside <think> and </think> tags followed immediately by the final answer.
  • Answer Regularization: A task-aware linear decay factor τ regularizes Ranswer using task-specific soft and hard length bounds because unconstrained verbosity can cause hallucinations and degrade simple-task performance.The method addresses heterogeneous emotion-task complexity and variable chain-of-thought length requirements.

3) Task-Aware Linear Decay Gating:

Task-aware linear decay gating computes decay factors for both reasoning trajectories and final answers, using their minimum to enforce brevity. The mechanism adapts answer granularity to task requirements while suppressing unbounded verbosity and inefficient reasoning.

  • Task-Aware Linear Decay Gating: The piecewise linear gate varies across the regimes lo ≤ lsoft, lsoft < lo < lhard, and lo ≥ lhard.The supplied formulation assigns distinct decay behavior across these three threshold regions.
  • Task-Aware Linear Decay Gating: Decay factors are computed separately for the reasoning trajectory and final answer, then combined through the minimum factor.The minimum modulates Ranswer to enforce brevity across both components.
  • Task-Aware Linear Decay Gating: The mechanism adaptively calibrates answer granularity to task requirements while suppressing unbounded verbosity and inefficient reasoning.The statistical procedure for determining the bounds is detailed in the Appendix.

V. Experimental Setup and Dataset … A. Main Results

OneEmo is evaluated under strict task-specific protocols and trained with a two-stage Emo-Chord setup. Across perception, understanding, and interaction, it achieves strong results against open-source and commercial baselines, including competitive human-likeness.

  • A. Implementation Details: OneEmo is trained on EmoWorld-130K with Emo-Chord using two NVIDIA H800 GPUs, AdamW, micro-batch size 1, and eight gradient-accumulation steps.The off-policy cold-start phase lasts five epochs at a learning rate of 1 × 10−5.
  • A. Implementation Details: The reinforcement-learning phase lasts two epochs, anneals µ from 0.5 to 0.02, and uses GRPO with learning rate 2 × 10−6, group size 8, and KL coefficient 0.06.Reward coefficients are γf = 0.5, γt = 0.3, and γa = 1.0; the backbone is Qwen3.5-4B with LoRA fine-tuning.
  • B. Benchmarks: Evaluation uses disjoint cross-corpus data for emotion perception and official dataset splits for understanding and interaction, with test sets held out for final evaluation.Training and validation data constitute EmoWorld-130K.
  • C. Evaluation Metrics: The study reports task-specific metrics, including WAF for MSA, MIR, MSU, and MHU, Hit Rate for B-MER, and EW for OV-MER.Interaction evaluation combines automatic 1–5 Likert-scale judgments with blind human pairwise comparisons.
  • A. Main Results: OneEmo surpasses all open-source generalists in emotion perception and beats GPT-5-Mini and MiMo-v2.5 on B-MER and OV-MER, trailing MSA by only 0.24 points despite a 70× parameter gap.The comparison includes 16B-scale multimodal generalists such as Cosmos3-Nano.
  • A. Main Results: On complex 20- and 30-class intent recognition, OneEmo outperforms Gemini-3.1-Pro by 3.08 points while operating at an efficient 4.5B-parameter scale.It comprehensively surpasses open-source affective specialists and generalists, while trailing Gemini-3.1-Pro by 1.63 points overall.
  • A. Main Results: OneEmo outperforms all open-source interaction baselines, showing superior empathy in empathetic dialogues and higher overall effect in emotional-support scenarios.Reinforcement-learning optimization causes a marginal decline in informativeness and contextual coherence, attributed to limited exhaustiveness in human-annotated reference responses.
  • A. Main Results: Human evaluation records an 84.33% peak win rate for human-likeness; OneEmo surpasses GPT-5-mini on ESC and ties statistically with 310B-parameter MiMo-v2.5.Win/loss differences are significant across dimensions at p-value <0.01, except for the tie against MiMo-v2.5.

B. Impact of Reward Functions · C. Impact of Different Training Strategy

Reward ablations show that answer, thought, and format rewards jointly improve OneEmo’s multi-task performance, while the thought reward adds semantic grounding across perception and interaction. Training-strategy studies further show that joint RL and SFT without off-policy cold-start severely degrades performance, especially on complex tasks.

  • B. Impact of Reward Functions: The full reward configuration yields comprehensive performance enhancements across OneEmo’s evaluated tasks.
  • B. Impact of Reward Functions: 2.97 points is the average decline across eight tasks after ablating the answer reward.
  • B. Impact of Reward Functions: 1.16 and 2.7 points are the average decrements from removing the thought and format rewards, respectively.
  • B. Impact of Reward Functions: The thought reward enforces multimodal factual consistency and reasoning coherence in perception, while supervising user-state modeling and strategy selection in interactive scenarios.
  • B. Impact of Reward Functions: Fine-grained semantic constraints prevent reliance on spurious correlations and ground final predictions in logically sound reasoning processes.
  • C. Impact of Different Training Strategy: Direct joint RL and SFT optimization without prior off-policy cold-start severely degrades performance, particularly on complex tasks such as MIR and MHU.

D. Task Synergy Analysis … IX. Limitations and Future Works

The paper analyzes task interference and cross-model generalization, then summarizes OneEmo’s contributions, ethical safeguards, and limitations that motivate future evaluation in authentic interactions.

  • D. Task Synergy Analysis: Naive joint SFT causes task interference, improving trained tasks while degrading untrained domains and failing to produce effective cross-task synergy.The analysis compares incremental task integration from basic perception to empathetic generation across SFT-v1 to v3.
  • D. Task Synergy Analysis: Explicit thought augmentation in SFT-v3 consistently outperforms the direct-reasoning SFT-v4 baseline, indicating benefits from incorporating explicit reasoning.The passage reports a consistent overall advantage for SFT-v3 over SFT-v4.
  • E. Generalization on Different Backbones: Across all eight evaluated affective tasks, Qwen3-VL-4B and InternVL-3.5-4B show consistent pre-training-to-post-training improvements, demonstrating EmoWorld-130K’s cross-model generalization.Both backbones exhibit strict Pareto improvements while avoiding inter-task capability conflicts.
  • VII. Conclusion: OneEmo combines EmoWorld-130K’s explicit reasoning trajectories across eight emotion tasks with Emo-Chord’s multi-task reinforcement learning and designed reward allocation.Emo-Chord integrates online-offline collaborative policy optimization.
  • VIII. Ethics Statement: EmoWorld-130K uses publicly licensed datasets without new human-subject data collection, with annotations produced by compensated volunteers under anonymity and informed consent.The dataset’s data-collection and annotation practices are presented as ethical safeguards.
  • VIII. Ethics Statement: OneEmo is a research prototype rather than a substitute for psychiatric diagnosis or treatment, and unsupervised clinical deployment is explicitly discouraged.The paper states that emotional-support tasks use clinical frameworks only as reasoning scaffolding.
  • VIII. Ethics Statement: Real-world applications require strict crisis-referral guardrails, while the dataset and model will be released under a restrictive license enforcing ethical and safety boundaries.These safeguards are required because the system is not intended for unsupervised clinical deployment.
  • IX. Limitations and Future Works: Future benchmarks should prioritize authentic, in-the-wild interactions because emotion is context-sensitive and culturally diverse, whereas current explainable datasets often use scripted cinematic data.The paper identifies this mismatch as a limitation of current evaluation resources.

I. Appendix Overview … A. Perception and Understanding Task Thought Reward Prompt

The appendix extends the paper with additional details, prompting protocols, and qualitative analysis. It further documents task-length handling, training-data mixing, and rubric-based evaluation of perception and understanding reasoning.

  • I. Appendix Overview: The appendix covers additional details, prompting protocols, and qualitative analysis beyond the main paper.It identifies Additional Details as section II, Prompting Protocols as section III, and Qualitative Analysis as section IV.
  • A. Distribution of Data Length: Perception and understanding tasks typically produce concise outputs, whereas interaction tasks require longer reasoning trajectories and conversational responses.Dynamic task-specific thresholds accommodate these heterogeneous output-length biases.
  • A. Distribution of Data Length: Most EmoWorld-130K samples remain within a penalty-free length region, while soft and hard thresholds penalize pathological verbosity.The thresholds are intended to preserve the natural output distribution while selectively penalizing excessive length.
  • B. Training Data Ratio: Training data sampling shifts from perception-focused Stage 1 to interaction-amplified Stage 2, while RL mix SFT adds ESC and balances task levels.This schedule targets multimodal grounding, affective synergy, and avoidance of catastrophic forgetting.
  • III. Prompting Protocols for Automated Evaluation and Rubric Scoring: The appendix provides complete automated-evaluation and rubric-scoring prompt templates in prompts.py through the project’s GitHub repository.The displayed material is a condensed version of the key instructions.
  • A. Perception and Understanding Task Thought Reward Prompt: The perception-and-understanding thought rubric evaluates multimodal reasoning using provided information, separating objective visual behaviors from noncontradictory subjective inferences.It assesses visual-fact consistency, penalizes fabricated or contradictory observations, and evaluates whether reasoning coherently supports the final answer.
  • A. Perception and Understanding Task Thought Reward Prompt: Reasoning-answer coherence is judged by whether the final answer naturally follows from the reasoning, emphasizing logical support rather than answer correctness.The rubric distinguishes convergence for single-label tasks from logical support for open-label tasks.

B. Interaction Task Thought Reward Prompt · 2. Response Strategy Alignment

The interaction-task thought reward evaluates candidate reasoning against reference reasoning for semantic alignment in user-state understanding and response strategy. It scores whether reasoning captures the user’s situation and follows the intended counseling strategy, sequencing, and priorities.

  • B. Interaction Task Thought Reward Prompt: The reward compares candidate and reference reasoning for empathetic and counseling tasks, emphasizing semantic alignment rather than wording style.This evaluation targets ERG/ESC interaction tasks.
  • B. Interaction Task Thought Reward Prompt: User-state alignment checks whether reasoning captures the user’s situation, emotional state, core conflict, needs, and drivers.Equivalent paraphrases are accepted when they preserve the meaning.
  • B. Interaction Task Thought Reward Prompt: The user-state rubric ranges from 5 for fully aligned reasoning without material drift to 1 for substantially misaligned descriptions.Intermediate scores represent mostly aligned, partially aligned, or weakly aligned coverage of the user’s state.
  • 2. Response Strategy Alignment: Response-strategy alignment compares the implied strategy with the reference response plan.The comparison focuses on the intended response direction.
  • 2. Response Strategy Alignment: The strategy assessment examines intended moves, sequencing, and prioritization, including validation, exploration, reassurance, and reframing.These examples define the types of supportive moves being compared.
  • 2. Response Strategy Alignment: The strategy rubric ranges from 5 for matching the reference strategy and support priorities to 0 for no usable or irrelevant strategy.Intermediate scores capture minor omissions, missing components, superficial overlap, or counterproductive direction.

C. Interaction Task Evaluation Prompt · IV. Qualitative Analysis

The interaction evaluation prompts use 1–5 rubrics for emotional understanding, coherence, informativeness, support skill, and overall effectiveness. Qualitative intent-recognition analysis shows OneEmo grounding multimodal cues correctly while baselines suffer label-mapping errors or contextual hallucinations.

  • C. Interaction Task Evaluation Prompt: ERG responses are evaluated on a 1–5 scale for recognition of the seeker’s emotional state.The rubric ranges from missing or contradicting emotion to deeply understanding and appropriately responding to it.
  • C. Interaction Task Evaluation Prompt: ERG coherence and consistency measure whether responses remain relevant, logically connected, and aligned with the seeker’s context and intent.Scores range from incoherent or contradictory to perfectly coherent and seamlessly aligned.
  • C. Interaction Task Evaluation Prompt: ERG informativeness measures whether responses provide substantive, specific, and genuinely useful content rather than generic or templatized language.The highest score denotes highly informative, uniquely tailored responses offering insight, relevant questions, or useful framing.
  • C. Interaction Task Evaluation Prompt: ESC evaluates emotional understanding by scoring how precisely and warmly responses capture and validate the patient’s emotional state.The rubric progresses from dismissive or invalidating responses to deeply human, attuned, and validating ones.
  • C. Interaction Task Evaluation Prompt: ESC support skill measures the selection, timing, and execution of conversational support strategies.The rubric penalizes harmful or poorly executed techniques and rewards appropriate strategies deployed with good timing.
  • C. Interaction Task Evaluation Prompt: ESC overall effectiveness measures whether a response leaves the seeker feeling heard, validated, and emotionally helped.Scores range from counter-therapeutic responses likely to worsen the seeker’s state to exceptionally effective responses producing deep understanding and relief.
  • IV. Qualitative Analysis: OneEmo accurately grounds visual and textual cues, including characters and objects, to logically deduce the correct intent.This qualitative comparison concerns reasoning trajectories on a multimodal intent-recognition task.
  • IV. Qualitative Analysis: InternVL-3.5-4B makes a label-mapping error, whereas Cosmos3-Nano fabricates an elevator confrontation and predicts an erroneous taunt.InternVL-3.5-4B correctly interprets the semantic inquiry but fails to map it to the correct label; Cosmos3-Nano exhibits severe contextual hallucinations.
Loading 2608.06013v1…