Source-linked AI summary
R1-Omni: Explainable Omni-Multimodal Emotion Recognition with Reinforcement Learning
Jiaxing Zhao, Xihan Wei, Liefeng Bo
TL;DR
Emotion recognition in video Omni-multimodal models requires visual and audio cues, but prior RLVR work had not explored this setting. The paper applies RLVR to HumanOmni and reports better reasoning, emotion recognition, and generalization, including stronger out-of-distribution robustness. It also identifies hallucinated reasoning and incomplete use of audio cues as remaining limitations.
Problem
Prior RLVR applications to multimodal models were limited to image-text modalities, leaving video emotion recognition with audio and dynamic visual content unexplored.
Method
The paper applies RLVR to HumanOmni and combines it with GRPO to optimize multimodal emotion-recognition reasoning using verifiable rewards.
Results
R1-Omni improves reasoning capability, emotion-recognition performance, and generalization, with reported strength on out-of-distribution data.
Takeaways & Limitations
Improved reasoning provides clearer insight into how visual and audio information contribute to emotion recognition.
Takeaways & Limitations
The model can hallucinate reasoning and does not always fully utilize tone and intonation cues from audio.
Abstract
from arXiv · showhide
In this work, we present the first application of Reinforcement Learning with Verifiable Reward (RLVR) to an Omni-multimodal large language model in the context of emotion recognition, a task where both visual and audio modalities play crucial roles. We leverage RLVR to optimize the Omni model, significantly enhancing its performance in three key aspects: reasoning capability, emotion recognition accuracy, and generalization ability. The introduction of RLVR not only improves the model's overall performance on in-distribution data but also demonstrates superior robustness when evaluated on out-of-distribution datasets. More importantly, the improved reasoning capability enables clear analysis of the contributions of different modalities, particularly visual and audio information, in the emotion recognition process. This provides valuable insights into the optimization of multimodal large language models.
1 Introduction
RLVR extends verifiable-reward optimization from language and vision settings toward video Omni-multimodal emotion recognition. R1-Omni applies it to visual-audio emotion recognition and reports stronger reasoning, understanding, and out-of-distribution generalization.
- RLVR uses rule-based, verifiable rewards to optimize large models efficiently and reliably, even with limited training data.
- Visual-RFT found that RLVR consistently outperformed SFT across nearly all classical computer-vision task categories.
- Prior MLLM applications of RLVR were limited to image-text modalities, leaving video models with audio and dynamic visual content unexplored.
- R1-Omni applies RLVR to HumanOmni for emotion recognition, where visual and audio modalities provide critical prediction cues.
- RLVR improves R1-Omni’s reasoning, emotion-understanding performance, and generalization, especially in out-of-distribution scenarios.
2 Preliminaries
RLVR optimizes outputs using directly verifiable rewards while KL regularization keeps the policy near a reference model. The paper combines this framework with GRPO, which compares groups of responses without a critic, for multimodal emotion recognition.
- Reinforcement Learning with Verifiable Rewards: RLVR evaluates generated outputs with a verification function instead of using an intermediate reward model trained on human preferences.
- Reinforcement Learning with Verifiable Rewards: For question q, policy πθ generates response o, and R(q, o) assigns a binary score according to ground-truth agreement.
- Reinforcement Learning with Verifiable Rewards: The RLVR objective balances verifiable reward maximization against proximity to πref through KL-divergence regularization controlled by β.
- Application to Emotion Recognition: The method trains on 15,306 MAFW and DFEW video samples annotated with emotion categories but without explicit reasoning supervision.
- Group Relative Policy Optimization: GRPO generates multiple responses, evaluates their rewards, and normalizes them relative to the group to prioritize higher-quality outputs without an additional critic.
- Integration: Combining GRPO with RLVR is intended to leverage both methods for improved reasoning, generalization, and emotion-recognition capabilities.
3 R1-Omni
R1-Omni is initialized with explainable multimodal emotion-reasoning examples before RLVR optimization. Its RLVR framework rewards both correct emotion labels and outputs that follow the required reasoning-and-answer format.
- Cold Start with EMER Dataset: 580 video samples initialize HumanOmni-0.5B through cold-start fine-tuning on EMER and manually annotated HumanOmni data.The combined set contains 232 EMER samples and 348 manually annotated HumanOmni samples.
- Cold Start with EMER Dataset: EMER provides multimodal visual-audio data with detailed annotations describing the reasoning behind emotion recognition.These annotations support initial learning about how visual and audio cues contribute to predictions.
- Cold Start with EMER Dataset: R1-Omni reasoning integrates visual appearance, audio characteristics, and speech content to analyze emotion.An example combines facial expressions, vocal tone and pace, and subtitle content in its reasoning process.
- RLVR: RLVR generates candidate responses from video and audio inputs, with each response accompanied by a detailed reasoning process.The pipeline is designed to expose how the model integrates multimodal evidence before producing an emotion prediction.
- RLVR: The total reward combines accuracy and format rewards, requiring correct emotion predictions and the specified structured output.Accuracy is scored against ground truth, while formatting requires reasoning in <think></think> tags and the final label in <answer></answer> tags.
- RLVR: Format reward is binary: outputs satisfying the formatting constraints receive 1, and all other outputs receive 0.This structured format facilitates downstream analysis and evaluation.
4 Experiments
The experiments compare R1-Omni with supervised and base-model baselines on emotion-recognition datasets, evaluating reasoning, multimodal understanding, and generalization. R1-Omni provides more coherent explanations and achieves stronger in-distribution and out-of-distribution performance.
- Experimental Setup: R1-Omni is compared with HumanOmni-0.5B, EMER-SFT, and MAFW-DFEW-SFT in the emotion-recognition experiments.MAFW-DFEW-SFT is trained directly on the MAFW and DFEW training sets.
- Enhanced Reasoning Capability: R1-Omni generates more coherent, accurate, and interpretable reasoning than the evaluated baselines on MAFW and DFEW.HumanOmni-0.5B and MAFW-DFEW-SFT show limited reasoning, while EMER-SFT is less coherent and prone to hallucinations.
- Improved Understanding Capability: R1-Omni consistently achieves the highest UAR and WAR scores on both MAFW and DFEW.The evaluation uses UAR and WAR under an open-vocabulary emotion-testing protocol.
- Improved Understanding Capability: 65.83% UAR and 56.27% WAR on DFEW exceed MAFW-DFEW-SFT’s 60.23% UAR and 44.39% WAR.These values are reported for R1-Omni and the SFT baseline on DFEW.
- Stronger Generalization Capability: 43.00% UAR and 44.69% WAR on the OOD RAVDESS dataset exceed the SFT model’s 29.33% UAR and 30.75% WAR.RAVDESS differs from MAFW and DFEW by featuring professional actors vocalizing lexically matched statements in a neutral North American accent.
5 Limitations
R1-Omni’s remaining limitations span subtitle recognition, grounding of reasoning outputs, audio-cue utilization, foundation-model capacity, and reasoning depth. The paper frames these issues as directions for further investigation.
- 5.1 Inaccurate Subtitle Recognition: Subtitle recognition remains inaccurate because HumanOmni, SFT, and RLVR do not explicitly optimize subtitle recognition.The paper suggests specialized subtitle datasets or advanced natural language understanding models as possible remedies.
- 5.3 Enhancing Audio Cue Utilization: Audio cues such as tone and intonation are sometimes underutilized relative to visual cues during emotion recognition.Future work should improve the extraction and integration of audio features.
- 5.4 Implications for Future Research: The foundation model’s inherent performance remains a critical determinant of overall success despite RLVR’s gains in reasoning and generalization.Suggested directions include larger-scale pretraining, more diverse datasets, and advanced architectural designs.
- 5.2 Hallucination in Reasoning Outputs: Reasoning hallucinations can arise from weak causal relationships in multimodal tokens and the lack of explicit supervision for reasoning content.The paper identifies hallucination detection and mitigation as important for improving reliability and usability.
- 5.4 Implications for Future Research: Current reasoning focuses mainly on observable visual and audio features rather than motivations, intentions, or internal emotional states.The paper proposes deeper psychological reasoning to improve emotional intelligence and capture complex emotional dynamics.