HumanOmniV2: From Understanding to Omni-Modal Reasoning with Context

1 Introduction

• Existing multimodal reasoning models can miss global context or crucial multimodal details, leading to misinterpretations and incorrect answers.Examples include using only part of a video or attending to a facial reaction while ignoring speech.Source: §1 · p. 2
• HumanOmniV2 requires a context summary before reasoning and uses context and logical rewards alongside format and accuracy rewards.An LLM judges context consistency against a reference and whether reasoning integrates multimodal information with logical analysis.Source: §1 · pp. 2–3
• HumanOmniV2 achieves the best performance among open-source omni-modal models, scoring 58.47% on Daily-Omni, 47.1% on WorldSense, and 69.33% on IntentBench.The reported benchmarks include the newly introduced IntentBench.Source: §1 · p. 3
• IntentBench evaluates human intentions and emotions in complex scenes using questions that require both auditory and visual cues.It contains 633 videos and 2,689 questions.Source: §1 · p. 3
• The omni-modal reasoning training dataset includes multimodal context information and reasoning paths for cold-start training and reinforcement learning.The data includes understanding tasks involving images, videos, and audio.Source: §1 · pp. 2–3

Omni-Modal Large Language Model and Benchmarks

• Omni-modal models add audio and video capabilities through approaches including modality alignment, memory, and real-time speech interaction.Examples include MiniCPM-o 2.6 and Ocean-Omni-1.5 [17] [2], Ola [3], VITA1.5 [4], and IXC2.5-OL [18].Source: §2 · p. 3
• Many multimodal benchmarks cover images or videos alone, while some evaluate both audio and video but focus on straightforward tasks or single-modality clues.Daily-Omni [14] and WorldSense [15] include general scenarios where some questions rely only on video or audio.Source: §2 · p. 3

Reinforcement Learning for MultiModal Reasoning

• Multimodal reasoning RL methods use varied strategies, including self-generated paths, rule-based rewards, and progressive thinking suppression.Vision-R1 [25] applies GRPO to a 10k multimodal math dataset, while R1-VL [26] uses rule-based rewards.Source: §2 · p. 3
• Visionary-R1 [28] and Observe-R1 [29] have models observe visual inputs before reasoning, but focus on vision and do not comprehensively evaluate context.The paper contrasts this scope with omni-modal reasoning for complex human intentions and general omni-modal benchmarks.Source: §2 · p. 4

3 IntentBench

• IntentBench addresses a gap in reasoning benchmarks that primarily focus on STEM tasks where audio cues are typically unnecessary.It centers on social interactions in videos, including intention, emotion, and deception.Source: §3 · p. 4
• The benchmark curates material from Social-IQ 2.0, EMER, and MDPE to assess human social understanding.Social-IQ 2.0 includes complexity annotations; EMER provides emotion explanations; MDPE concerns deception in interviews.Source: §3 · pp. 4–5
• For Social-IQ 2.0, the authors select 300 videos and 2,356 questions, modify easy options, and manually verify that questions require multimodal information.Text-only GPT-4o performance falls from 75.71% on the original testing set to 60.02% after modifications.Source: §3 · p. 5
• For EMER, the authors organize emotion descriptions into hierarchical categories and create multiple-answer questions using distractor terms from other groups.The testing set contains 133 videos, and performance is assessed with F1-score.Source: §3 · p. 5
• The MDPE subset contains 200 videos selected from confidence-rated deception interviews, including uncertain, confident, and no-deception cases.The selection includes 60 uncertain-deception clips, 20 confident-deception clips, and 120 no-deception clips.Source: §3 · p. 5
• IntentBench comprises 2,689 questions and 633 videos, with each question requiring analysis of both visual and audio cues from the videos.Figure 2 shows examples of IntentBench.Source: §3 · p. 5

4.1 Preliminary

• GRPO uses token-level loss, removes question-level normalization, and dynamically reduces the KL penalty during the first S iterations.The objective also uses a clipping bound and a KL penalty coefficient.Source: §4.1 · pp. 5–6

4.2 Shortcuts and Context in Multimodal Reasoning

• Vanilla GRPO can miss global context or shortcut multimodal cues; the method requires context interpretation before reasoning and multimodal integration throughout reasoning.Responses use <context>, <think>, and <answer> tags, with reflection and review encouraged during reasoning.Source: §4.2 · p. 6

4.3 Context Reward and Reasoning Logical Reward with Casual Mask

• LLM-judged context and logical rewards assess context quality and multimodal reasoning, and are masked to corresponding tokens and normalized separately.The logical assessment considers multimodal information, reflection, confirmation, and deduction.Source: §4.3 · p. 7
• Accuracy rewards use answer similarity for open-ended questions, accuracy or F1 for multiple-choice answers, 1 − WER for OCR/ASR, and numerical equality checks.Single-answer multiple choice uses accuracy; multiple-answer questions use F1.Source: §4.3 · p. 7

4.4 Training Recipe

• Training begins with long-CoT image/video cold start, then Stage 1 uses context-annotated samples and LLM rewards, while Stage 2 uses Video-R1 and OmniInstruct with format and accuracy rewards.Stage 1 samples eight completions and retains samples with accuracy in (0, 0.75); Stage 2 applies the same sampling and retention range.Source: §4.4 · pp. 7–8

5.1 Experimental Settings

• Training uses 24K rewritten video-audio examples; evaluation covers IntentBench, Daily-Omni, and WorldSense, with comparisons including SFT(CoT), no-extra-reward training, and vanilla GRPO.Video evaluation samples at 2 FPS, up to 32 frames.Source: §5.1 · p. 8

5.2 Main Results

• HumanOmniV2 achieves the best open-source results in out-of-domain testing and the best results against methods and baselines on IntentBench.The comparisons cover WorldSense, Daily-Omni, and IntentBench; the supplied table passages identify the WorldSense and IntentBench tables but do not provide their cell values.Source: §5.2 · pp. 8–9, Table 2 · p. 9, Table 3 · p. 9
• In Daily-Omni's reasoning task, performance rises from 61.71 to 74.28; on WorldSense, the method trails Qwen2.5-Omni on Performance and Music.The authors also report that the method achieves the best in-domain results on IntentBench.Source: §5.2 · p. 9

5.3 Ablation Study and Analyses

• SFT(CoT) slightly trails Qwen2.5-Omni; the authors suggest its smaller CoT dataset and possible overlap with Qwen2.5-Omni's training data as explanations.The cited discussion says the dataset is much smaller than Qwen2.5-Omni's 1,200 billion tokens and describes the overlap explanation as possible.Source: §5.3 · p. 9
• The context reward targets context quality, while the logical reward targets complex reasoning that integrates multimodal input; removing the additional rewards leaves performance below the full method.Source: §5.3 · pp. 9–10
• Without context and extra rewards, performance is 66.72, below the full method, although it remains above Qwen2.5-Omni.The authors say context provides comprehensive understanding for subsequent reasoning.Source: §5.3 · p. 10

• HumanOmniV2 summarizes global context before reasoning to address insufficient context understanding and overlooked multimodal inputs.Source: §6 · p. 10
• IntentBench is introduced as an omni-modal benchmark focused on human intentions and emotions in complex social interactions.Source: §6 · p. 10
• The method surpasses other open-source omni-modal methods and vanilla GRPO across various benchmarks.Source: §6 · p. 10

• The method reduces the probability of shortcut and context errors but does not completely solve them.Source: §7 · p. 11
• The response format might restrict correction of context information during subsequent thinking.Source: §7 · p. 11
• Experiments are primarily based on a 7B model, so consistent conclusions at larger scales are not guaranteed.Source: §7 · p. 11
• Shorter response length restricts the model's ability to reflect thoroughly on multimodal information.Source: §7 · p. 11

8.1 More visualizations of our method

• Figures 6–9 visualize full-method and comparison outputs on IntentBench, illustrating global-context comprehension and fine-grained audio-video cue capture.Source: §8.1 · p. 11, Figure 6 · p. 11, Figure 7 · p. 12

8.2 The prompts used in our method

• The system prompt guides context understanding, self-reflection, verification, and output format; assessment prompts judge context coverage and multimodal integration in reasoning.Source: §8.2 · pp. 15–16

8.3 The Differences between IntentBench and Its Original Datasets.

• GPT-4o scores 75.71% on the original Social-IQ 2.0 test and 60.02% after easy options are replaced and questions are manually checked for multimodal relevance.The authors use text-only testing to identify challenging questions before refining options.Source: §8.3 · p. 16, Figure 13 · p. 17
• EMER questions use refined hierarchical emotion vocabulary and multiple-choice options with multiple answers, avoiding frequent LLM evaluation that the authors say can be unstable.Source: §8.3 · p. 16, Figure 14 · p. 17
• MDPE is reformatted as question-answer data and sampled into 200 deception clips: 60 uncertain-deception, 20 confident-deception, and 120 no-deception clips.Source: §8.3 · pp. 16–17