PaperlayerView exact revision on arXiv

Source-linked AI summary

AffectOmni: RL-Verifiable People-Centric Grounded Affective Reasoning for Social and Art-Related Scenes

Yibo Wang, Rui Yang, Jisheng Dang, Bimei Wang, Yitao Wu, Pengfei Cao, Wencan Zhang, Hong Peng, Bin Hu, Tat-Seng Chua

arXiv:2608.26193v1cs.AI

TL;DR

AffectOmni targets shortcut behavior in affective reasoning, where models may neglect people-centric cues and produce weakly verifiable reasoning. It uses GRPO with people-centric and temporal rewards, comparative scoring, and post-hoc grounding of summarized evidence instructions through SAM3. Across three benchmarks, it reports consistent improvements over open-source 7B-scale baselines, including gains on emotion recognition and temporally sensitive tasks.

  • Problem

    Affective reasoning may overlook people-centric cues, while existing reward and judging approaches do not sufficiently enforce or discriminate evidence-grounded reasoning.

  • Method

    AffectOmni combines People Focus and Temporal Order rewards, within-group comparative scoring, and a Thinking Summarizer that converts reasoning into SAM3-grounded evidence instructions.

  • Results

    AffectOmni shows consistent improvements over open-source 7B-scale baselines on IntentBench, Daily-Omni, and WorldSense, including +4.66% on emotion recognition and +14.29% on temporally sensitive tasks.

  • Takeaways & Limitations

    The framework provides affective reasoning with people-centered evidence selection, temporal organization, and an externally auditable interface for checking whether reasoning claims can be localized.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) achieve strong performance on VQA and scene understanding, yet affective reasoning remains vulnerable to shortcut behavior. Models may predict correct answers while neglecting people-centric cues such as micro expressions and body language, which weakens traceability and external verification. Prior reinforcement learning approaches mainly reward context or logical coherence without explicitly enforcing attention to human evidence. In addition, LLM as a Judge scoring often suffers from score clustering, which reduces reward discriminability. We propose AffectOmni, a GRPO trained framework for verifiable affective reasoning. AffectOmni introduces People Focus and Temporal Order rewards to encourage people-centric evidence selection and temporally structured reasoning, and it adopts within-group comparative scoring to produce more stable and discriminative reward signals. For verification, a Thinking Summarizer converts free form rationales into executable evidence instructions, which are grounded into pixel level evidence regions via SAM3 to provide an externally auditable interface outside the training loop. Experiments on IntentBench, Daily Omni, and WorldSense show consistent improvements over open source 7B scale baselines, including gains of 4.66% on emotion recognition and +14.29% on temporally sensitive tasks. Code is available at https://github.com/eliot127825-rgb/AffectOmni_nobody.

I. INTRODUCTION

AffectOmni addresses shortcut behavior in affective reasoning by rewarding people-centric, temporally organized evidence and making reasoning externally verifiable. It combines fine-grained rewards, comparative scoring, and reasoning-to-evidence grounding, with reported gains over open-source 7B baselines.

  • Affective reasoning can rely on global or background cues while overlooking micro expressions and body language, weakening traceability and sometimes producing incorrect answers.
  • The paper frames trustworthy affective reasoning around people-centric evidence selection, discriminative reinforcement-learning rewards, and executable verification of reasoning claims.
  • The Thinking Summarizer converts affective reasoning into executable evidence instructions that SAM3 grounds in pixel-level regions for external verification.
  • AffectOmni introduces People Focus and Temporal Order rewards to encourage attention to human cues and organization of reasoning over time.
  • Within-group comparative scoring mitigates calibration drift and score clustering in LLM-as-a-Judge evaluation, producing more stable and discriminative optimization signals.
  • 4.66% on emotion recognition and 14.29% on temporally sensitive tasks are reported gains over open-source 7B-scale baselines across IntentBench, Daily-Omni, and WorldSense.

II. RELATED WORK

Prior work advances multimodal affect understanding, reinforcement-learning reward design, and visual grounding, but AffectOmni combines these directions into a people-centric, verifiable reasoning framework. Its method trains structured reasoning with GRPO and complementary rewards, then grounds claims into visual evidence.

  • II. RELATED WORK: Multimodal affect understanding has progressed from unimodal and multimodal recognition toward MLLM-based explainable, open-vocabulary, and temporally aligned reasoning.
  • II. RELATED WORK: Existing affective reasoning methods often exhibit shortcut learning, motivating approaches that connect reasoning with fine-grained visual evidence.
  • II. RELATED WORK: Prior reinforcement-learning work applies GRPO across visual, video, omni-modal, and audio-visual reasoning, while reward designs target context, step-level, or consistency signals.
  • II. RELATED WORK: Visual grounding aligns language with image or video regions and has been used to support faithful grounding, hallucination mitigation, and multimodal fact checking.
  • II. RELATED WORK: AffectOmni trains a policy to generate structured context, thinking, and answer segments from multimodal inputs under GRPO.
  • II. RELATED WORK: Its framework combines people-centric reward shaping, within-group comparative scoring, and post-hoc grounding of reasoning claims into visual evidence.

B. People-Centric Fine-Grained Reward Shaping

AffectOmni replaces coarse context-level evaluation with task-aligned rewards for people-centric cues and temporal affective structure. Joint evaluation and causal masking reduce overhead and prevent reward leakage into answer tokens.

  • People Focus Reward: People Focus Reward evaluates facial expressions, body motion, and interpersonal interaction rather than only global context.The criteria include micro expressions, gestures, posture changes, and affective exchanges between interlocutors.
  • Temporal Order Reward: Temporal Order Reward assesses temporal markers and coherence of the affective trajectory.Examples include “initially,” “then,” and “finally,” alongside transitions such as surprise to relief.
  • Implementation Details: A joint prompt returns both reward scores in one call, while causal masking restricts them to context and think segments.This design reduces API overhead and prevents reward leakage into the answer segment.

C. Within-Group Comparative Scoring

AffectOmni replaces isolated absolute judging with within-group comparison, producing differentiated candidate scores and group-relative advantages for reinforcement learning.

  • Comparative Scoring Mechanism: Within-group comparative scoring presents all G candidates jointly and requires relative comparison before assigning scores.The judge produces differentiated rankings rather than evaluating each response in isolation.
  • Comparative Scoring Mechanism: Each candidate receives a relative score si ∈[1, 10] under the comparative scoring formulation.The comparison prompt explicitly instructs the judge to rank candidates distinctly.
  • Advantage Computation: Comparative scores estimate judge-based reward components rather than forming a separate reward term.Overall candidate rewards remain weighted sums of multiple components.
  • Advantage Computation: A within-group mean baseline produces candidate advantages, with optional standardization for scale consistency and training stability.The standardization divides by σg + ϵ.
  • Distributed Implementation: Distributed training gathers candidates to rank 0 for comparative judging and broadcasts the resulting scores to all ranks.Backpropagation resumes after score synchronization across devices.

D. Reasoning to Evidence Grounding Framework

AffectOmni converts long free-form reasoning into structured evidence and grounds its instructions in video pixels for post-hoc auditing. The pipeline links reasoning claims to target entities and segmentation masks across frames.

  • D. Reasoning to Evidence Grounding Framework: The framework transforms low-density, loosely structured reasoning into executable evidence instructions for visual verification.Thinking Summarizer and SAM3 serve as a decoupled auditing interface rather than an end-to-end optimization loop.
  • Thinking Summarizer: Thinking Summarizer maps reasoning to key points, primary objects, emotional indicators, and SAM3 instructions.It is trained through API-generated targets followed by LoRA fine-tuning of a lightweight autoregressive model.
  • SAM3 Based Segmentation Integration: The SAM3 instruction field becomes a target entity list used to prompt pixel-level segmentation over video frames.Each entity is processed by SAM3 to produce a corresponding segmentation mask.
  • Verification Framework: Segmentation success externally tests whether textual affective claims correspond to the correct visual regions.For a claimed shift from surprise to relief, the instructions should identify the relevant person and SAM3 should segment that target.
  • D. Reasoning to Evidence Grounding Framework: Figure 3 visualizes the conversion from free-form reasoning to key points, focus objects, and SAM3 grounding instructions.

IV. EXPERIMENTS

Experiments evaluate AffectOmni across three multimodal benchmarks using a Qwen2.5-Omni-7B base, cold-start SFT, and two-stage GRPO training. The setup compares the method with proprietary and open-source omni-modal baselines.

  • IV. EXPERIMENTS: The section reports experimental results and further analysis, with implementation details placed in the appendix.
  • Training Data and Configuration: Qwen2.5-Omni-7B-Thinker is initialized with structured-output alignment and cold-start SFT before two-stage GRPO training.The training corpus contains 24K video-audio samples, with G=4 candidates per instance and reward weights λp=0.2 and λt=0.2.
  • Evaluation Benchmarks: AffectOmni is evaluated on IntentBench, Daily-Omni, and WorldSense, spanning intent, emotion, daily-scenario, and world-knowledge audiovisual questions.The benchmarks contain 633, 684, and 1,662 videos, respectively.
  • Baselines: The experiments compare AffectOmni with proprietary models and open-source omni-modal systems, including HumanOmniV2, Unified-IO-2, VideoLLaMA2, and VITA-1.5.

B. Main Results

AffectOmni improves affective reasoning across three multimodal benchmarks and uses people-centric rewards, comparative scoring, and visual grounding to strengthen evidence-based reasoning. Its gains are strongest on emotion- and temporally sensitive tasks, while performance remains uneven on deception and knowledge-intensive domains.

  • IntentBench: 71.89 average accuracy on IntentBench exceeds HumanOmniV2 (7B) at 69.23, with the largest gains on Emotion (+4.66 pp) and How (+3.64 pp).These categories are more directly aligned with people-centric evidence and temporally organized reasoning.
  • Daily-Omni: 61.90 average accuracy on Daily-Omni improves over HumanOmniV2 (7B) at 58.47, with gains in Context (+4.15 pp), Reason (+4.01 pp), and 60s (+5.46 pp).Infer is slightly lower than HumanOmniV2, and gains are not uniform across dimensions.
  • WorldSense: 48.80 average accuracy on WorldSense exceeds HumanOmniV2 (7B) at 47.70 and slightly surpasses Gemini1.5 Pro, led by a 3.9 pp gain in Music.AffectOmni remains behind Gemini1.5 Pro on Film, Tech, and Perform., which rely more on external knowledge and cross-domain semantics.
  • Reward ablation: The full reward design improves IntentBench average accuracy by 2.66 pp and increases the When category by 14.29 pp, while Deception declines without a statistically reliable degradation.The Deception confidence interval includes zero, indicating a boundary of the current inductive bias for mismatches between surface behavior and underlying intent.
  • Reward discrimination: Comparative scoring raises overall CV from 0.153 to 0.325 and within-group CV from 0.049 to 0.138, producing clearer reward differences among candidates.The corresponding IntentBench average accuracy rises from 69.78 to 71.89.
  • Verification: AffectOmni connects structured reasoning chains to localized visual evidence, providing a verification interface for checking whether cited affective cues support the analysis.The framework’s rewards target people-centric and temporal reasoning, while visual localization supports external auditing.

V. DISCUSSION

The discussion frames MEP as an executable evidence interface for post-hoc auditing, connecting structured reasoning to visual verification while supporting reusable downstream integration. Diagnostic and benchmark results support its operational relevance.

  • V. DISCUSSION: MEP converts reasoning into an executable, checkable evidence carrier rather than treating segmentation quality as the primary goal.The interface is intended for post-hoc auditing in settings requiring auditable and reviewable evidence packages.
  • V. DISCUSSION: The framework summarizes reasoning into focus entities, segmentation instructions, and pixel-level regions for consistency checks and human inspection.SAM3 grounds the extracted evidence instructions into visual regions referenced by the reasoning.
  • V. DISCUSSION: AffectOmni surpasses HumanOmniV2 on NExT-QA and gains on Temporal and Descriptive subsets without domain-specific fine-tuning.The reported results are presented as evidence that affective reinforcement learning does not cause catastrophic forgetting of general video understanding.
  • V. DISCUSSION: MEP uses {FocusObj, TimeSpan, EvidenceType} as a minimal query form that supports heterogeneous verifiers through lightweight adaptation.This interface can be reused without modifying the upstream reasoning compression core.
  • V. DISCUSSION: Three lightweight diagnostic experiments evaluate evidence extractability, evidence-to-verification consistency, and falsifiability without retraining.These experiments assess MEP as an executable evidence interface rather than merely as a visualization component.

A. Evidence Extractability

The evidence extractability experiment tests whether free-form reasoning contains stable, structured information that can be compressed into executable verification instructions. Most sampled outputs contain multiple composable evidence primitives.

  • A. Evidence Extractability: The first experiment samples 100 IntentBench think outputs and evaluates visual, temporal, and behavioral evidence elements.Visual evidence covers observable cues such as expression, gaze, and posture; temporal evidence covers order or change; behavioral evidence covers executable actions or interactions.
  • A. Evidence Extractability: A strict binary criterion, minimal supporting spans, and schema-constrained parsing reduce ambiguity and free-form output drift.Qwen-Max is used as the judge for detecting the structured evidence elements.
  • A. Evidence Extractability: 84%, 72%, and 89% are the occurrence rates of visual, temporal, and behavioral evidence, respectively.The statistic diagnoses interface feasibility rather than serving as a final evaluation metric.
  • A. Evidence Extractability: 93% of samples contain at least two evidence types, providing composable primitives for the Summarizer to construct an MEP.The evidence types are reported from 100 randomly sampled think outputs.

B. Evidence to Verification Consistency

The evidence-to-verification consistency experiment measures whether reasoning descriptions contain enough person-centered detail for downstream visual localization. Most samples satisfy both basic and high-quality grounding criteria, with ambiguity remaining in crowded or occluded scenes.

  • B. Evidence to Verification Consistency: The second experiment evaluates whether focus-object descriptions contain person mentions, appearance attributes, visual features, and locatable actions.These elements determine whether downstream visual verifiers can operationalize the reasoning.
  • B. Evidence to Verification Consistency: 98% of samples mention a person, 91% include visual feature descriptions, and 94% include locatable actions.The elements include attributes such as color and clothing, features such as expression and posture, and executable actions.
  • B. Evidence to Verification Consistency: 98% of samples satisfy Basic Grounding, while 86% satisfy High Quality Grounding.Basic Grounding requires an explicitly mentioned and segmentable person; grounding attainment remains consistent across question types.
  • B. Evidence to Verification Consistency: Under occlusion or in crowded multi-person scenes, grounding can become ambiguous without additional appearance cues or spatial references.The stated boundary concerns localization specificity in visually complex scenes.

C. External Auditing and Falsifiability

External auditing tests whether SAM3 evidence visualizations provide a falsifiable signal for answer trustworthiness. Relevant segmentation is associated with higher conditional accuracy, while false negatives limit diagnostic coverage without changing training or final-answer accuracy.

  • C. External Auditing and Falsifiability: The third experiment asks annotators to judge SAM3 segmentation relevance using only the question and segmentation results.Annotators do not see the model’s textual reasoning or the ground-truth label.
  • C. External Auditing and Falsifiability: 83.3% conditional accuracy follows relevant segmentation, compared with 35.3% for irrelevant segmentation, a difference of 48.0 percentage points.The contingency table reports an overall agreement rate of 77% and Phi coefficient ϕ = 0.48.
  • C. External Auditing and Falsifiability: Among 67 correct-answer cases, 12 have irrelevant segmentation, yielding a 17.9% false-negative rate.These false negatives reduce diagnostic coverage but do not affect training or final-answer accuracy because verification is post-hoc and excluded from GRPO optimization.
  • C. External Auditing and Falsifiability: AffectOmni combines people-focused and temporal rewards with post-hoc SAM3 grounding to support external consistency and falsifiability diagnostics.The conclusion reports consistent gains over open-source 7B-scale baselines on people-centric and temporally sensitive tasks.

APPENDIX SUPPLEMENTARY MATERIAL

The supplementary material provides additional details and analyses supporting the main text, including training, evaluation, reward, sensitivity, and reasoning-quality analyses.

  • Appendix A describes the full three-stage training pipeline and Thinking Summarizer configuration.
  • Appendix B reports complete per-subcategory results on Daily-Omni and WorldSense.
  • Appendix C analyzes how group size G affects reward discriminability and verifies the absence of positional bias.
  • Appendix D examines sensitivity to the reward weights λp and λt.
  • Appendix E evaluates reasoning-chain quality using five diagnostic dimensions with automatic and human assessment.
Loading 2608.26193v1…