PaperlayerView exact revision on arXiv

Source-linked AI summary

CogniRoute: Learning to Route Social Evidence in Omni-Modal Models

Yifan Shen, Pei Tian, Xinzhuo Li, Bowen Fang, Shujun Xia, Bingxuan Li, Ana Jojic, Wenming Ye, Xu Cao, James Matthew Rehg, Ismini Lourentzou

arXiv:2606.20970v1cs.CV

TL;DR

Social video QA requires models to use the right evidence across modalities, reasoning demands, and time. CogniRoute learns schema-guided expert routing with route-aware reinforcement learning, achieving consistent gains, including 59.38% average accuracy on OmniSocialBench.

  • Problem

    Social video QA lacks explicit supervision for which cross-modal and temporal cues support answers, despite their importance for social meaning.

  • Method

    CogniRoute uses training-only schema-guided expert routing and route-aware reinforcement learning to align computation with modality relations, reasoning demands, and temporal evidence.

  • Results

    CogniRoute shows consistent gains across OmniSocialBench and public audiovisual tasks, achieving 59.38% average accuracy on OmniSocialBench.

  • Takeaways & Limitations

    Evidence-structured routing is an effective training signal for improving social omni-modal reasoning, especially with cross-modal coordination, conflict resolution, and temporal grounding.

  • Takeaways & Limitations

    The evaluation focuses on pre-segmented clips and text-form answers, excluding online streaming interaction, real-time dialogue management, and speech generation quality.

Abstract

from arXiv · show

Omni-modal models can ingest video, audio, and text, but unified access to multiple modalities does not guarantee that a model uses the right evidence. This gap is especially pronounced in social video question answering, where the answer may hinge on a gesture, vocal tone, temporal cue, or mismatch between what is said and what is visually expressed. We introduce CogniRoute, a schema-guided Mixture-of-Experts framework for social omni reasoning. CogniRoute uses a training-only cognitive schema that factorizes each example by cross-modal relation, reasoning demand, and temporal scope, and aligns global routing signatures with this structure during supervised fine-tuning. We further introduce route-aware reinforcement learning, which jointly optimizes token generation and expert allocation using rewards for answer correctness, modality-consistent reasoning, and cognitive temporal grounding. To support training and evaluation, we construct OmniSocialBench, a diagnostic social video QA resource with 118K structured training examples, grounded reasoning traces, schema labels, temporal evidence spans, and a manually verified evaluation split. CogniRoute achieves 59.38\% average accuracy on OmniSocialBench, improving over the strongest proprietary baseline by 15.33 percentage points and the strongest open-source omni baseline by 26.77 points, with the largest gains on questions requiring audio-visual coordination, conflict resolution, and temporally grounded social inference.

1. Introduction

The introduction identifies a gap between multimodal access and selecting appropriate evidence for social video QA, where answers depend on cross-modal relations and temporal cues. It presents CogniRoute, its schema-guided routing and reinforcement learning framework, OmniSocialBench, and reported benchmark gains.

  • Motivation: Omni models can process video, audio, and text, but often fail to use the right evidence for social video questions.Social meaning may depend on expression–action–speech relations and other multimodal cues.
  • Motivation: Existing benchmarks provide answers without specifying trusted modalities, cross-modal relations, reasoning operations, or temporal evidence locations.This can yield correct answers based on irrelevant segments, incorrect modalities, or dataset priors.
  • CogniRoute: CogniRoute aligns expert routing with each question’s modality relation, reasoning demand, and temporal scope through a training-only Cognitive Schema.The framework supports visual reasoning, audio reasoning, audio-visual coordination, conflict resolution, and temporally grounded inference.
  • CogniRoute: Route-Aware MoE Reinforcement Learning jointly optimizes token generation and expert allocation using rewards for answer correctness, modality-consistent reasoning, and cognitive temporal grounding.Answer likelihood alone cannot ensure that the model uses the relevant modality or temporal region.
  • Evaluation: 59.38% average accuracy on OmniSocialBench exceeds the strongest proprietary baseline by 15.33 percentage points and the strongest open-source omni baseline by 26.77 points.OmniSocialBench includes structured training data and a manually verified split for evaluating modality use, conflict resolution, and temporal grounding.

2. Related Work

Recent omni-modal models unify text, vision, and audio processing, while MoE multimodal models conditionally route inputs to specialized experts. However, unified modality access does not specify evidence selection, and existing routers are typically optimized through next-token prediction.

  • Omnimodal Large Language Models and audio-visual reasoning: Omni-modal models jointly process text, vision, and audio within unified interaction frameworks, with Qwen-Omni and Gemini demonstrating real-time, holistic audio-visual understanding.These systems extend MLLMs beyond image-text understanding.
  • Omnimodal Large Language Models and audio-visual reasoning: Unified multimodal access does not specify how evidence should be selected.The passage identifies evidence selection as an unresolved issue despite unified access.
  • MoE-based Multimodal Models: MoE multimodal models route inputs to subsets of experts, specializing computation across modalities, tasks, and domains.Prior methods use modality- or task-conditioned experts, sparse routing, and parameter-efficient expert adapters.
  • MoE-based Multimodal Models: Existing multimodal MoE methods improve capacity and efficiency, but their routers are typically optimized through next-token prediction.The passage frames router optimization as a limitation of prior approaches.

3. Method

CogniRoute trains an omni-modal Mixture-of-Experts model to route evidence-aware computation using schema-aligned supervised learning and route-aware reinforcement learning. Its rewards jointly target answer correctness, modality-consistent reasoning, expert allocation, and temporal grounding of decisive social cues.

  • Method overview: CogniRoute factorizes each sample by evidence source, reasoning demand, and temporal scope to align routing with its social reasoning requirements.The input comprises video, audio, and a question, while the output contains a reasoning trace and final answer.
  • Stage 1: Schema-Aligned Predictive Routing: Schema-Aligned Predictive Routing pools layerwise router distributions by modality and maps a global routing signature to training-only schema targets.The signature uses input tokens rather than generated reasoning tokens, and modality pooling prevents long streams from overwhelming question tokens.
  • Stage 2: Route-aware reinforcement learning: Route-aware reinforcement learning jointly updates generated tokens and expert allocation from complete-rollout feedback using a composite reward.The objective combines answer correctness, Modality-Consistent Reasoning, and Cognitive Temporal Grounding rewards, with shared rollout-level advantages for both branches.
  • Cognitive Temporal Grounding: Cognitive Temporal Grounding uses audio-visual attention over a short reasoning suffix to encourage focus on the annotated evidence span.Audio and video tokens are aligned on a shared temporal grid, and the suffix includes the closing reasoning token.

4 OMNISOCIALBENCH DIAGNOSTIC BENCHMARK FOR SOCIAL OMNI REASONING

OmniSocialBench is a diagnostic social video QA benchmark that makes multimodal evidence, reasoning structure, and temporal support explicit rather than supervising only final answers. It combines 118K structured training examples with a manually verified evaluation split and annotations designed to support schema-guided routing, grounded reasoning, and temporal evaluation.

  • Dataset construction: 118K structured training examples and a manually verified evaluation split form OmniSocialBench’s diagnostic benchmarking resource.The benchmark draws from diverse sources spanning multi-person interaction, affective and pragmatic reasoning, egocentric scenarios, and socially interpreted video understanding.
  • Dataset construction: Each example preserves the video, audio, question, and answer while adding audio-visual evidence, schema labels, grounded reasoning traces, and temporal evidence spans.Recorded evidence includes social cues such as gaze, expression, gesture, speech content, speaker changes, laughter, silence, and tone-relevant acoustic events.
  • Schema annotations: The schema assigns one tag on each axis for evidence source, reasoning demand, and temporal scope.Evidence-source tags are Visual, Audio, Joint, or Conflict; reasoning-demand tags include Direct Perception, Temporal, Causal, Mental State, Social Norm, and Sarcasm.
  • Grounded supervision and evaluation: The annotations support schema-aligned routing, supervised fine-tuning, modality-consistent rewards, and Cognitive Temporal Grounding evaluation.Temporal grounding compares the model’s attention-derived temporal focus with annotated evidence regions, without requiring a separate temporal grounding model.

5 EXPERIMENTS

CogniRoute achieves the strongest social-understanding performance across all four OmniSocialBench categories, while schema-guided routing and route-aware reinforcement learning improve coordinated audio-visual reasoning. Additional evaluations cover ten public benchmarks, ablations, and a real-time VR smart-glasses deployment for inferring social intent.

  • OmniSocialBench Social Understanding Results: 59.38 is CogniRoute’s average accuracy on OmniSocialBench, surpassing Gemini 3.1 Pro’s 44.05% and Qwen3 Omni’s 32.61%.CogniRoute achieves the best result in every social inference category.
  • OmniSocialBench Social Understanding Results: Schema-guided routing and route-aware reinforcement learning substantially improve tasks requiring coordinated visual and audio evidence.The evaluation compares CogniRoute with Gemini 3.1 Pro response examples in Appendix E.
  • Public Audio-Visual Benchmarks: CogniRoute is compared with Qwen3-Omni-30B-A3B across ten public audio-visual, audio-only, and video-only benchmarks.The joint audio-visual benchmarks probe cross-modal integration across varied temporal scales and conditions.
  • Ablation Studies: The full token-and-gate objective performs best, showing that explicitly optimizing expert allocation contributes beyond token generation alone.Additional ablations examine SAPR design, schema supervision quality, tag-embedding collapse, token-versus-gate optimization, and routing behavior under gate optimization.
  • Real-world VR Smart Glasses Deployment: In VR smart-glasses deployment, CogniRoute combines hand trajectories with partial speech and detects hesitation and conflicting gestures to infer true intent.Examples include contextual assistance and recognizing a polite refusal.

6. Conclusion … B.2. Task Annotation

CogniRoute aligns expert routing with question-specific evidence structure and combines answer, temporal-grounding, and modality-consistency rewards for omni-modal social reasoning. Its annotation pipeline extracts structured multimodal evidence, filters answer-consistent samples, and labels each clip-question pair by evidence source, reasoning demand, and temporal scope.

  • 6. Conclusion: CogniRoute is a schema-guided MoE framework that aligns expert routing with each question’s evidence structure.Schema-Aligned Predictive Routing and route-aware reinforcement learning improve accuracy while encouraging evidence-aware computation across modalities and time.
  • A.1. Answer Correctness Reward: Answer correctness is a deterministic binary reward: correct answers receive 1, while incorrect answers receive 0.The final answer is extracted and compared with ground truth under the benchmark evaluation protocol without an external judge model.
  • A.2. Modality-Consistent Reasoning Reward: Modality-Consistent Reasoning complements temporal grounding by evaluating whether reasoning uses the appropriate visual, audio, or combined evidence.A frozen LLM judge computes this reward from Structured Evidence Extraction annotations specifying the expected modality evidence.
  • B. Detailed Annotation Protocol: The dataset draws from six audiovisual sources covering multi-person interaction, affect-centered scenes, egocentric videos, and socially interpreted general videos.The sources are OmniMMSI-YouTube, SocialIQ, MTMEUR, VideoChat-Conversation, NeXTQA, and LLaVA-Video.
  • A. Reward Details: The overall reward combines answer correctness, Cognitive Temporal Grounding, and Modality-Consistent Reasoning.These components respectively reward correct final answers, attention to annotated temporal evidence, and reasoning grounded in required visual and/or audio evidence.
  • B. Detailed Annotation Protocol: Samples are retained only when their normalized generated answers match the original ground truth, while evaluation examples also receive final manual verification.The evaluation set follows the same curation pipeline before manual checking.
  • B.1. Structured Evidence Extraction: Structured Evidence Extraction produces JSON observations of context, participants, timestamped visual and audio events, dialogue structure, cross-modal relations, and quality flags.The extractor remains at the observation level rather than social interpretation, providing a shared evidence source for later stages and automatic checks.
  • B.2. Task Annotation: Each clip-question pair receives evidence-source, reasoning-demand, and temporal-scope labels determined by the question.Evidence source distinguishes Visual, Audio, Joint, and Conflict; reasoning demand includes six operations; temporal scope captures how much of the clip is needed.

B.3. Evaluation Split … B.7. Social Dimension Taxonomy and Prompt

The evaluation and annotation pipeline separates observation, routing labels, grounded reasoning, consistency filtering, and benchmark-only social-dimension analysis. Structured evidence and manual verification support modality- and temporally aligned social video QA supervision and evaluation.

  • B.3. Evaluation Split: The evaluation split adds benchmark-only social_dimension labels covering four social phenomena and manually verifies samples before release.The four classes are Mental State Inference, Pragmatic Meaning Inference, Action Goal Inference, and Social Norm Inference.
  • B.4. Structured Evidence Fields and Extraction Prompt: A structured evidence step records observation-level scene, participant, visual, audio, dialogue, cross-modal, and quality information before label prediction.The extractor is instructed not to answer questions or infer hidden mental states.
  • B.5. Task Labels and Prompts: Task labels are assigned per question from the question, answer, and structured evidence, so one clip can receive different routing labels across questions.The labels describe evidence source, reasoning demand, and temporal scope.
  • B.5. Task Labels and Prompts: Evidence-source labels distinguish Visual, Audio, Joint, and Conflict cases according to modality support and cross-modal mismatch.Conflict applies when visual and audio cues point to different meanings; the label provides a modality prior to the router.
  • B.5. Task Labels and Prompts: Reasoning-demand labels cover direct perception, temporal, causal, mental state, and social norm operations, while temporal-scope labels select momentary, local window, long range, or multi segment evidence.Temporal scope is the smallest temporal field needed to answer correctly and guides the router’s temporal field.
  • B.6. Reasoning Prompt and Consistency Filter: Grounded response generation uses the clip, question, answer, structured evidence, and predicted labels, restricting explanations to facts present in the JSON.The labels keep reasoning aligned with the intended answer path.
  • B.6. Reasoning Prompt and Consistency Filter: Generated samples are discarded when normalized answers fail to match the original answer, making reasoning generation a quality check rather than a new labeling source.Normalization lowercases text, removes punctuation and articles, collapses spaces, and can accept aliases.
  • B.7. Social Dimension Taxonomy and Prompt: social_dimension is used only for OmniSocialBench analysis, not training targets or expert routing, and classifies social phenomena such as mental states, pragmatic meaning, and action goals.All released samples receive manual verification over evidence, task labels, reasoning, answers, and social_dimension.

C OMNISOCIALBENCH EXAMPLES · D ADDITIONAL EXPERIMENTAL DETAILS

This material presents OmniSocialBench examples and the supporting prompt designs for generating tagged responses and assigning social-dimension labels. The examples pair multimodal evidence with schema annotations, grounded reasoning, and temporal spans for diagnostic evaluation.

  • C OMNISOCIALBENCH EXAMPLES: The reasoning-generation prompt conditions final tagged responses on the video clip, audio, question, ground-truth answer, structured evidence JSON, and predicted task labels.
  • C OMNISOCIALBENCH EXAMPLES: The predicted task labels include evidence_source, reasoning_demand, and temporal_scope.
  • C OMNISOCIALBENCH EXAMPLES: The social-dimension prompt assigns a benchmark label from the structured evidence JSON.
  • C OMNISOCIALBENCH EXAMPLES: OmniSocialBench examples pair questions with structured audio-visual evidence, schema labels, grounded reasoning, and temporal evidence spans.
  • C OMNISOCIALBENCH EXAMPLES: The benchmark covers cases relying on visual cues, audio cues, joint interpretation, or conflicts between speech and visual expression.
  • C OMNISOCIALBENCH EXAMPLES: Explicit supporting evidence enables diagnostic evaluation beyond final-answer accuracy.
  • D ADDITIONAL EXPERIMENTAL DETAILS: Figure 14 visualizes representative social video QA instances with questions, answers, schema tags, grounded evidence, reasoning traces, and temporal evidence spans.

D.1. Training Details

Training uses Qwen3 Omni 30B with supervised fine-tuning on eight NVIDIA H200 GPUs, followed by reinforcement learning with equal loss weights for answer, cognitive temporal grounding, and modality-consistent reasoning.

  • SFT trains Qwen3 Omni 30B on 8 NVIDIA H200 GPUs for one day with per-device batch size 4 and gradient accumulation step 1.The learning rate is 1 × 10−4, with low-rank adaptation at rank 64.
  • The SFT learning rate is 1 × 10−4, and low-rank adaptation uses rank 64.
  • RL assigns equal coefficients of 1 to λans, λctg, and λmcr.The reported RL loss coefficients are λans = λctg = λmcr = 1.
  • Optimization uses AdamW with standard β1 = 0.9.

D ADDITIONAL EXPERIMENTAL DETAILS · D.2. Data Details

The additional experimental details illustrate how OmniSocialBench captures diverse modality requirements, reasoning demands, and temporal scopes. Data construction uses structured evidence extraction, schema annotation, grounded reasoning generation, and answer-consistency filtering for high-quality supervision.

  • D ADDITIONAL EXPERIMENTAL DETAILS: A follow-up comment directed to a prior respondent is classified as Social Norm reasoning over a Local Window.The example identifies Jessica as the prior respondent and answers A (consistent curiosity).
  • D ADDITIONAL EXPERIMENTAL DETAILS: A boy’s stable wide-eyed curiosity is treated as Mental State reasoning over a Long Range.The answer is 3 (wrong artist reference).
  • D ADDITIONAL EXPERIMENTAL DETAILS: A wrong-artist correction triggering laughter is classified as Social Norm reasoning over a Local Window.The example’s answer is C (consistently serious).
  • D ADDITIONAL EXPERIMENTAL DETAILS: A man’s serious, focused expression remaining stable throughout conversation is categorized as Direct Perception over a Local Window.The scene description emphasizes maintenance of the expression through the conversation.
  • D ADDITIONAL EXPERIMENTAL DETAILS: OmniSocialBench examples span audio-visual integration and socially grounded inference across diverse modality requirements, reasoning demands, and temporal scopes.Figure 15 provides additional visualizations of these example types.
  • D.2. Data Details: The data pipeline stages structured evidence extraction, schema annotation, and grounded reasoning generation.Each training sample pairs multimodal inputs with structured evidence and schema-level annotations.
  • D.2. Data Details: Training samples are retained only when generated answers match the original ground-truth answers after normalization.This filtering step is used to ensure high-quality supervision.

E. Qualitative Response Visualization and Analysis … F.6. SFT and RL Rewards

CogniRoute’s qualitative and ablation analyses show that schema-aligned routing improves discourse, turn-taking, temporal, and modality-grounded social reasoning. Supervised schema alignment and joint token–gate reinforcement learning are both necessary for stronger, schema-aware routing and benchmark performance.

  • E. Qualitative Response Visualization and Analysis: CogniRoute better matches discourse reference, multi-party turn-taking, and long-range affect continuity than Gemini 3.1 Pro in diagnostic examples.Gemini often relies on locally plausible cues, whereas CogniRoute follows discourse structure, dialogue sequence, and temporal continuity.
  • F.1. Training Generalization: CogniRoute consistently outperforms OmniVinci and its retrained variant across all four social dimension categories under identical training settings.The comparison indicates stronger transfer and generalization under the same supervision regime.
  • F.2. SAPR Design: Full SAPR performs best across all four categories, while shuffled and random tags remain close to standard SFT and shared projections underperform.SAPR uses correct schema tags, the full 128-dimensional gate distribution, modality-wise pooling, and layer-specific projections; actual MoE computation selects 8 experts from 128.
  • F. Additional Experiments and Ablations: Across the additional experiments and ablations, performance gains consistently depend on correct schema supervision, layer-aware routing alignment, explicit gate optimization, and the complete reward design.These findings collectively distinguish meaningful routing improvements from auxiliary-loss effects, embedding collapse, router trainability alone, or partial reinforcement learning.
  • F.3. Checking Learnable Tag Embedding Collapse: Full SAPR avoids tag-embedding collapse and achieves much higher route mAP than shuffled and random tag controls.Its low pairwise cosine values and high normalized effective rank indicate diverse embedding directions, while routing schema awareness depends on correct sample–tag relations.
  • F.4. Token Branch Versus Gate Branch: Full RMRL jointly optimizing token generation and route allocation outperforms token-only, gate-only, and trainable-router variants across all four categories.The route probability ratio is required to assign outcome-level credit to expert selection; merely making the router trainable is insufficient.
  • F.5. Routing Behavior Under Gate Optimization: Full RMRL produces the most schema- and modality-aware routing while preserving balanced expert use.It yields the highest route mAP, largest ∆rel, smallest ∆irrel, and avoids routing collapse according to Load CV and Active Exp.
  • F.6. SFT and RL Rewards: Reinforcement learning significantly improves over SFT, and removing any individual reward degrades performance across all social dimensions.The evaluated rewards are Cognitive Temporal Grounding, Modality-Consistent Reasoning, and Answer Correctness.

F.7. CTG Evaluation Beyond Its Reward Value … H. Broader Impacts

The paper validates temporal grounding, causal evidence use, and route exploration, while situating CogniRoute within related multimodal reasoning work and noting broader applications and sensitivities. Full RMRL improves point-based grounding and responds more to annotated evidence than random content, with M = 12 selected for route exploration.

  • F.7. CTG Evaluation Beyond Its Reward Value: Temporal grounding is evaluated against a single annotated evidence time using attention mass, hit rate, and peak error rather than temporal IoU.Mass@0.5s and Mass@1.0s measure attention near the annotation; Hit measures whether the most attended bin falls within the window; PeakErr measures absolute temporal distance.
  • F.7. CTG Evaluation Beyond Its Reward Value: Full RMRL improves answer accuracy and point-based temporal grounding across both 0.5-second and 1.0-second tolerance windows, while lowering PeakErr.Mass and Hit are reported as percentages, and PeakErr in seconds; the full CTG reward outperforms the hard binary variant by granting partial credit to near misses.
  • F.8. Causal Evidence Deletion: GT Mask lowers average accuracy to 46.63, compared with 56.13 under equal-length Random Mask, indicating greater sensitivity to annotated evidence than random audiovisual content.GT Mask removes audiovisual tokens within [tgt −0.5s, tgt + 0.5s], while Random Mask removes an equal-length window elsewhere in the clip.
  • F.9. Candidate Expert Pool Size for Route Exploration: M = 12 yields the best route-exploration accuracy, whereas deterministic M = 8 is weaker and larger pools such as M = 15 and M = 16 reduce accuracy.The model activates k =8 experts and samples them from the top M router candidates; larger pools can introduce low-probability, noisy expert choices.
  • G. Additional Related Work: Related work extends reinforcement learning and process supervision into multimodal language models to improve reasoning across multimodal tasks.These approaches supervise final predictions and intermediate reasoning steps through reward signals and verification.
  • G. Additional Related Work: Social omni-modal research includes human-centric emotion and action understanding, context-aware reasoning about intentions, and targeted interaction benchmarks.The cited benchmarks address speaker identification, interruption timing and generation, and identity-attributed interaction from audio, vision, and speech cues.
  • H. Broader Impacts: The work aims to encourage appropriate use of visual, audio, and temporal evidence instead of relying only on final-answer supervision.Potential applications include assistive video understanding, educational analysis, human-computer interaction, and socially aware dialogue systems.

I. Limitations

CogniRoute is evaluated only on pre-segmented social video clips with text-form answers, leaving several interactive and generative capabilities unassessed. Its compact Cognitive Schema may also require finer-grained categories for domain-specific applications.

  • Evaluation scope: The study focuses on omni-modal social video QA with pre-segmented clips and text-form answers.This scope excludes broader online interaction settings.
  • Evaluation scope: It does not evaluate online streaming interaction, real-time dialogue management, or speech generation quality.
  • Schema granularity: The Cognitive Schema uses a compact label space, which may need finer-grained social categories for domain-specific applications.
Loading 2606.20970v1…