PaperlayerView exact revision on arXiv

Source-linked AI summary

CogniRoute: Learning to Route Social Evidence in Omni-Modal Models

Yifan Shen, Pei Tian, Xinzhuo Li, Bowen Fang, Shujun Xia, Bingxuan Li, Ana Jojic, Wenming Ye, Xu Cao, James Matthew Rehg, Ismini Lourentzou

arXiv:2606.20970v1cs.CV

TL;DR

Social video question answering can depend on gestures, vocal tone, timing, and conflicts between audio and visual cues, so access to multiple modalities does not ensure use of the right evidence. CogniRoute trains expert routing with a schema of modality relations, reasoning demands, and temporal scope, then jointly optimizes answers and expert allocation. It achieves 59.38% average accuracy on OmniSocialBench, exceeding the strongest proprietary baseline by 15.33 percentage points and the strongest open-source omni baseline by 26.77 points.

  • Problem

    Social video QA can hinge on gestures, vocal tone, timing, and audio-visual conflicts, while access to multiple modalities does not ensure models use the right evidence.

    Source: §1 · p. 1

  • Method

    CogniRoute uses a training-only schema of modality relations, reasoning demands, and temporal scope to guide expert routing, with reinforcement learning jointly optimizing answers and expert allocation.

    Source: §1 · pp. 2–3

  • Results

    59.38% average accuracy on OmniSocialBench, exceeding the strongest proprietary baseline by 15.33 percentage points and the strongest open-source omni baseline by 26.77 points.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Evidence-structured routing is an effective training signal for improving social omni-modal reasoning.

    Source: §6 · p. 11

    The study focuses on pre-segmented clips and text-form answers, not online streaming interaction, real-time dialogue management, or speech generation quality.

    Source: §I · p. 36

Abstract

from arXiv · show

Omni-modal models can ingest video, audio, and text, but unified access to multiple modalities does not guarantee that a model uses the right evidence. This gap is especially pronounced in social video question answering, where the answer may hinge on a gesture, vocal tone, temporal cue, or mismatch between what is said and what is visually expressed. We introduce CogniRoute, a schema-guided Mixture-of-Experts framework for social omni reasoning. CogniRoute uses a training-only cognitive schema that factorizes each example by cross-modal relation, reasoning demand, and temporal scope, and aligns global routing signatures with this structure during supervised fine-tuning. We further introduce route-aware reinforcement learning, which jointly optimizes token generation and expert allocation using rewards for answer correctness, modality-consistent reasoning, and cognitive temporal grounding. To support training and evaluation, we construct OmniSocialBench, a diagnostic social video QA resource with 118K structured training examples, grounded reasoning traces, schema labels, temporal evidence spans, and a manually verified evaluation split. CogniRoute achieves 59.38\% average accuracy on OmniSocialBench, improving over the strongest proprietary baseline by 15.33 percentage points and the strongest open-source omni baseline by 26.77 points, with the largest gains on questions requiring audio-visual coordination, conflict resolution, and temporally grounded social inference.

1. Introduction

Social video QA requires selecting reliable evidence across visual cues, speech, prosody, and timing, yet answer accuracy alone cannot reveal whether a model used the right modality or temporal segment. CogniRoute trains expert routing with a cognitive schema and route-aware reinforcement learning, and OmniSocialBench supplies structured supervision and diagnostic evaluation for this task.

1. Introduction

  • Social QA may hinge on gestures, speech content or tone, cross-modal agreement or conflict, and timing, while answer accuracy alone cannot reveal whether the right evidence was used.Existing benchmarks generally specify answers but not which modality to trust, the required reasoning operation, or the relevant temporal evidence.Source: §1 · p. 1

1 INTRODUCTION

  • Standard multimodal MoE routing lacks sample-level supervision to distinguish the evidence structure and reasoning demand required by each question.Existing routing is typically driven by local token representations and optimized indirectly through next-token prediction and load balancing.Source: §1 · p. 2
  • CogniRoute uses a training-only Cognitive Schema to represent each example’s audio-visual relation, reasoning operation, and temporal scope, shaping routing during supervised fine-tuning.At inference, the model receives only the original video, audio, and question.Source: §1 · pp. 2–3
  • Route-Aware MoE Reinforcement Learning jointly optimizes answer generation and expert allocation with rollout rewards for correctness, modality-consistent reasoning, and cognitive temporal grounding.The rewards encourage correct answers grounded in appropriate modalities and time spans.Source: §1 · pp. 2–3
  • OmniSocialBench provides structured training annotations for evidence, schema, reasoning, routing, and temporal spans, plus a manually verified split for diagnostic social-inference evaluation.A staged pipeline derives observation-level audio and visual evidence before assigning schema labels and generating reasoning from that shared evidence.Source: §1 · pp. 2–3
  • 59.38% average accuracy on OmniSocialBench exceeds the strongest proprietary baseline by 15.33 percentage points and the strongest open-source omni baseline by 26.77 points.CogniRoute also improves over its base model on 8 out of 10 public audio-visual and video reasoning benchmarks.Source: §1 · p. 2

2. Related Work

Unified omni-modal access supports audio-visual understanding but does not specify evidence selection or computation allocation. CogniRoute addresses this gap with schema-guided routing and route-aware reinforcement learning.

Omnimodal Large Language Models and audio-visual reasoning

  • Omni-modal models, including Qwen-Omni [72] and Gemini [10] [58] [59], support unified audio-visual understanding, but do not specify how evidence is selected or computation allocated.Existing training primarily optimizes generated responses, indirectly supervising modality reliability, cross-modal relations, temporal grounding, and social evidence use.Source: §2 · p. 3

MoE-based Multimodal Models

  • Prior multimodal MoE methods specialize computation across modalities [3] [6] [42] [53], tasks [33], or domains, while CogniRoute aligns routing with sample-level schema labels and optimizes allocation through route-aware reinforcement learning.Existing routers typically optimize next-token prediction, load balancing, or efficiency rather than modality reliability, cross-modal relations, reasoning needs, and temporal cue location.Source: §2 · p. 3

3. Method

CogniRoute trains an omni-modal Mixture-of-Experts model in two stages: schema-aligned supervised routing followed by reinforcement learning that rewards correct, modality-consistent, temporally grounded reasoning and updates both generation and expert allocation.

3. Method

  • The method generates a reasoning trace and final answer from video, audio, and a question, and its two stages align routing before jointly optimizing output and expert allocation.The reinforcement stage uses rollout-level rewards for answer correctness, modality-consistent reasoning, and temporal grounding.Source: §3 · pp. 3–4

3.1. Schema-Aligned Predictive Routing (SAPR)

  • SAPR aligns global routing signatures with training-only tags for cross-modal evidence relations, reasoning demands, and temporal scope.It pools routing distributions by input modality, excludes generated reasoning tokens, and predicts the schema embedding from the resulting signature.Source: §3.1 · pp. 4–5

3.2. Route-Aware MoE Reinforcement Learning (RMRL)

  • Route-aware reinforcement learning rewards answer correctness, modality-consistent reasoning, and temporal grounding, then uses shared rollout advantages to update generated tokens and expert routing.The modality-consistency reward is gated by answer correctness; both policy branches also use KL regularization, and the router retains load balancing.Source: §3.2 · pp. 5–6

3.3. Cognitive Temporal Grounding Reward

  • Cognitive Temporal Grounding rewards focus on the annotated evidence span, using the model’s attention over audio-visual tokens as a proxy for temporal grounding.It averages native causal attention from a short suffix of the reasoning span, including its closing token, across deep layers and attention heads.Source: §3.3 · p. 6

4 OMNISOCIALBENCH DIAGNOSTIC BENCHMARK FOR SOCIAL OMNI REASONING

OmniSocialBench is a diagnostic social video QA resource that makes the audio-visual evidence, reasoning, and temporal structure behind answers explicit. Its structured annotations support training and evaluation of whether models use appropriate modalities, reasoning operations, and temporal evidence.

4 OMNISOCIALBENCH DIAGNOSTIC BENCHMARK FOR SOCIAL OMNI REASONING

  • The Cognitive Temporal Grounding reward compares the model’s attention over the clip timeline with annotated evidence spans, providing process-level feedback without a separate grounding model.A smooth target distribution tolerates span boundaries; the reward is zero without a valid reasoning suffix and otherwise decreases with KL divergence from the target.Source: §4 · p. 7

4. OmniSocialBench Diagnostic Benchmark for Social Omni Reasoning

  • OmniSocialBench provides 118K structured training examples and a manually verified evaluation split, preserving clips, audio, questions, answers, evidence, schema labels, reasoning traces, and temporal spans.Its sources cover multi-person interaction, affective and pragmatic reasoning, egocentric scenarios, and video understanding requiring social interpretation.Source: §4 · pp. 7–8, Figure 2 · p. 8

4 OMNISOCIALBENCH DIAGNOSTIC BENCHMARK FOR SOCIAL OMNI REASONING

  • Each example is labeled for evidence source, reasoning demand, and temporal scope, distinguishing Visual, Audio, Joint, or Conflict evidence and four temporal scopes.Reasoning-demand tags include Direct Perception, Temporal, Causal, Mental State, Social Norm, and Sarcasm; temporal scopes are Momentary, Local Window, Long Range, and Multi Segment.Source: §4 · p. 8
  • The annotations supervise schema-guided routing, grounded fine-tuning, and modality-consistent and temporal-grounding rewards, while evaluation tests appropriate evidence use and analyzes four social dimensions.Evaluation examples are manually verified for answer validity and evidence consistency; social dimensions include Mental State, Pragmatic Meaning, Action Goal, and Social Norm Inference.Source: §4 · p. 8

5 EXPERIMENTS

CogniRoute achieves the highest reported average accuracy across OmniSocialBench’s social inference categories, while additional experiments report gains on most audio-visual benchmarks and demonstrate real-time social inference on VR glasses.

5. Experiments

  • 59.38% average accuracy makes CogniRoute the best model in every OmniSocialBench category, ahead of Gemini 3.1 Pro at 44.05% and Qwen3 Omni at 32.61%.Qwen3 Omni is the strongest open-source omni model; Gemini 3.1 Pro leads the VLMs that take omni inputs.Source: Table 1 · p. 9, §5 · p. 9
  • CogniRoute’s largest gains over the strongest baseline are +18.79% on Action Goal and +24.80% on Social Norm inference.The paper identifies these categories as especially dependent on joint reasoning over visual and audio cues.Source: §5 · p. 9
  • CogniRoute improves over Qwen3-Omni-30B-A3B on most audio-visual benchmarks, while audio-only gains are marginal.The comparison covers ten public audio-visual, audio-only, and video-only benchmarks; the paper attributes the benefit to aligning routing with the correct evidence structure.Source: §5 · pp. 9–10, Table 2 · p. 10

5 EXPERIMENTS

  • Starting from the same SAPR-trained checkpoint, RMRL raises performance to 59.38; full token-and-gate optimization performs best, beyond token-level RL alone.The ablation indicates that explicitly optimizing expert allocation adds benefit beyond optimizing token generation alone.Source: Figure 4 · p. 10, §5 · pp. 10–11
  • On VR glasses, CogniRoute combines hand trajectories with partial speech for contextual assistance and uses hesitation or conflicting gestures to infer intent.The examples include detecting a polite refusal from subtle social cues.Source: Figure 5 · p. 11, §5 · p. 11

6. Conclusion … B.2. Task Annotation

CogniRoute aligns expert routing with the evidence structure of social video questions through schema-guided routing and route-aware reinforcement learning. The paper reports consistent gains on OmniSocialBench and public audiovisual tasks, especially for cross-modal coordination, conflict resolution, and temporally grounded inference.

6. Conclusion

  • CogniRoute shows consistent gains over strong omni baselines on OmniSocialBench and public audiovisual tasks, especially on cross-modal coordination, conflict resolution, and temporally grounded social inference.The conclusion characterizes evidence-structured routing as an effective training signal for social omni-modal reasoning.Source: §6 · p. 11

A.1. Answer Correctness Reward

  • Answer correctness receives a deterministic binary reward: 1 for a ground-truth match and 0 otherwise, without an external judge model.The final answer is extracted from the model output and compared under the benchmark evaluation protocol.Source: §A.1 · p. 19

A.2. Modality-Consistent Reasoning Reward

  • A frozen LLM judge scores whether reasoning uses the required visual, audio, or joint evidence and correctly characterizes supportive or conflicting cross-modal relations.The rubric ranges from 0 to 1 in five increments and rewards concrete modality-specific observations rather than keyword matching; answer correctness is handled separately.Source: §A.2 · p. 19

B. Detailed Annotation Protocol

  • A staged annotation pipeline draws from six video sources, separates evidence extraction from labeling and reasoning, and retains samples whose normalized answers match ground truth.The evaluation set follows the same pipeline and adds final manual verification.Source: B. Detailed Annotation Protocol · pp. 19–20

A. Reward Details

  • The reward combines answer correctness, Cognitive Temporal Grounding, and Modality-Consistent Reasoning to encourage correct answers grounded in the right evidence and time span.The three components are ℛans, ℛctg, and ℛmcr, respectively.Source: Figure 6 · p. 20

B.1. Structured Evidence Extraction

  • The first-stage extractor returns JSON observations of scene context, participants, timestamped visual and audio events, dialogue, cross-modal relations, and quality flags without answering the question.This shared evidence source supports later stages and simple automatic checks.Source: §B.1 · p. 20

B.2. Task Annotation

  • Each clip-question pair receives evidence-source, reasoning-demand, and temporal-scope labels that guide routing across modality, reasoning operation, and temporal context.The labels cover four evidence-source classes, six reasoning-demand classes, and four temporal-scope classes; generated targets use only cues in the evidence JSON, and answer-mismatched samples are removed after normalization.Source: §B.2 · p. 20

B.3. Evaluation Split … B.7. Social Dimension Taxonomy and Prompt

The pipeline separates observation-level evidence extraction from task labeling and grounded reasoning generation, then verifies answers and benchmark annotations. Three task labels guide modality, reasoning, and temporal routing, while social dimensions are used only for OmniSocialBench analysis.

B.3. Evaluation Split

  • All OmniSocialBench samples receive manual verification, and the evaluation split adds a benchmark-only social_dimension field for subset analysis and error reporting.Verification covers the evidence JSON, task labels, reasoning, answer, and social_dimension; automatic annotations use Gemini-3.1-Pro, with fields stored separately in JSON.Source: §B.3 · pp. 20–22

B.4. Structured Evidence Fields and Extraction Prompt

  • Evidence extraction precedes labeling and stores observation-level details in JSON across scene context, participants, visual, audio, dialogue, cross-modal nodes, and quality flags.Extractors record observable cues such as gaze, pauses, and prosody without interpreting them as sarcasm or hidden frustration.Source: §B.4 · p. 22

B.5. Task Labels and Prompts

  • Evidence-source labels—Visual, Audio, Joint, and Conflict—encode the minimum answer evidence and give the router a modality prior.The three task labels are assigned from the question, ground-truth answer, and structured evidence; they are sample-level, so one clip can receive different labels for different questions.Source: §B.5 · pp. 22–23
  • Reasoning-demand labels specify direct perception, temporal, causal, mental state, social norm, or sarcasm operations to guide reasoning-expert selection.Source: §B.5 · p. 23
  • Temporal-scope labels—momentary, local window, long range, and multi segment—identify the smallest temporal field needed and guide the router’s temporal field.Long-range evidence spans much of a clip, while multi-segment evidence combines separated spans.Source: §B.5 · pp. 23–24

B.6. Reasoning Prompt and Consistency Filter

  • Grounded response generation uses the predicted labels to align explanations with the intended answer path and permits no facts beyond the structured evidence JSON.Generated answers are normalized and checked against the original answer, including aliases when available; mismatches are discarded, making reasoning generation a quality filter rather than a source of labels.Source: §B.6 · p. 24

B.7. Social Dimension Taxonomy and Prompt

  • The benchmark-only social_dimension taxonomy covers Mental State, Pragmatic Meaning, Action Goal, and Social Norm Inference, and may vary for different questions about the same clip.It is not a training target or an expert-routing input.Source: §B.3 · p. 20, §B.7 · p. 24

C OMNISOCIALBENCH EXAMPLES · D ADDITIONAL EXPERIMENTAL DETAILS

OmniSocialBench examples pair social-video questions with structured audio-visual evidence, schema labels, grounded reasoning, and temporal spans. They illustrate diverse social reasoning cases and make supporting evidence explicit for diagnostic evaluation beyond final-answer accuracy.

C OMNISOCIALBENCH EXAMPLES

  • A reasoning-generation prompt produces the final tagged response from structured evidence and predicted task labels, while a social-dimension prompt assigns a benchmark label from structured evidence JSON.Source: Figure 12 · p. 25, Figure 13 · p. 25

C. OmniSocialBench Examples

D ADDITIONAL EXPERIMENTAL DETAILS

  • Representative OmniSocialBench instances pair questions and answers with schema tags, grounded evidence, reasoning traces, and temporal evidence spans.Source: Figure 14 · p. 26

D.1. Training Details · D.2. Data Details

The appendices specify CogniRoute’s training configuration and staged data pipeline, including schema-supervised training and a manually verified 800-sample evaluation benchmark.

D.1. Training Details

  • SFT trains Qwen3 Omni 30B for one day on 8 NVIDIA H200 GPUs, with per-device batch size 4, gradient accumulation 1, learning rate 1 × 10−4, and rank-64 low-rank adaptation.Training uses the base model with supervised fine-tuning.Source: §D.1 · p. 26
  • RL loss coefficients are λans = λctg = λmcr = 1; optimization uses AdamW with β1 = 0.9, β2 = 0.95, weight decay 0.1, cosine scheduling, and short warmup.Training also uses bfloat16 mixed precision and gradient clipping.Source: §D.1 · pp. 26–27
  • OmniSocialBench examples illustrate diverse modality requirements, reasoning demands, and temporal scopes, including audio-visual integration and socially grounded inference.Source: Figure 15 · p. 27
  • The mixture-of-experts model has 128 experts and selects 8 per forward pass from 12 routing-score candidates to encourage broader expert exploration.The candidate subset allows experts beyond the highest-scoring ones a chance to be selected.Source: §D.1 · p. 27

D.2. Data Details

  • The data pipeline assigns 10k samples to routing MLP training, 90k to schema-aligned SFT, and 10k to RL optimizing generation and expert routing.Training samples pair multimodal inputs with structured evidence and schema annotations; only samples whose normalized generated answers match the ground truth are retained.Source: §D.2 · p. 28
  • The 800-sample evaluation benchmark spans four tasks and is manually verified for audio-visual evidence supporting each original answer.Gemini outputs are neither correctness labels nor an inclusion criterion for evaluation; Gemini assisted only with training annotations, while final evaluation labels are human-verified.Source: §D.2 · p. 28

E. Qualitative Response Visualization and Analysis … F.6. SFT and RL Rewards

The analyses show that CogniRoute’s schema-guided routing and joint token–gate optimization support social evidence selection, schema-aware routing, and temporally grounded reasoning. Across qualitative cases and ablations, the results favor correct schema alignment and explicit gate optimization over weaker controls.

E. Qualitative Response Visualization and Analysis

  • Qualitative cases show CogniRoute using discourse-level reference tracking and temporal continuity where Gemini 3.1 Pro relies on locally plausible but mismatched cues.The examples cover reference resolution, multi-party turn-taking, and long-range affect continuity, illustrating evidence selection, cross-modal grounding, and temporal reasoning.Source: E. Qualitative Response Visualization… · p. 28, Figure 16 · p. 29, Figure 17 · p. 29

F.1. Training Generalization

  • Under identical training settings, CogniRoute outperforms the baseline and retrained OmniVinci across all four social dimension categories.The retrained OmniVinci also improves under the same supervision regime, indicating transfer and generalization gains from that setting.Source: §F.1 · p. 28, Table 3 · p. 30

F.2. SAPR Design

  • Full SAPR performs best across all four categories; shuffled and random tags approach standard SFT, and shared projections score below Full SAPR.The comparison supports linking each sample to its correct schema tags and retaining layer-specific projections when mapping routing statistics.Source: §F.2 · pp. 30–31, Table 4 · p. 31

F.3. Checking Learnable Tag Embedding Collapse

  • Full SAPR’s tag embeddings show low pairwise cosine values and high normalized effective rank, while its route mAP exceeds shuffled- and random-tag controls.These diagnostics indicate that embeddings do not collapse and that correctly associating samples with schema tags is needed for schema-aware routing.Source: §F.3 · p. 31, Table 5 · p. 31

F.4. Token Branch Versus Gate Branch

  • Full RMRL improves all four categories beyond token-only RL, while a trainable router without the gate objective remains clearly weaker.Gate-only updates yield a smaller gain, supporting joint optimization of token generation and expert allocation.Source: §F.4 · p. 32, Table 6 · p. 32

F.5. Routing Behavior Under Gate Optimization

  • Full RMRL yields the highest Route mAP, largest relevant-modality route shift, and smallest irrelevant-modality shift without routing collapse.The trainable router without the gate objective is weaker, indicating that trainability alone is insufficient for outcome-level credit assignment to expert selection.Source: §F.5 · p. 33, Table 7 · p. 33

F.6. SFT and RL Rewards

  • Removing any single RMRL reward degrades performance across all social dimensions, while introducing reinforcement learning improves on the SFT baseline.The ablation evaluates Answer Correctness, Modality-Consistent Reasoning, and Cognitive Temporal Grounding rewards.Source: §F.6 · p. 33, Table 8 · p. 33

F.7. CTG Evaluation Beyond Its Reward Value … H. Broader Impacts

The appendix tests temporal evidence use and expert-route exploration, compares CogniRoute with related work, and identifies potential benefits and deployment risks. Its evaluations show that the final model responds more to annotated evidence than to random audiovisual content.

F.7. CTG Evaluation Beyond Its Reward Value

  • Because annotations mark a single evidence time, the CTG evaluation omits temporal IoU and measures reasoning-suffix attention proximity using mass, hit, and peak-error metrics.Mass and Hit use 0.5-second and 1.0-second windows; PeakErr is the average absolute distance from the most-attended time to the annotation.Source: §F.7 · pp. 33–34
  • Full RMRL improves answer accuracy and point-based grounding under both tolerance windows, while reducing PeakErr; hard point CTG helps, but trails full CTG.The shuffled-time control is close to the model without the CTG reward, supporting the importance of correct evidence times; the binary hard reward gives no partial credit to near misses.Source: §F.7 · p. 34, Table 9 · p. 34

F.8. Causal Evidence Deletion

  • The causal deletion test compares the original input with removal of the annotated audiovisual window or an equal-length random window, while keeping question tokens unchanged.The annotated window spans tgt −0.5s to tgt +0.5s, and all settings use the main evaluation decoding strategy.Source: §F.8 · p. 34
  • On the final Full RMRL model, average accuracy was 46.63 under GT Mask, which removes the window centered at annotated evidence time, versus 56.13 under equal-length Random Mask.Annotated-window removal decreased accuracy across all four categories, while equal-length random-window removal had a much smaller effect.Source: §F.8 · p. 34, Table 10 · p. 35

G ADDITIONAL RELATED WORK

  • Table 10 evaluates causal evidence deletion on the final Full RMRL model by comparing annotated-window removal with equal-length random-window removal.The table reports accuracy for the two deletion conditions.Source: Table 10 · p. 35

F.9. Candidate Expert Pool Size for Route Exploration

  • M = 12 gives the best RL route-sampling performance, M = 13 is close, and larger pools such as M = 15 or 16 reduce accuracy.The model activates eight of 128 experts per routed token and samples from the top M router candidates; M = 8 is deterministic and weaker, while larger pools can add noisy low-probability experts.Source: §F.9 · p. 35, Table 11 · p. 35

G. Additional Related Work

  • Prior work addresses multimodal reasoning, social emotion and intent, turn-taking, and identity attribution [84] [83] [75] [70] [30], while CogniRoute frames social video QA as evidence allocation.CogniRoute supervises routing by modality relation, reasoning demand, and temporal scope, and rewards answer correctness, modality-consistent reasoning, and temporal grounding.Source: G. Additional Related Work · pp. 35–36

H. Broader Impacts

  • The work may benefit assistive, educational, human-computer interaction, and socially aware dialogue applications, but social video systems also risk privacy harms, bias, surveillance, and profiling.The benchmark is intended for research evaluation, not high-stakes decisions; deployment should require consent-aware collection, privacy protection, demographic and cultural bias audits, and restrictions against treating outputs as definitive judgments of people.Source: H. Broader Impacts · p. 36

I. Limitations

CogniRoute is evaluated on pre-segmented social video QA with text-form answers, leaving streaming interaction, dialogue management, and speech generation quality unassessed. Its schema is compact and its implementation uses an MoE omni backbone, though the authors state these limits do not affect their core conclusion about schema-guided routing.

I. Limitations

  • The evaluation covers pre-segmented social video QA with text-form answers, not online streaming, real-time dialogue management, or speech generation quality.These capabilities are not evaluated in the work.Source: §I · p. 36
  • The Cognitive Schema uses a compact label space, which may need finer-grained social categories for domain-specific applications.Source: §I · p. 36
  • The current implementation uses an MoE omni backbone, and adapting it to other architectures may require minor engineering changes.Source: §I · p. 36
  • The authors state that these limitations do not affect their core conclusion that sample-level schema guidance can improve expert routing and audio-visual reasoning.Source: §I · p. 36

Related papers

Loading 2606.20970v1…