SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Models

Tianyu Xie; Jinfa Huang; Yuexiao Ma; Rongfang Luo; Yan Yang; Wang Chen; Yuhui Zeng; Yixuan Zou; Qingchuan Ma; Zhiqiang Lu; Ruize Fang; Xiawu Zheng; Jiebo Luo; Rongrong Ji

Source-linked AI summary

SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Models

Tianyu Xie, Jinfa Huang, Yuexiao Ma, Rongfang Luo, Yan Yang, Wang Chen, Yuhui Zeng, Yixuan Zou, Qingchuan Ma, Zhiqiang Lu, Ruize Fang, Xiawu Zheng, Jiebo Luo, Rongrong Ji

arXiv:2603.16859v2Mar 17, 2026cs.AI

TL;DR

Existing omni-modal benchmarks largely test static correctness rather than social interactivity in dynamic dialogue. SocialOmni evaluates who speaks, when to interject, and how to respond, revealing that perceptual accuracy and natural interruption generation can diverge substantially.

Problem
Existing OLM benchmarks largely evaluate static accuracy, leaving dynamic social-interaction competence in dialogue insufficiently assessed despite the importance of semantic correctness and timing.
Method
SocialOmni benchmarks speaker identification, turn-entry timing, and context-appropriate interruption generation using 2,000 perception samples and 209 controlled interaction-generation instances.
Results
Across 12 OLMs, perceptual accuracy and interruption-generation quality are decoupled, with gains on one social-interaction axis not implying robustness on others.
Takeaways & Limitations
Understanding accuracy alone cannot characterize conversational social competence, motivating dedicated interaction-oriented evaluation across who, when, and how.
Takeaways & Limitations
The controlled generation subset does not exhaustively cover dialogue transitions, and transcribed outputs may underweight visual grounding and prosodic cues.

Abstract

from arXiv · show

Omni-modal large language models (OLMs) redefine human-machine interaction by natively integrating audio, vision, and text. However, existing OLM benchmarks remain anchored to static, accuracy-centric tasks, leaving a critical gap in assessing social interactivity, the fundamental capacity to navigate dynamic cues in natural dialogues. To this end, we propose SocialOmni, a comprehensive benchmark that operationalizes the evaluation of this conversational interactivity across three core dimensions: (i) speaker separation and identification (who is speaking), (ii) interruption timing control (when to interject), and (iii) natural interruption generation (how to phrase the interruption). SocialOmni features 2,000 perception samples and a quality-controlled diagnostic set of 209 interaction-generation instances with strict temporal and contextual constraints, complemented by controlled audio-visual inconsistency scenarios to test model robustness. We benchmarked 12 leading OLMs, which uncovers significant variance in their social-interaction capabilities across models. Furthermore, our analysis reveals a pronounced decoupling between a model's perceptual accuracy and its ability to generate contextually appropriate interruptions, indicating that understanding-centric metrics alone are insufficient to characterize conversational social competence. More encouragingly, these diagnostics from SocialOmni yield actionable signals for bridging the perception-interaction divide in future OLMs.

B Corresponding Author

The SocialOmni project page and dataset are publicly available through GitHub and Hugging Face, respectively.

The project page is available at github.com/MAC-AutoML/SocialOmni.
The dataset is available at huggingface.co/datasets/alexisty/SocialOmni.

1 Introduction

SocialOmni addresses the gap between static correctness evaluation and social interactivity in omni-modal dialogue by evaluating who speaks, when to interrupt, and how to respond. Its benchmark combines perception and interaction-generation diagnostics with audio-visual conflict probes, revealing that perceptual accuracy does not reliably predict natural interruption quality.

Motivation: Existing OLM benchmarks emphasize static answer correctness, overlooking dynamic dialogue cues, turn timing, and socially coherent responses.This limits evaluation of coherent understanding across multi-turn conversations and interaction competence.
Benchmark design: SocialOmni evaluates three dimensions of social interactivity: speaker identification, interruption timing control, and natural interruption generation.The dimensions integrate multimodal speaker cues, real-time turn-taking analysis, and contextually coherent responses.
Benchmark design: 2,000 perception samples and 209 interaction-generation instances form SocialOmni’s diagnostic benchmark across 15 dialogue domains.The benchmark tests the pipeline from audio-visual grounding through turn-entry decisions to adaptive continuation under strict latency constraints.
Robustness: Controlled audio-visual inconsistency scenarios probe model robustness under cross-modal conflict.These mismatch probes quantify robustness and generalization under realistic audio-visual conflict scenarios.
Findings: 12 OLMs show differing error profiles across who–when–how, while strong speaker identification does not consistently yield natural interruptions.The benchmark uses frame-level perception diagnosis and multi-judge generation scoring to analyze this perception-generation decoupling.

2 Related Work

Related work has progressed from perception-centric multimodal models to omni-modal large language models, while benchmarks have expanded from answer-centric understanding to interactive behavior. However, existing evaluations still fragment social interaction capabilities and lack integrated, fine-grained testing under multimodal conflict.

Omni-Modal Large Language Models: Omni-modal large language models extend earlier perception-centric and instruction-tuned vision-language paradigms by unifying text, vision, and audio within one interaction loop.Recent multimodal perception and representation-learning studies further broaden the design space for modern multimodal large language models.
Answer-Centric Benchmarks for OLMs: Answer-centric OLM benchmarks evaluate multimodal knowledge through question answering or retrieval over pre-segmented stimuli, typically measuring factual response accuracy.Cross-modal question-answering suites pair audio-visual clips with factual questions and score answer correctness using unified metrics.
Behavior-Centric Benchmarks for OLMs: Behavior-centric benchmarks probe social reasoning, turn-taking, interruption detection, and multimodal conversation, but remain limited by missing timing, speaker grounding, or cross-modal-conflict controls.Audio-only studies often lack multimodal grounding, while multimodal interaction benchmarks frequently lack frame-level temporal supervision and diagnostic control of cross-modal conflict.
Behavior-Centric Benchmarks for OLMs: Because who, when, and how are causally entangled in dialogue, evaluating them separately can overestimate interactive competence by masking cascading failures.Joint evaluation of multi-party interaction under fine-grained temporal alignment and controlled cross-modal conflict remains an open problem.

3 SocialOmni: Evaluating Omni-Modal Multi-Party Interactivity

SocialOmni is a unified benchmark for evaluating multi-party conversational interactivity in omni-modal language models through the coupled who–when–how abilities. It combines speaker perception with turn-taking timing and context-appropriate response generation under controlled audio-visual conditions.

Benchmark scope: SocialOmni evaluates who is speaking, when to take the floor, and how to respond within a unified multi-party interaction framework.These abilities correspond to speaker perception, turn-taking timing, and generative interaction.
Dataset construction: 15 dialogue subcategories span Entertainment, Professional, Daily Life, and Narrative domains after filtering over 3,000 crawled videos into 10–30-second multi-party clips.Eight trained annotators review every video and filter for audio clarity, face visibility, and turn-structure quality.
Dataset construction: 2,209 evaluation instances comprise 2,000 perception questions and 209 open-ended generation items with multi-reference responses.The perception split includes 1,725 consistent and 275 inconsistent questions.
Who — Perception: Task I uses four-way classification to identify the active speaker and spoken content at timestamp t from video and audio.Distractors independently vary speaker identity and textual content.
When & How — Generation: Task II first makes a binary decision about whether to speak, then generates a context-appropriate utterance when the answer is affirmative.Generation clips require sufficiently clear alternating speaker turns and undergo two rounds of adjudication.
Evaluation metrics: The benchmark measures timing with signed response offset and On-time rate, while response quality uses three independent LLM judges and four-level scores.Judges are GPT-4o, Gemini 2.5 Pro, and Qwen3-Omni; auxiliary metrics include response coverage and large-gap rate.

4 Experiments

Experiments evaluate twelve omni-modal models across SocialOmni’s who–when–how axes and diagnose perception reliability, timing behavior, and failure modes. Results show axis-specific leadership, a commercial advantage in response quality, and a clear separation between perception and generation abilities.

Experimental setup: 12 omni-modal large language models are evaluated across commercial APIs and open-source systems using unified raw video-audio inputs and fixed prompts.MiniOmni2 is evaluated only on perception because it lacks a stable generation interface.
Main results: Qwen3-Omni leads who at 69.25%, Gemini 3 Pro Preview leads when at 67.31%, and Gemini 2.5 Flash leads how at 85.08%.No single model dominates all three axes, and the capability profiles are visibly lopsided.
Main results: The best open-source how score, Qwen2.5-Omni’s 66.15, trails Gemini 2.5 Flash’s commercial score of 85.08 by nearly 19 points.Commercial APIs also consistently outperform open-source models on when, while Qwen3-Omni leads who overall.
Diagnostic analysis: Perception and generation rankings decouple: Qwen3-Omni-Thinking scores 18.06 on how despite competitive who, while GPT-4o scores 36.75% on who but 69.64 on how.These inversions motivate evaluating conversational interactivity as a multidimensional profile rather than through perception accuracy alone.
Diagnostic analysis: Timing errors split between premature interruption and excessive delay: Qwen2.5-Omni has E = 22.5%, whereas OmniVinci has L = 54.5%.Aggressive models interrupt before turn boundaries, while conservative models miss the conversational window; precision and recall therefore expose behavior that On-time alone misses.
Failure analysis: Generation failures include mistaking prosodic pauses for turn-final cues and producing contextually incoherent continuations, reflecting shallow timing or insufficient discourse integration.Perception failures include attributing off-screen speech to a visually salient face and matching correct transcription content to the wrong speaker.

5 Conclusion

SocialOmni provides a joint who–when–how benchmark for omni-modal large language models, with experiments across 12 models revealing decoupled perceptual accuracy and generation quality. Its controlled diagnostic scope and transcription-based evaluation motivate broader multi-turn, human, prosodic, and gesture-aware assessments.

Conclusion: SocialOmni evaluates speaker identification, turn timing, and response generation in omni-modal large language models through two tasks.Task I targets speaker identification, while Task II targets turn timing and response generation.
Conclusion: 12 OLMs show rank decoupling between perceptual accuracy and generation quality, with heterogeneous robustness under speaker-camera mismatch.These findings indicate that understanding accuracy alone cannot characterize the evaluated capabilities.
Limitations and Future Work: The generation subset is a controlled diagnostic that does not exhaustively cover all dialogue transitions.This limitation constrains how comprehensively the benchmark represents conversational interaction.
Limitations and Future Work: The Task II response-quality evaluation relies on transcribed model outputs and may underweight visual grounding and prosodic cues.Future work will scale SocialOmni to multi-turn trajectories, add human evaluation, and extend modality coverage to prosody and gestures.

Appendix · A Additional Method Details for SocialOmni · A.1 Inter-Annotator Agreement

The subsection uses raw percent agreement as its primary inter-annotator agreement metric, with disagreements adjudicated by a senior reviewer. Adjudication flags and, where licensing permits, rationales and error categories accompany released final labels, while chance-corrected measures remain future work.

A.1 Inter-Annotator Agreement: Raw percent agreement is the primary inter-annotator agreement metric.The annotation types comprise 4-way perception labels and binary consistency labels.
A.1 Inter-Annotator Agreement: Percent agreement is selected because it is directly interpretable for the annotation types.The reported annotation types are 4-way perception labels and binary consistency labels.
A.1 Inter-Annotator Agreement: All annotation disagreements are resolved through senior-reviewer adjudication.Adjudication is part of the labeling procedure for the reported agreement statistics.
A.1 Inter-Annotator Agreement: Adjudication flags are released alongside the final labels.Where licensing permits, the release also includes brief rationales and error categories.
A.1 Inter-Annotator Agreement: Where licensing permits, brief rationales and error categories accompany the released final labels.These materials are provided alongside adjudication flags.
A.1 Inter-Annotator Agreement: Chance-corrected measures such as Cohen’s κ are deferred to future work.The subsection explicitly leaves these measures unreported for now.

A.2 Domain and Subcategory Definitions

SocialOmni stratifies 15 dialogue subcategories across four domains: entertainment, professional, daily life, and narrative. These labels support balanced splits and per-domain analysis, with source metadata and clip assignments released subject to licensing constraints.

Domain and Subcategory Definitions: 15 dialogue subcategories are organized into four benchmark domains: entertainment, professional, daily life, and narrative.Entertainment covers interactive media formats; professional covers formal or task-oriented settings; daily life covers everyday conversations; and narrative covers scripted scenes from movies and dramas.
Domain and Subcategory Definitions: Domain labels are used for split balancing and per-domain analysis.Exact source metadata and clip-level assignments are released subject to licensing constraints.

A.3 Option Balance in the Perception MCQ

The 2,000 perception items have an intentionally unbalanced correct-option distribution reflecting real dialogue footage, with fairness addressed through per-option and stratified analyses.

Option distribution: 569 A, 561 B, 453 C, and 417 D are the correct-option counts across the 2,000 perception items.The imbalance reflects natural speaker prominence and camera-framing biases rather than annotation artifacts.
Option distribution: Artificial rebalancing is avoided because it could distort real-world statistics and introduce selection bias.
Fairness analysis: Fairness is assessed using per-option accuracy, detailed confusion matrices, and analyses stratified by domain and consistency split.

A.4 Consistency Labeling and Boundary Cases … A.19 Representative Failure Cases

The appendix specifies controlled procedures for labeling, timing, generation judging, modality ablation, and statistical evaluation, then analyzes challenging human-feedback cases and representative failures across who, when, and how.

A.4 Consistency Labeling and Boundary Cases: Consistency labels require independent annotation, second-reviewer verification, adjudication, and timestamped visual evidence, including partial visibility, weak identity cues, and off-screen speech.Boundary cases cover partially visible active speakers, heavily occluded identity cues, and reaction shots with off-screen speech.
A.5 Generation Subset Size (209 Items); A.9 Multiple References for Generation Calibration: 209 generation items prioritize comparability through fixed prompts, rubrics, judges, and multi-reference calibration because open-ended annotation varies in decision points, reference quality, and judge sensitivity.For Kmr = 30 tasks, semantically equivalent references calibrate judge tolerance and support reporting score variance across references.
A.6 Q1 Step Size and Temporal Granularity; A.7 Q1 Timing-Label Mapping: Q1 uses 30-fps frame timestamps for who but queries streaming decisions every 1 s, while timing labels classify offsets as Interrupted, Perfect, Delayed, TooLate, or NoResponse using thresholds (1, 2, 5) s.Multi-second windows reduce sensitivity to small step-size changes, though finer 0.5-s steps are encouraged when computation permits.
A.8 Delta-Window Binary Metrics for Q1: Q1 tolerance-window evaluation reports precision, recall, and F1 at δ ∈{0.2, 0.5, 1.0} s, alongside offset diagnostics, with undefined ratios set to 0 for stable aggregation.The reported formulas are Prec = TP/(TP + FP), Rec = TP/(TP + FN), and F1 = 2 Prec · Rec/(Prec + Rec).
A.10 Generation Judging Scope and Visual Grounding; A.12 Modality Ablation Implementation: Generation judging evaluates interruption appropriateness, coherence, and pragmatics primarily from dialogue context, while visual grounding is tested through inconsistency splits, modality ablations, and prompts requiring references to visible events or entities.Judges penalize hallucinated visual references and the corresponding visually grounded prompts are released.
A.15 Perception Results and Macro Metrics: 44.69% vs. 64.99% overall accuracy accompanies consistency gaps of +8.7 and +9.2 for Gemini 2.5 Pro and Gemini 3 Pro, while Qwen3-Omni-Thinking has a −5.9% gap.These results show that higher absolute accuracy does not remove split-specific brittleness, and deliberative reasoning may interfere with immediate cue integration in mismatch-heavy scenes.