PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue

Haolin He, Yunfei Chu, Qi Chen, Wen Huang, Yuan Feng, Muzhi Zhu, Zheqi Dai, Haoning Xu, Dongchao Yang, Chunyat Wu, Zining Liang, Zhengxi Liu, Xiquan Li, Xie Chen, Xize Cheng, Qize Yang, Jin Xu, Qiuqiang Kong

arXiv:2609.21465v1eess.AScs.AIeess.IV

TL;DR

Native audio-visual dialogue has limited recordings and is difficult to evaluate when good replies depend on rich context and can be phrased in many ways. The paper builds synthetic dialogue resources and an RL reward design; training improves results on both synthetic and human-recorded benchmarks.

  • Problem

    Recordings of people using their own devices are scarce, and context-rich replies are difficult to assess reliably with keyword matching or other rule-based methods.

    Source: §1 · p. 2

  • Method

    The paper synthesizes single- and multi-turn dialogues, builds a five-category benchmark, and trains with an RL reward design targeting correctness, efficiency, and style.

    Source: §1 · p. 2, §4.1 · p. 7

  • Results

    OmniVChat-Bench Subcategory Mean rises from 0.465 to 0.652, and OmniVChat-Bench-Human recorded score rises from 0.402 to 0.632 after training.

    Source: §4.3 · p. 9

  • Takeaways & Limitations

    The gains support the training design and the use of synthetic data for training and evaluation in real-world dialogues.

    Source: §4.3 · p. 9

Abstract

from arXiv · show

We define OmniVChat (Omni Video Chat) as the task of native audio-visual dialogue between a user and an omni model. In OmniVChat, omni models directly and simultaneously receive audio and video from a user and return text. The user's query is embedded in the audio and video, without a separate text question, external captioning, or speech recognition. Direct audio-visual input reduces external latency and computation while preserving perceptual cues. However, research on OmniVChat faces two constraints: data availability and evaluation. Recordings of people using their own devices are scarce. Furthermore, a good reply often needs to account for the user's surroundings, facial expressions, and nearby objects, and such responses can be expressed in many different ways, making keyword matching unreliable for evaluating reply quality. Recent progress in agent systems and video generation makes generation for comprehension viable, which means using synthesized dialogues for training and evaluation. Therefore, we present OmniVChat-Studio, a multi-agent data engine for synthesizing single- and multi-turn audio-visual dialogues. We use synthesized dialogues to build OmniVChat-Bench, an evaluation benchmark that evaluates omni models' basic dialogue abilities across five ability categories. We also present OmniVChat-RL, a reinforcement learning reward design that jointly targets reply correctness, efficiency, and style in OmniVChat. Training Qwen3-Omni-Instruct with OmniVChat-RL on synthesized dialogues improves its performance on both OmniVChat-Bench and the human-recorded OmniVChat-Bench-Human. These gains validate the reward design and show transfer to real-world dialogues in training and evaluation.

1 INTRODUCTION

OmniVChat is native audio-visual dialogue: the model receives audio and video together and replies in text, without a separate text question or external speech recognition. The paper introduces synthetic-data and evaluation resources, plus an RL reward design, to address limited recordings and difficult reply evaluation.

1 INTRODUCTION

  • OmniVChat is native dialogue in which an omni model receives audio and video simultaneously and returns text, without a separate text question or external captioning or ASR.Direct audio-visual input reduces external latency and computation while preserving acoustic and visual cues.Source: §1 · p. 1
  • Open recordings of people using their own devices are scarce, while context-rich replies are difficult to evaluate reliably with keyword matching or other rule-based methods.External ASR cascades can omit subtle prosody and vocal affect while adding latency, computation, and potential transcription errors.Source: §1 · p. 2
  • OmniVChat-Studio synthesizes single- and multi-turn dialogues, OmniVChat-Bench evaluates five ability categories, and OmniVChat-RL jointly targets correctness, efficiency, and style.The benchmark uses tiered rubrics, and its training corpus is synthesized by OmniVChat-Studio.Source: §1 · p. 2
  • Fine-tuning Qwen3-Omni-Instruct with OmniVChat-RL on synthesized dialogues improves performance on both OmniVChat-Bench and the human-recorded OmniVChat-Bench-Human while making responses more concise.The authors report that gains in shared single-turn subcategories suggest alignment between synthetic and recorded dialogue distributions.Source: §1 · p. 2

2 OMNIVCHAT-STUDIO: SYNTHESIZING DIALOGUES FROM CORPUS

OmniVChat-Studio synthesizes audio-visual dialogue instances from a text corpus, with reference replies and rubrics. Its single-turn and multi-turn subsystems share four agents but use different workflows to construct and review dialogue and media.

2 OMNIVCHAT-STUDIO: SYNTHESIZING DIALOGUES FROM CORPUS

  • OmniVChat-Studio takes a text corpus and produces synchronized audio-visual clips, reference replies, and rubrics; its multi-turn subsystem builds dependent turns using accepted earlier clips and dialogue as history.The Director handles text tasks, the Renderer creates clips, the Reviewer checks rendered media, and the Validator checks records against configured rules.Source: §2 · p. 2
  • Its completed instance contains a reference reply and rubric whose tiers gate credit: failure at Tier 0 gives zero, and later tiers earn points only after earlier-tier criteria are met.An incomplete tier retains its earned points but blocks later tiers.Source: §2 · p. 3, §2.1 · p. 5

2.1 SINGLE-TURN SUBSYSTEM

  • The single-turn subsystem samples configured attributes and corpus passages, plans and validates a script, renders and reviews a clip, then produces and checks a grounded reply and tiered rubric.Review can request focused questions about the rendered audio or video; unsupported scripted details are left out of the reply.Source: §2.1 · pp. 4–5

2.2 MULTI-TURN SUBSYSTEM

  • The multi-turn subsystem plans all turns jointly, selects earlier media for later turns within a reference budget, and generates and reviews a separate rendering prompt for each turn.Prompt review can trigger a local rewrite or a plan update.Source: §2.2 · p. 5

3 OMNIVCHAT-BENCH: EVALUATING BASIC DIALOGUE ABILITIES

OmniVChat-Bench evaluates basic omni-model dialogue abilities with synthesized instances spanning five ability categories. Its instances include reference replies and tiered rubrics, with scores gated by criteria met in earlier tiers.

3 OMNIVCHAT-BENCH: EVALUATING BASIC DIALOGUE ABILITIES

  • OmniVChat-Bench contains 2,800 synthesized instances across five ability categories, 17 subcategories, and 22 scenario domains.Figure 3 depicts category and subcategory counts on the left and turn and duration distributions for 250 multi-turn instances on the right.Source: §3 · pp. 5–6, Figure 3 · p. 6
  • Each instance has a reference reply and tiered rubric; a criterion earns credit only when all criteria in earlier tiers are met, and failing the language-check Tier 0 gives a zero score.The rubric score divides credited points by the total number of criteria outside Tier 0.Source: §3 · p. 6
  • For multi-turn evaluation, models receive the same earlier clips and reference replies, and only the final reply is scored.The benchmark reports Subcategory Mean, which averages within subcategories before weighting all 17 subcategories equally.Source: §3 · p. 6
  • Its five categories assess dialogue-state and link perception, multimodal entity alignment, model self-awareness, anti-hallucination, and emotion recognition.The recorded probe contains only single-turn dialogues and does not test live interruption.Source: §3 · p. 7

4 OMNIVCHAT-RL: REWARDS FOR BETTER DIALOGUE

OmniVChat-RL trains a model with rewards for reply correctness, efficiency, and style, then evaluates it on synthetic and recorded dialogues. The reported results include gains on both evaluation sets, alongside trade-offs among correctness, brevity, and style.

4.1 TRAINING SETUP

  • Training uses 5,600 synthetic dialogues, a separate 560-dialogue development split for checkpoint selection, and 360 recorded single-turn dialogues reserved for transfer evaluation.The base model is the Thinker of Qwen3-Omni-30B-A3B-Instruct.Source: §4.1 · p. 7
  • GSPO uses group-mean reward advantages and clipped reply-probability ratios normalized by reply token count; training uses rank-64 LoRA adapters for 1,000 iterations.Each iteration uses 32 inputs with four sampled replies per input, and a checkpoint is saved every twentieth iteration.Source: §4.1 · p. 7

4.2 REWARDS AND EVALUATION METRICS

  • The reward combines rubric correctness, a training-only valid-format term, relative reply efficiency, and style; evaluation reports correctness, efficiency, and style.Efficiency is rubric credit relative to counted reply length, while the style score requires all seven grading criteria to pass.Source: §4.2 · p. 8

4.3 RESULTS AND DISCUSSION

  • Training curves show rubric reward rising from 0.308 to 0.629 and style from 0.750 to 0.992, while mean reply length falls from 91.5 to 36.6 words.Format stays at 1.000 without a within-group learning signal, and raw reply efficiency rises from 3.37 to 17.18 rubric-credit points per thousand counted words.Source: §4.3 · p. 8
  • 0.465 to 0.652 on OmniVChat-Bench and 0.402 to 0.632 on OmniVChat-Bench-Human at step 940; the Chinese synthetic control rises from 0.437 to 0.634.The selected checkpoint is marked by the dashed line; scores across the three corpora are not directly comparable.Source: §4.3 · pp. 8–9, Figure 4 · p. 9
  • Removing efficiency raises Mean to 0.697 and Human to 0.691, but length grows from 36 to 99 words and RE falls from 18.38 to 7.12.Removing style yields Mean 0.661, Human 0.632, and Style 0.788, compared with 0.992 for OmniVChat-RL.Source: §4.3 · p. 9
  • OmniVChat-RL improves rubric scores over the base model while raising RE from 5.75 to 18.38 and Style from 0.710 to 0.992.Among released systems, Gemini-3.5-Flash leads Mean at 0.667; Gemini-3.7-Flash leads Human at 0.575 and RE at 16.96, while Gemini-3.1-Pro leads Style at 0.871.Source: §4.3 · p. 9, Table 1 · p. 10
  • The mean multi-minus-single gap is −0.026 for OmniVChat-RL and −0.114 for Gemini-3.7-Flash, but independently sampled dialogues do not isolate the effect of more turns.OmniVChat-RL gains on connection checks and speaker transitions but scores lower on the other three extensions.Source: §4.3 · p. 10

5 CONCLUSION

The conclusion presents a pipeline spanning dialogue synthesis, benchmark evaluation, and reinforcement-learning rewards, with improvements reported on synthetic and recorded dialogue sets.

5 CONCLUSION

  • OmniVChat-Studio synthesizes single- and multi-turn audio-visual dialogues, and OmniVChat-Bench evaluates basic dialogue abilities across five categories and 17 subcategories.Studio supports single-turn probes and dependent turns, with reference replies and tiered rubrics.Source: §5 · p. 10
  • OmniVChat-RL jointly targets reply correctness, efficiency, and style; training on 5,600 synthetic dialogues raises OmniVChat-Bench from 0.465 to 0.652.Mean reply length falls from 79 to 36 words, while RE rises from 5.75 to 18.38 and Style from 0.710 to 0.992.Source: §5 · p. 10
  • On recorded OmniVChat-Bench-Human, the score rises from 0.402 to 0.632 without training on its recordings.The paper says these results support the training design and the use of synthetic data for training and evaluation in real-world dialogues.Source: §5 · p. 10

6 ETHICS STATEMENT

The ethics statement specifies informed consent for recorded participants.

6 ETHICS STATEMENT

  • All participants appearing in OmniVChat-Bench-Human sign consent authorizing research use of their recorded audio and video.The statement concerns the human-recorded benchmark.Source: §6 · p. 10

7 REPRODUCIBILITY STATEMENT

The reproducibility statement maps supplementary appendices to procedures, analyses, and training details.

7 REPRODUCIBILITY STATEMENT

  • Appendix C covers sampling and repair, Appendix D benchmark construction and judge agreement, and Appendix E training settings, grading prompts, and supplementary results.Artifact availability is stated in the Code and Data Release section.Source: §7 · p. 11

8 AI USE STATEMENT

The section declares that a language model assisted manuscript writing, while scientific claims, experimental designs, and results remain the authors’ responsibility. The related-work section reviews five research areas, including omni models.

8 AI USE STATEMENT

  • A language model assisted manuscript writing; the authors retain responsibility for all scientific claims, experimental designs, and results.Source: §8 · p. 11

B RELATED WORK

  • The related-work review covers omni models, benchmarks, joint audio-video generation, agent systems, and reinforcement learning.Source: B RELATED WORK · p. 15

B.1 OMNI MODELS

  • The omni-model review discusses systems with varied modality processing, alignment, routing, and real-time dialogue designs.Source: §B.1 · pp. 15–16

B.2 BENCHMARKS FOR AUDIO-VISUAL UNDERSTANDING AND DIALOGUE

The section surveys benchmarks and generation systems related to audio-visual understanding, alongside agentic generation and reinforcement-learning approaches. The appendix explains design choices and supplementary checks for OmniVChat-Studio.

B.2 BENCHMARKS FOR AUDIO-VISUAL UNDERSTANDING AND DIALOGUE

  • Audio-visual benchmarks span joint perception, temporal and streaming understanding, hallucination evaluation, and dialogue, with some preserving embedded queries in the original stream.Source: §B.2 · p. 16

B.3 JOINT AUDIO-VISUAL GENERATION

  • Recent systems generate audio and video jointly from multimodal conditions, with approaches supporting synchronized sound, reference inputs, or conversational editing.Source: §B.3 · pp. 16–17

B.4 AGENTIC VIDEO AND AUDIO GENERATION

  • Agentic generation systems coordinate specialized agents for planning, generation, review, and revision across video and audio production tasks.Source: §B.4 · p. 17

B.5 RL ALGORITHMS

  • The reviewed reinforcement-learning methods differ in policy objectives and reward sources, including rubric-based grading for responses without directly verifiable answers.Source: §B.5 · p. 17

C OMNIVCHAT-STUDIO: SUBSYSTEM DESIGN, ATTRIBUTE & CORPUS SAMPLING, AND VALIDATION

C.1 DESIGN RATIONALE

  • Repairs keep each sampled assignment fixed; reviewers report rendered evidence for the Director’s judgment, and repairs return to the stage that caused a defect.Source: §C.1 · pp. 17–18
  • Multi-turn media references use compact selection to retain needed older evidence without passing every preceding clip into the next turn.Source: §C.1 · p. 18

C.2 SUPPLEMENTARY SUBSYSTEM DETAILS

  • The supplementary subsystem details cover configuration, stage checks, bounded repair, and a simplified multi-turn trace.Source: §C.2 · p. 18

C.2.1 SUPPLEMENTARY CONFIGURATION AND SAMPLING DETAILS

The supplementary material details how attribute assignments are sampled and how the synthesis pipeline checks and repairs instances. It also describes benchmark post-processing and human inspection before delivery.

C.2.1 SUPPLEMENTARY CONFIGURATION AND SAMPLING DETAILS

  • Attribute sampling combines per-value probabilities with preference multipliers and exclusions, then selects assignments by random keys; an optional requirement pool can emphasize rare cases.Source: §C.2.1 · p. 18
  • Sampling coverage and cost depend on the joint assignment space: small batches may miss rare combinations, and failures or quality filters can alter the assignments ultimately delivered.Source: §C.2.1 · p. 18

C.2.2 SUPPLEMENTARY STAGE CHECKS, REPAIRS, AND MULTI-TURN TRACE

  • Stage checks enforce target-specific constraints, verify cross-turn media references, and use bounded repairs that can discard a run when required repair budgets are exhausted.Source: §C.2.2 · pp. 18–19

D OMNIVCHAT-BENCH: CONSTRUCTION AND AUDITS

D.1 POST-PROCESSING

  • Post-processing combines human inspection with deterministic repair of recurring audio tail artifacts.Source: §D.1 · p. 19

D.1.1 HUMAN INSPECTION OF THE DELIVERED SUITE

  • Every delivered instance passes synthesis gates, audio repair, and human review; inspectors check whether each clip shows the intended dialogue and can catch errors shared by the Reviewer and Director.Source: §D.1.1 · p. 20

D.1.2 TAIL-ARTIFACT DETECTION AND REPAIR

Tail repair reduces two recurring audio defects while preserving clip duration and earlier content. OmniVChat-Bench-Human adds 360 improvised Chinese recordings across twelve single-turn subcategories, with no multi-turn coverage.

D.1.2 TAIL-ARTIFACT DETECTION AND REPAIR

  • Detection combines fixed measurements with model-based ranking of candidate tails; in the strong tier, the model flagged 654 of the 692 clips found by the fixed measure.Isolated judgments can flag valid tails or report unsupported measurements, so repair uses the relative severity tier supported by both methods.Source: §D.1.2 · p. 21
  • The repair applies a 20 ms fade at the estimated cutoff, leaving earlier samples unchanged and preserving clip duration and audio-visual synchronization.At most three passes move the fade onset earlier if a transient remains; the video stream is copied without re-encoding.Source: §D.1.2 · p. 21
  • 695 clips were repaired, from 571 of 2,800 delivered instances; clips ending more than 300 ms after the content boundary were skipped.The first changed sample was a median of 80.2 ms before the file end and at most 136.2 ms before it.Source: §D.1.2 · p. 22
  • After repair, the median transient score fell from 86.6 to 0.02, and energy above 13 kHz at the original defect fell by a median of 53.1 dB; Figure 5 shows one repaired pair.Original and repaired clips were paired for playback in severity order, and their spectrograms were checked for residual artifacts.Source: §D.1.2 · p. 22, Figure 5 · p. 22

D.2 CONSTRUCTION OF OMNIVCHAT-BENCH-HUMAN

  • OmniVChat-Bench-Human contains 360 recordings across twelve single-turn subcategories, with 30 recordings each; its scope excludes five multi-turn subcategories.All recordings are in Chinese.Source: §D.2 · p. 22, Figure 6 · p. 23
  • Performers use subcategory-specific guides to improvise a selected dialogue variant on a phone in a real setting, without a clip-level script.Recordings are checked for target subcategory, clear speech, and usable audio and video; failures are corrected or recorded again.Source: §D.2 · p. 22
  • Three independent captions are merged into a factual record, and a human corrects transcript and factual errors against the recording.Before correction, one caption omitted user speech in 62 of 360 clips (17.2%); all merged records include a transcript.Source: §D.2 · pp. 22–23
  • Each accepted record receives a spoken reference reply and a validated rubric with three to five criteria, including exactly one Tier 0 criterion and at least one Tier 1 criterion.Validation rejects criteria referring to a hidden caption and checks that reference replies contain spoken words.Source: §D.2 · p. 23

D.3 JUDGE AGREEMENT AND SCORE VARIATION

Judge scores vary in strictness, but the judges produce similar subcategory profiles; repeated-grading and bootstrap variation provide reference scales for interpreting score differences.

D.3 JUDGE AGREEMENT AND SCORE VARIATION

  • 0.0007 was the pooled-score change across repeated qwen3.7-max grading runs; the bootstrap standard deviation for gpt-5.6-terra was 0.0066.The runs disagreed on 2.1% of replies; differences near 0.01 should be compared with these reference scales, which do not imply identical judgments.Source: §D.3 · p. 23
  • 0.034 was the pooled-score spread across six judges, while their pairwise criterion agreement ranged from 87.8% to 96.5%.The pooled spread was 5.2 times the bootstrap standard deviation; judge severity accounted for 1.1% of variance, while subcategory difficulty accounted for 95.7%.Source: §D.3 · pp. 23–24, Table 2 · p. 24
  • At least 0.936 Pearson correlation across subcategories indicates similar judge profiles despite score differences; all six judges ranked DSLP-VTT-UPC weakest.Spearman correlations were at least 0.860.Source: §D.3 · p. 24

D.4 COMPARING SINGLE-TURN AND MULTI-TURN SUBCATEGORIES

The appendix compares paired single-turn and multi-turn subcategories, defines the benchmark’s ability tests, and summarizes the OmniVChat-RL analyses covered there.

D.4 COMPARING SINGLE-TURN AND MULTI-TURN SUBCATEGORIES

  • -0.202 was the largest single-to-multi-turn gap, on MRR; Gemini-3.7-Flash showed a significant paired difference, while most OmniVChat-RL gaps fell within estimated standard errors.For Gemini-3.7-Flash, the paired test gave t(4) = −4.48 and p = 0.011; the OmniVChat-RL test was not evidence of equivalence, with 50 instances per multi-turn subcategory and only four degrees of freedom.Source: §D.4 · p. 24, Table 3 · p. 24
  • Across five pairs, post-training Pearson correlation was 0.75 and Spearman correlation 0.10, reflecting uneven OmniVChat-RL gains across pairs.Before training, the corresponding correlations were 0.88 and 0.90.Source: §D.4 · p. 24

D.5 DETAILED SUBCATEGORY DEFINITIONS

  • The benchmark has 17 subcategories: 12 single-turn tests with 2,550 instances and five multi-turn extensions with 250 instances.The extensions test memory, retrieval, or updates across turns.Source: §D.5 · p. 25
  • VCC tests connection checks, VTT tests whether to wait or answer, and VDMC tests user-relative left-right mapping, including after camera changes.Fixed connectivity clips use direct questions; left-right mapping reverses image sides for a user-facing camera but not an outward-facing camera.Source: §D.5 · p. 25
  • Reference Comprehension tests resolving speech, pronouns, gestures, or earlier unseen objects; diarization tracks the active speaker, while Identity Consistency resists misleading identity claims.MRR requires recalling visual evidence after the object leaves view and intervening topics occur.Source: §D.5 · p. 25

E OMNIVCHAT-RL: CONFIGURATION, ABLATIONS, AND TRANSFER

E.1 TRAINING CONFIGURATION

The appendix specifies OmniVChat-RL’s training configuration, including adapter scope, sampling, grading, and video-input settings.

E.1 TRAINING CONFIGURATION

  • Training uses rank-64 LoRA adapters on attention and mixture-of-experts projections, freezes vision and audio towers, and samples 32 prompts with 128 completions per iteration.The text grader handles rubric criteria and style separately without video; format is checked directly, the objective has no KL penalty, and video is sampled at 2 FPS.Source: §E.1 · p. 25

E.2 HELD-OUT LEARNING CURVES BY SUBCATEGORY

Held-out curves show learning gains across ability categories on synthetic and human-recorded benchmarks, with some subcategory gaps and distinct multi-turn trajectories.

E.2 HELD-OUT LEARNING CURVES BY SUBCATEGORY

  • Curves rise together most clearly for ICC, UCR, UPC, AOR, FGC, and PLA, while DR and SPSD retain a gap between synthetic and recorded data.Source: §E.2 · pp. 25–26, Figure 7 · p. 26
  • Development curves are omitted because each subcategory has only 10 to 60 examples.Source: Figure 7 · p. 26
  • MRR, MVMS, and MTIC improve over training, whereas MIST and MDCC remain more stable across training.Five multi-turn subcategories have Bench Only panels, with no recorded probe.Source: §E.2 · p. 26, Figure 7 · p. 26

E.3 REWARD ABLATIONS

The reward ablations compare removing style or efficiency while retaining the other auxiliary term, and their traces show distinct training behavior. The policy and grading prompts separate reply-style requirements from rubric judgments.

E.3 REWARD ABLATIONS

  • Each ablation removes either style or efficiency, retains the rubric, format reward, and other auxiliary term, and runs for 1,000 iterations.Both use the corpus, objective, adapter, reward, and decoding setup from Sections 4.1 and 4.2.Source: §E.3 · p. 26
  • Benchmark results are available only through step 960, whereas development reaches 0.734 at step 1000; accordingly, Table 1 reports checkpoint selection, not a convergence claim.Source: §E.3 · p. 27
  • Without efficiency, reply length grows while the rubric curve rises; without style, the style signal is absent throughout training.Figure 8 tracks reward terms, absolute Reply Efficiency, and development Subcategory Mean; its vertical line marks the checkpoint reported in Table 1.Source: §E.3 · p. 27, Figure 8 · p. 27

E.4 POLICY AND GRADING PROMPTS

  • The rubric grader judges explicit coverage of criteria independently, with model-name rules applying only to criteria that require reporting the model’s own name.The prompt says matching need not be verbatim and requires the remaining parts of a criterion to hold even when the name is correct.Source: §E.4 · p. 28
  • The style grader returns satisfied checklist items, while style reward is given only when all seven requirements pass.Evaluation Style averages the same indicator over successful grading requests and excludes request failures.Source: §E.4 · pp. 28–29
  • The policy prompt distinguishes concise fluent replies from replies that strip essential grammar, and the preliminary examples retain rubric content while omitting function words.Requirement 7 treats dropped grammar—not brevity—as the failure mode.Source: §E.4 · p. 29

E.5 CORRELATIONS ACROSS HELD-OUT LEARNING CURVES

This section examines held-out learning-curve correlations and presents synthetic and recorded audio-visual dialogue examples with reference replies and criteria for judging responses. The examples span single- and multi-turn prompts and include recorded cases that use visual context to address user questions.

E.5 CORRELATIONS ACROSS HELD-OUT LEARNING CURVES

  • 0.983 and 0.963 are the development–benchmark and development–probe level correlations, while their change correlations are only 0.291 and 0.087.Each curve has a Spearman correlation of at least 0.932 with training step; single-checkpoint change signs agree on 31, 27, and 28 of 50 intervals, and a sign test does not reject chance agreement.Source: §E.5 · p. 30
  • Independent additive measurement variation could reduce change correlations, but its assumptions are not established for repeated grading of fixed examples by one judge.The appendix interprets agreement on longer intervals as improvement together and says human-recording agreement supports transfer of the tested subcategories, without identifying the transfer mechanism.Source: §E.5 · p. 30
  • Residual standard deviations are 0.0066, 0.0044, and 0.0113 for development, benchmark, and probe, while their mean checkpoint gains are 0.0039, 0.0039, and 0.0044.The residual-to-mean-gain ratios range from 1.1 to 2.6, so short-term variation is comparable to the mean 20-step gain; residuals do not isolate measurement-noise variance.Source: §E.5 · p. 30
  • With 100-step blocks, development–benchmark and development–probe change correlations are 0.905 and 0.933, with matching signs on 8 of 9 transitions.For k = 10, correlations range from 0.966 to 0.996 and all four transitions agree; few transitions and possible temporal dependence limit inference.Source: §E.5 · p. 30

F OMNIVCHAT-BENCH: SYNTHETIC AND RECORDED EXAMPLES

F.1 A SYNTHETIC EXAMPLE FROM EACH SUBCATEGORY

  • Each of the seventeen subcategories has one English example, and each displayed synthetic record simplifies the full execution log.Spoken lines and reference replies remain verbatim; scene descriptions and rubrics are concise English translations of the Director’s Chinese records, and frames are sampled roughly once per second.Source: §F.1 · p. 30
  • A mocktail question tests whether a reply identifies the drink’s orange color while acknowledging glare, despite construction noise.The criteria also call for a precise color description and a natural conversational tone.Source: §F.1 · p. 36
  • An overlapping office conversation tests whether the reply answers the camera-facing speaker’s micro-lecture question without responding to colleagues in the background.The criteria reward actionable design advice, clear spoken ordering, and speaker distinction.Source: §F.1 · pp. 36–37
  • A user angry about back pain receives an empathetic reply with immediate self-care steps and warning signs for medical attention.The criteria call for at least two concrete measures and a calm, professional tone.Source: §F.1 · pp. 37–38
  • A casual identity question and a request to inspect pull-up form test truthful self-description, physical limitations, and useful alternatives.The examples include acknowledging visible context, suggesting a recording for review, and naming form details such as a side angle.Source: §F.1 · pp. 38–39
  • A two-turn festival exchange tests continuity by reassuring a user about a delay and offering to resume the French-drop discussion amid loud noise.The criteria require acknowledging the extreme noise and maintaining the earlier topic.Source: §F.1 · pp. 39–40
  • Lantern questions test left-right orientation across selfie and rear-facing views, with criteria requiring camera-mode recognition and current-frame perception.The second view places the large glowing carousel lantern on the user’s right, unlike the earlier selfie example’s left-shoulder lantern.Source: §F.1 · pp. 40–41
  • Synthetic examples and rubrics illustrate multi-turn recall, answering the current speaker amid noisy broadcasts, and handling a follow-up identity question after a workout exchange.The examples include recalling an earlier basket’s shape, answering a Diwali question while ignoring station announcements, and responding to a model-identity question without carrying over the prior topic.Source: §F.1 · pp. 44–46

F.2 A RECORDED EXAMPLE FROM EACH SUBCATEGORY

  • One displayed example represents each of the twelve human-recorded subcategories; records summarize annotation traces, use checked captions instead of synthesis scripts, and preserve Chinese speech and replies with English translations.Displayed faces are hidden, and displayed identifiers use eight-character hexadecimal strings, with hash-derived prefixes replacing non-hexadecimal source identifiers.Source: §F.2 · p. 46
  • Recorded cases and rubrics show tasks including confirming both audio and video, inviting a user to finish an incomplete question, and grounding a reply in a held object or gesture.The examples include channel confirmation, a listening prompt for an unfinished watch question, identifying which hand holds a peach, and resolving a gesture to a desk.Source: §F.2 · pp. 47–48
  • A reply corrects the user’s false Starbucks-logo description by identifying the pale cup’s bear motif and lettering, while remaining gentle and conversational.The rubric also requires the reply to be in Chinese and not accept the false description.Source: §F.2 · p. 51
  • For a book described as red, the reply corrects its off-white cover and still recommends bookmark colours that suit it and echo the nearby toy.The rubric calls for answering the bookmark question rather than stopping at the correction.Source: §F.2 · p. 51
  • For a user reporting a week-long cough and a prescription barrier, the reply explains the rule, suggests general over-the-counter options, and recommends medical care if symptoms persist.The rubric specifies answering both questions without naming a drug brand or inventing medical advice.Source: §F.2 · p. 52
Loading 2609.21465v1…