PaperlayerView exact revision on arXiv

Source-linked AI summary

Ex-Omni-2D: Expressive Omni-Modal Dialogue Models with Native Visual Presence

Haoyu Zhang, Zhipeng Li, Xiaoying Tang, Tianshu Yu, Yiwen Guo

arXiv:2608.10720v1cs.AIcs.CLcs.CV

TL;DR

Omni-modal dialogue systems can answer multimodal queries with speech but remain visually disembodied. Ex-Omni-2D coordinates response text, personalized speech, and reference-conditioned video through visual planning and shared speech units, with Teacher and Streaming Student models offering complementary quality and efficiency. The framework provides a practical quality–efficiency operating point for incremental omni-modal responses.

  • Problem

    Omni-modal dialogue systems lack visually embodied responses, while dialogue-derived visual guidance, paired training data, and efficient streaming generation remain limited.

  • Method

    Ex-Omni-2D predicts a Visual Thought Plan and response, then uses native multi-codebook speech units to coordinate personalized speech with frame-aligned, reference-conditioned video.

  • Results

    The full-sequence Teacher and incremental Streaming Student provide complementary quality- and efficiency-oriented operating points across dialogue, speech, video, synchronization, and efficiency benchmarks.

  • Takeaways & Limitations

    The framework supports dialogue-native video responses by reconnecting separately trained modality pathways through visual plans and shared speech units.

  • Takeaways & Limitations

    Reference-speaker similarity remains improvable, while the full-sequence Teacher is computationally expensive and the Streaming Student’s long-horizon analysis is limited.

Abstract

from arXiv · show

Omni-modal dialogue models can understand multimodal inputs and synthesize spoken replies, yet their responses remain visually disembodied. We introduce \textbf{Ex-Omni-2D}, an omni-modal dialogue framework that generates a coordinated response comprising text, personalized speech, and reference-conditioned video. Given a multimodal query, reference image, and reference audio, the model predicts a structured \textit{Visual Thought Plan} (VTP) describing scene, emotion, and motion, followed by response text and native multi-codebook speech units. These units form a shared acoustic-temporal interface: they are decoded into speech and aligned online with video frames. This interface enables the response and avatar pathways to be learned from heterogeneous speech, dialogue, and avatar-video data, avoiding the need for large-scale query--text--speech--video supervision. A full-sequence Video Generator serves as the primary Teacher. For efficient incremental generation, we further distill it into a few-step block-causal \emph{Streaming Student} whose Prefix Streaming mechanism carries a clean latent across consecutive chunks to reduce cumulative late-chunk degradation. With four-step inference, the complete four-GPU pipeline achieves an end-to-end RTF of 1.293 at $400\times720$/$720\times400$, providing a practical quality--efficiency operating point.

1 Introduction

Ex-Omni-2D extends omni-modal dialogue with dialogue-native, reference-conditioned video alongside response text and personalized speech. It uses a Visual Thought Plan and native speech units to coordinate visual behavior and audio, with Teacher–Student video generation for quality and efficient incremental inference.

  • Problem and setting: Ex-Omni-2D generates response text, personalized speech, and synchronized reference-conditioned video from a speech and/or text query, reference image, and reference audio.Its dialogue-native interface plans visual behavior from the current dialogue and produces the acoustic-temporal condition within the response process.
  • Motivation: Existing avatar methods typically use completed or externally supplied waveforms and optional manual prompts, lacking a direct mechanism to derive response-specific visual intent from dialogue state.The paper motivates inferring visual guidance from dialogue context and the reference image instead.
  • Dialogue-native visual planning: A structured Visual Thought Plan converts dialogue context into explicit guidance for scene, emotion, movement style, and motion details.The VTP is generated before the user-facing response.
  • Shared speech–video interface: Native multi-codebook speech units provide a shared acoustic-temporal interface for personalized speech rendering and online frame-aligned video conditioning.The interface supports pathway-specific learning from heterogeneous speech, dialogue, and avatar-video data without fully paired video-response dialogues.
  • Efficient video generation: A full-sequence Video Generator serves as the Teacher, while a block-causal Streaming Student with Prefix Streaming enables efficient few-step incremental inference.Prefix Streaming is introduced to reduce cumulative late-chunk subject degradation; the Teacher and Student provide complementary quality- and efficiency-oriented operating points.

2 Related Work

Prior work advances omni-modal dialogue and coordinated audio-video generation through unified, cascaded, and joint paradigms. Ex-Omni-2D instead formulates dialogue-conditioned responses around explicit semantic-planning and acoustic-timing interfaces with modality-specific generators.

  • Omni-modal Dialogue: Omni-modal dialogue models unify understanding and generation across speech, vision, and language, including instruction following, spoken responses, cross-modal alignment, and multimodal generation.These systems are developed within LLM-based architectures.
  • Unified Audio-Video Generation: Audio-video generation includes cascaded systems for audio-driven video synthesis and video-to-audio dubbing, alongside joint approaches for coordinated content.Audio-driven systems animate a reference identity from speech, whereas reverse-direction systems generate sound or speech for silent video.
  • Positioning: Ex-Omni-2D is a dialogue-conditioned response formulation rather than a new standalone renderer or a claim that all modalities share one generator.Its modality-specific generators are trained with the supervision available to them.
  • Positioning: The framework introduces explicit interfaces that produce semantic planning and acoustic timing within the response process.The overview describes a Visual Thought Plan, user-facing response, reference-conditioned speech generation, and synchronized video synthesis.

3 Method

Ex-Omni-2D coordinates text, personalized speech, and reference-conditioned video through a structured Visual Thought Plan and native multi-codebook speech units. A full-sequence Video Generator provides the visual Teacher, while a block-causal Streaming Student enables incremental generation with Prefix Streaming for long-horizon stability.

  • Framework Overview: Ex-Omni-2D converts a multimodal query, reference image, and reference audio into response text, personalized speech, and reference-conditioned video.The framework routes image and voice references through the components that directly consume them.
  • Dialogue and Visual Planning: The dialogue backbone generates a structured five-field Visual Thought Plan before response text, separating internal visual intent from the user-facing answer.The VTP describes first-frame scene, overall scene, emotion, movement style, and detailed motion, and is encoded as a semantic video condition.
  • Shared Speech–Video Interface: 16 acoustic codebooks form the shared speech representation, decoded into waveform audio and projected into video-conditioning features aligned at 12.5 Hz speech and 25 FPS video.Each acoustic feature is repeated for two video frames, enabling explicit temporal alignment while decoupling speech and video supervision.
  • Video Generation and Distillation: The full-sequence Video Generator uses bidirectional temporal context and later serves as the Teacher for distilling the incremental Streaming Student.The Teacher is built on Wan2.1-T2V-1.3B with OmniAvatar-1.3B LoRA initialization and conditions denoising on video, reference, speech, and VTP features.
  • Streaming Student: Prefix Streaming gives the block-causal Student a persistent clean reference anchor and preceding terminal latent, preventing cumulative drift across four-slot denoising windows.Stop-gradient and overlap-deduplicated cache updates prevent the anchor from being regenerated or counted twice.

4 Experiments

Experiments evaluate Ex-Omni-2D across audio-video quality, speech QA, multi-turn dialogue, and streaming efficiency under fixed reference and controller conditions. The model achieves strong dialogue and QA results, while Prefix Streaming improves later-chunk visual consistency and quality.

  • Benchmarks and reference conditions: Evaluation uses 200 CommonEval speech queries with fixed paired reference image and speech conditions across compared methods.Spoken QA uses AlpacaEval, CommonEval, and BBH; multi-turn dialogue uses the 400-dialogue OmniCharacter benchmark and its official CharacterEval protocol.
  • Metrics and comparison protocol: Audio-video evaluation reports PQ, CU, and SIM for audio, SC, IQ, and DD for video, plus A-V synchronization metrics.Cascaded baselines share Qwen2.5-Omni-7B as the response controller, while downstream renderers do not receive credit for PQ or CU.
  • Multi-turn dialogue quality: 3.938 average Fluency, Coherency, and Consistency exceeds Qwen3-8B’s 3.537, while the twelve-dimension average is 3.283 versus 3.264.Ex-Omni-2D has the highest reported values among compared methods on the three dialogue metrics, but metricwise increases and decreases indicate a capability trade-off.
  • Speech question answering: 4.28 on AlpacaEval, 3.71 on CommonEval, and 58.70 on BBH are the second-highest listed scores after Qwen2.5-Omni.The appendix separately reports backbone and VTP ablations, including the language-capability trade-off associated with VTP generation.
  • Streaming Student analysis: Prefix Streaming changes SC from 92.85 to 93.65, IQ from 55.18 to 57.40, and Sync-C from 3.69 to 3.90, while DD remains 32.00.Across 200 CommonEval samples, the Prefix-Streaming Student reduces later-chunk facial or body degradation relative to the earlier no-prefix Student.

5 Conclusion

Ex-Omni-2D is an expressive omni-modal dialogue framework with dialogue-native video responses. It conditions appearance and voice on reference image and audio, makes visual intent explicit through a Visual Thought Plan, and uses native multi-codebook speech for speech rendering and frame-aligned avatar motion.

  • Ex-Omni-2D provides dialogue-native video responses within an expressive omni-modal dialogue framework.
  • Reference image and audio condition appearance and voice, while a Visual Thought Plan specifies response-specific visual intent.
  • A native multi-codebook speech space supports both speech rendering and frame-aligned avatar motion.

Limitations · A Evaluation Metric Details

Ex-Omni-2D’s speech similarity, video-control guidance, and cross-modal planning remain limited, while evaluation uses specialized audio, video, synchronization, and speech-QA metrics. These metrics assess perceptual quality, intelligibility, temporal consistency, frame quality, motion, audiovisual alignment, and question-answering correctness.

  • Limitations: Speech similarity to the reference speaker remains an area for improvement despite intelligible generated speech.
  • Limitations: VTP provides high-level semantic guidance, while final video realization also depends on frame-aligned speech units and fixed Text CFG and Audio CFG balance.The fixed guidance balance may not transfer uniformly across responses.
  • Limitations: Generating VTP in the shared autoregressive channel creates a measurable trade-off involving speech QA and reasoning.The passage motivates planner isolation and adaptive cross-modal guidance as future directions.
  • A Evaluation Metric Details: AudioBox-Aesthetics evaluates generated speech with PQ for perceptual production quality and CU for practical usefulness, intelligibility, and semantic completeness.CU is intended to capture usability and understandability better than strict word-error-rate matching.
  • A Evaluation Metric Details: VBench evaluates generated videos without requiring ground-truth video, using SC for subject consistency, IQ for frame quality, and DD for substantial motion.DD uses optical-flow magnitude to report the proportion of videos with substantial motion.
  • A Evaluation Metric Details: SyncNet confidence, or Sync-C, measures temporal alignment between mouth movements and generated speech.Higher Sync-C values indicate more confident synchronization because the metric compares the best alignment against competing temporal offsets.
  • A Evaluation Metric Details: VoiceBench speech-QA evaluation uses AlpacaEval, CommonEval, and BBH with the open-source evaluation code.AlpacaEval and CommonEval use GPT-4o-mini, while BBH subtasks convert responses into binary labels through task-specific matching or regular expressions.
  • A Evaluation Metric Details: The final BBH score is computed as the percentage of correct predictions.

B Additional Dialogue Capability Results

This section evaluates the language-capability effects of VTP supervision and compares Ex-Omni-2D with its original Qwen3-8B backbone under open-ended VoiceBench evaluation. VTP supervision lowers language scores, while Ex-Omni-2D achieves reported text-input scores on AlpacaEval and CommonEval.

  • Effect of VTP supervision: Adding VTP supervision changes CommonEval from 3.82 to 3.71 and BBH from 61.10 to 58.70, decreasing scores by 0.11 and 2.40 points.The comparison uses response-only training without VTP supervision as the reference setting.
  • Backbone capability comparison: With text input, Ex-Omni-2D obtains 4.48 on AlpacaEval and 4.35 on CommonEval in the open-ended VoiceBench protocol.The comparison is against the original Qwen3-8B backbone; the text-only backbone has no native speech interface.

C Multi-turn Context Ablation

The ablation compares context-free and full dialogue-history settings on the 400-dialogue OmniCharacter test set under identical Role Card and CharacterEval evaluation. Providing preceding question–answer turns improves every evaluated dimension, raising the dialogue-level macro average from 2.769 to 3.283.

  • C Multi-turn Context Ablation: The ablation isolates dialogue history on the 400-dialogue OmniCharacter test set.Both settings receive the benchmark Role Card and are scored on the final turn with the same CharacterEval reward model.
  • C Multi-turn Context Ablation: The context-free setting receives only the final user query, while the full setting also receives all preceding question–answer turns.This isolates the contribution of multi-turn dialogue context.
  • C Multi-turn Context Ablation: Every evaluated dimension scores higher when dialogue history is provided, with especially large gains in persona behavior, persona utterance, expression diversity, and response consistency.The passage identifies these dimensions as showing especially large differences.
  • C Multi-turn Context Ablation: 3.283 versus 2.769: the dialogue-level macro average increases from 2.769 (95% CI: 2.725–2.812) to 3.283 (3.242–3.322).The paired difference is 0.514 with a 95% bootstrap CI of 0.464–0.562.

D Streaming Student Denoising-Step Analysis

The denoising-step analysis varies only the inference-time video denoising budget while holding the Prefix-Streaming checkpoint, evaluation inputs, and random seed fixed. Increasing the budget consistently improves video quality and synchronization metrics, while the speech pathway remains invariant.

  • Experimental setup: The experiment changes only the inference-time denoising budget, using the same validation-selected Prefix-Streaming checkpoint, inputs, and random seed.The evaluation uses 200 CommonEval requests with reference images, VTPs, speech units, and waveforms.
  • Metric scope: The speech pathway is invariant across denoising-step counts, so the analysis reports only video and synchronization metrics.
  • Denoising-step effects: Increasing the denoising budget consistently increases SC, IQ, DD, and Sync-C.For DD, the increase indicates a larger fraction of outputs with substantial motion.

E Long-Horizon Prefix Analysis · F User-Visible Streaming Latency

The long-horizon analysis evaluates subject consistency and facial dynamics across all 16 available chunks, while user-visible latency measures when speech and video first become playable. Prefix Streaming improves facial continuity and motion smoothness, but initial audible speech and video remain delayed by response generation.

  • E Long-Horizon Prefix Analysis: Prefix Streaming is evaluated across all 16 available 0.48-second chunks for broad subject-level consistency.The comparison uses DINO curves from Figure 4.
  • E Long-Horizon Prefix Analysis: Facial dynamics are computed from 106 rigid-aligned InsightFace landmarks after removing translation, scale, and in-plane rotation.Each video’s first second defines its own dynamics baseline.
  • E Long-Horizon Prefix Analysis: Prefix Streaming yields higher face-detection continuity at every tail threshold.This result is reported in Table 11.
  • E Long-Horizon Prefix Analysis: Prefix Streaming yields lower motion-normalized acceleration and jerk at every tail threshold.These facial-dynamics results are reported relative to each video’s first-second baseline.
  • F User-Visible Streaming Latency: 2.308 seconds is the delay to first audible speech in the four-GPU, four-step Prefix-Streaming pipeline.Table 12 measures event-level latency from request arrival and distinguishes internal planning from user-hearable output.
  • F User-Visible Streaming Latency: 3.142 seconds is the delay to the first playable 0.48-second video chunk.The measurement is taken from request arrival in the four-GPU, four-step Prefix-Streaming pipeline.

G Full Inference Scaling Results · H Upstream Response Source Ablation · I Additional Visual Planning Analysis

The paper evaluates inference scaling for the Streaming Student, diagnoses upstream response-source effects on fixed downstream generators, and analyzes visual-plan grounding and multimodal realization alignment. These evaluations separate serving efficiency, renderer sensitivity, and consistency between plans, speech, video, and reference images.

  • G Full Inference Scaling Results: All scaling measurements use the same validation-selected Prefix-Streaming checkpoint, while denoising-step groups differ only in inference-time budgets.The Teacher uses single-request model parallelism in multi-GPU settings.
  • G Full Inference Scaling Results: Four GPUs provide the lowest or near-lowest single-request Student E2E RTF, whereas eight GPUs use two replicas for higher aggregate throughput.The workload contains 30 OmniCharacter requests, with 10 requests per resolution; warmups are excluded.
  • H Upstream Response Source Ablation: The study compares Qwen2.5-Omni and Ex-Omni-2D outputs as upstream sources for each fixed downstream generator.Reference conditions, downstream checkpoints, evaluation tools, and 200 CommonEval requests remain unchanged within each pair.
  • H Upstream Response Source Ablation: This upstream ablation is diagnostic, measuring downstream renderer responses to generated speech or response conditions rather than end-to-end dialogue capabilities.The comparison keeps the downstream setup fixed within each pair.
  • I Additional Visual Planning Analysis: Table 15 evaluates whether the Visual Thought Plan grounds its first-frame scene description in the supplied reference image.The first_frame_scene field is compared against the original reference image.
  • I Additional Visual Planning Analysis: Remaining plan fields are judged against nine chronological generated-video samples, excluding frame 0 because it repeats the reference image.These diagnostics assess consistency among the naturally co-generated plan, speech, and video rather than isolating one conditioning pathway.

J Additional Qualitative Results … L.1 Data Construction

The paper reports qualitative sensitivity to video guidance and describes heterogeneous training objectives that match available supervision. It constructs speech, dialogue, and avatar-video data with structured VTP annotations and native multi-codebook speech conditions.

  • J Additional Qualitative Results: Guidance scale 4.0 produces larger motion amplitude than scale 2.0, while a smiling reference yields a visually smooth facial transition.These examples demonstrate sensitivity to guidance settings but do not establish a generally optimal scale.
  • K Optimization Details: The framework enables different training objectives according to the supervision available in each stage.Its structured assistant sequence contains both the target VTP and user-facing response, which language modeling supervises autoregressively.
  • K Optimization Details: The Speech Generator predicts a target multi-codebook sequence from response-span hidden states and text tokens, using autoregressive and residual-codebook losses.The first codebook is autoregressive, while residual codebooks are predicted within each acoustic frame.
  • K Optimization Details: The VTP is directly supervised within the language-modeling objective, then encoded by a frozen video text encoder for video denoising.The denoising model predicts the flow target from the noised latent and three video conditions.
  • L Training Details: Each objective term is enabled only for the pathway and supervision available in the current batch, without requiring fully paired dialogue, speech, and video supervision.This formulation supports heterogeneous training stages with different available supervision.
  • L.1 Data Construction: Stage 1 uses about 800K ASR examples and 1M TTS examples, drawing ASR data from LibriSpeech and Emilia and synthesizing TTS speech with Qwen3-TTS.The authors use Qwen3-TTS because its speech quality supports learning usable speech output from moderate synthesized-data scale.
  • L.1 Data Construction: Stage 2 combines InstructS2S-200K speech-to-speech supervision with OmniCharacter character-grounded multi-turn dialogue to adapt the VTP–response protocol and shared speech interface.These data update the LLM, Speech Projector, and Speech Generator toward interactive and character-consistent behavior.
  • L.1 Data Construction: Stage 3 filters SpeakerVid into about 140K clips and converts each retained clip into a record with a reference frame, five-field VTP, target video, and speech tokens.Aligned speech is tokenized into 16 codebook streams at 12.5 Hz, with source clips aligned at 16 kHz audio and 25 FPS video.

L.2 Framework Training and Student Distillation … M Prompt Templates

The framework is trained through staged speech alignment, omni-modal response adaptation, avatar video realization, and two-phase streaming-student distillation, using heterogeneous supervision without target response videos. Prefix caching and rate-aligned chunks support incremental video generation, while prompt templates define VTP annotation and visual-grounding evaluation.

  • L.2 Framework Training and Student Distillation: Stage 1 aligns speech interfaces with approximately 800K ASR and 1M TTS examples while freezing the LLM, Speech Encoder, and Vision Encoder.ASR updates the Speech Projector, while TTS trains the Qwen3-TTS-initialized Speech Generator to predict native multi-codebook units.
  • L.2 Framework Training and Student Distillation: Stage 2 adapts the aligned interface using InstructS2S-200K and OmniCharacter, training structured VTP-response generation without target response videos.Speech supervision is restricted to the response span, while the LLM learns the <thinking>–<response> protocol.
  • L.2 Framework Training and Student Distillation: Stage 3 trains the full-sequence avatar diffusion model on 140K SpeakerVid-derived records with reference images, annotated VTPs, speech units, and target videos.The interface-based decomposition lets dialogue data teach response planning and SpeakerVid data teach rendering, without synthetic user queries.
  • L.2 Framework Training and Student Distillation: Stage 4 freezes the full-sequence Video Generator as Teacher and distills a Streaming Student through flow-map learning followed by on-policy chunked rollouts with distribution matching.The flow-map objective is retained during Phase II, while the hierarchy keeps the three data-oriented training stages distinct from distillation.
  • L.2.1 Prefix and Cache Implementation: The Prefix Streaming cache re-injects a clean prefix before denoising and advances the global offset by three, preventing duplicated positions and transient noisy features in persistent history.The completed clean chunk is encoded into the cache only after the final transition through an additional identity-map forward.
  • L.2.2 Rate-Aligned Streaming Chunks: 0.48 s streaming blocks map six acoustic frames to exactly 12 decoded video frames, with one overlap prefix and three new latent frames per causal block.The overlap is excluded from VAE emission and external output, so the public stream advances by exactly 12 frames.
  • M.1 VTP Annotation Prompt: M.1 prompts annotate retained videos with XML-style VTP fields covering first-frame scene, scene, emotion, movement style, and detailed motion description.The rules require visible, verifiable, concise, generation-oriented descriptions and prohibit unsupported semantic speculation.
  • M.2 VTP Visual-Grounding Judge Prompt: M.2 evaluates VTP-video alignment using the original reference image for first-frame scene and chronological generated-video samples for the remaining fields.Each of the five fields receives exactly one binary point based on visible support, with movement requiring temporal evidence.
Loading 2608.10720v1…