Source-linked AI summary
OmniShow: Unifying Multimodal Conditions for Human-Object Interaction Video Generation
Donghao Zhou, Guisheng Liu, Hao Yang, Jiatong Li, Jingyu Lin, Xiaohu Huang, Yichen Liu, Xin Gao, Cunjian Chen, Shilei Wen, Chi-Wing Fu, Pheng-Ann Heng
TL;DR
HOIVG seeks realistic human-object interaction videos conditioned on text, reference images, audio, and pose, while existing approaches do not accommodate all four conditions. OmniShow unifies these modalities through specialized conditioning, attention, staged training, and a dedicated benchmark. It achieves state-of-the-art or superior performance across multimodal conditioning settings, while evaluation remains limited to short clips and AI-generated reference images and extreme inputs can produce artifacts.
Problem
Existing HOIVG methods do not provide an end-to-end framework that harmonizes text, reference image, audio, and pose conditions.
Method
OmniShow combines Unified Channel-wise Conditioning, Gated Local-Context Attention, Decoupled-Then-Joint Training, and HOIVG-Bench to unify multimodal generation and evaluation.
Results
OmniShow achieves state-of-the-art performance across various multimodal conditioning settings on HOIVG-Bench.
Takeaways & Limitations
The framework provides a unified basis for multimodal HOIVG and supports coordinated generation across diverse conditioning settings.
Takeaways & Limitations
Evaluation focuses on 5-second clips, uses AI-generated human reference images, and may produce artifacts or blur under extreme motion or conflicting inputs.
Abstract
from arXiv · showhide
In this work, we study Human-Object Interaction Video Generation (HOIVG), which aims to synthesize high-quality human-object interaction videos conditioned on text, reference images, audio, and pose. This task holds significant practical value for automating content creation in real-world applications, such as e-commerce demonstrations, short video production, and interactive entertainment. However, existing approaches fail to accommodate all these requisite conditions. We present OmniShow, an end-to-end framework tailored for this practical yet challenging task, capable of harmonizing multimodal conditions and delivering industry-grade performance. To overcome the trade-off between controllability and quality, we introduce Unified Channel-wise Conditioning for efficient image and pose injection, and Gated Local-Context Attention to ensure precise audio-visual synchronization. To effectively address data scarcity, we develop a Decoupled-Then-Joint Training strategy that leverages a multi-stage training process with model merging to efficiently harness heterogeneous sub-task datasets. Furthermore, to fill the evaluation gap in this field, we establish HOIVG-Bench, a dedicated and comprehensive benchmark for HOIVG. Extensive experiments demonstrate that OmniShow achieves overall state-of-the-art performance across various multimodal conditioning settings, setting a solid standard for the emerging HOIVG task.
1 Introduction
HOIVG requires coordinating text, reference images, audio, and pose, but existing methods do not unify these conditions end to end. OmniShow addresses this gap with multimodal conditioning, specialized attention, staged training, and HOIVG-Bench evaluation.
- Motivation: HOIVG synthesizes high-quality human-object interaction videos from text, reference images, audio, and pose for applications including e-commerce and entertainment.Text provides global semantics, reference images specify appearance, audio supports synchronized movements, and pose provides explicit motion control.
- Motivation: Existing methods separately support reference preservation, audio synchronization, or combined audio-image control, but no unified framework handles all four required conditions end to end.R2V methods lack audio responsiveness, A2V methods generally use only an initial frame, and multimodal approaches often omit pose or high-fidelity subject preservation.
- OmniShow: OmniShow introduces Unified Channel-wise Conditioning and Gated Local-Context Attention to provide precise multimodal control without compromising generation quality.The former injects image and pose information through channel concatenation, while the latter restricts video tokens to corresponding audio segments and uses learnable gates.
- OmniShow: Decoupled-Then-Joint Training uses staged specialization and model merging to harness heterogeneous sub-task datasets despite scarce paired HOIVG data.Separate A2V and R2V models are trained first, then fused for joint training before pose is introduced.
- Evaluation: HOIVG-Bench provides a dedicated comprehensive evaluation, where OmniShow achieves superior or competitive performance across multiple multimodal conditioning settings.The benchmark addresses the field’s lack of standardized evaluation for diverse HOIVG conditions.
2 Related Work
Related work has developed controllable video generation through reference images, audio, and pose, but these streams remain insufficiently unified for HOIVG. The paper positions robust coordination of all four conditions as an open challenge.
- Controllable Video Generation: Controllable video generation extends conditioning beyond text, including reference-to-video methods that preserve the identity of subjects in input images.The related literature spans conditional image synthesis and controllable video generation.
- Controllable Video Generation: Audio-to-video research has progressed from talking heads to portrait animation and multi-person conversation, while pose-guided methods direct motion using structural signals.Pose controls include skeleton maps and dense correspondences.
- HOIVG: Recent multimodal approaches combine selected conditions, but a robust framework coordinating text, reference image, audio, and pose remains an open challenge.HOIVG specifically targets realistic human-object interaction videos grounded in multimodal conditions.
3 Methodology
OmniShow unifies multimodal conditioning through channel-wise image and pose injection, gated local audio attention, heterogeneous-data training, and a dedicated benchmark. Its components preserve reference semantics, align audio with video, and support systematic HOIVG evaluation.
- OmniShow combines Unified Channel-wise Conditioning, Gated Local-Context Attention, Decoupled-Then-Joint Training, and HOIVG-Bench for multimodal HOIVG.The framework is built on Waver 1.0 and supports flexible combinations of multimodal inputs.
- Unified Channel-wise Conditioning: Reference images and pose are injected by concatenating encoded tokens with noisy video and pseudo-frame tokens along the temporal axis.The unified design preserves the native conditioning structure while assimilating global appearance and temporally aligned pose cues.
- Unified Channel-wise Conditioning: A reference reconstruction loss supervises pseudo-frames, while the conditioning mechanism improves video quality and reference consistency over token concatenation.Removing the reconstruction loss particularly reduces visual fidelity for human identity.
- Gated Local-Context Attention: Gated Local-Context Attention packs contextual audio features, restricts attention to corresponding audio segments, and uses a near-zero learnable gate for stable injection.The gate also indicates audio impact and motivates placing audio attention in dual-stream blocks.
- HOIVG-Bench: HOIVG-Bench provides a comprehensive evaluation suite for multimodal synergy, including text, human and object images, audio, and pose.The benchmark contains 135 carefully curated samples.
4 Experiments
OmniShow is evaluated against existing methods across varied multimodal settings using quantitative, qualitative, human-preference, ablation, and application studies. The results show competitive or superior performance, with ablations supporting the proposed conditioning, attention, and training designs.
- Quantitative Comparison: OmniShow achieves superior or highly competitive performance across diverse multimodal conditioning settings while supporting a broader condition spectrum than existing baselines.Baselines lack support for all four inputs, so evaluations use R2V, RA2V, and RP2V settings; OmniShow uniquely supports RAP2V generation.
- Qualitative Comparison: OmniShow preserves references, produces natural motion, and synchronizes audio and visuals across varied qualitative scenarios.In R2V, it avoids implausible object composition; in RA2V, it combines natural body movements with precise lip synchronization.
- Human Evaluation: Human evaluators preferred OmniShow in most side-by-side comparisons for condition adherence and overall visual quality.The study included 30 RA2V participants and 33 RP2V participants, each assessing randomly selected subsets of 20 samples.
- Ablation Studies and Analysis: Unified Channel-wise Conditioning improves video quality and reference consistency over token concatenation, while reconstruction loss reinforces semantic and human-identity preservation.Removing the reference reconstruction loss reduces visual fidelity, especially for human identity.
- Ablation Studies and Analysis: Gated Local-Context Attention improves temporal coherence and synchronization, and adaptive gating supports final visual quality and training stability.Audio context improves Sync-D, attention constraints boost synchronization, and disabling adaptive gating degrades visual quality.
- Ablation Studies and Analysis: Decoupled-Then-Joint Training achieves the best trade-off between reference consistency and audio-visual synchronization among compared training strategies.Single-stage training suffers from poor convergence, while naive multi-stage training struggles to incorporate new modal inputs.
5 Discussion
The discussion identifies minimalist intervention and decoupled-then-joint training as key principles for multimodal control and heterogeneous-data utilization. It also notes evaluation scope limits, possible artifacts, and future directions.
- Minimal architectural changes enable multimodal control by repurposing native channel concatenation and analyzing gate vectors.
- Weight merging can produce RA2V capabilities by combining R2V and A2V models, suggesting controllability can emerge without relying solely on paired data.
- Evaluation focuses on 5-second clips although OmniShow can generate videos up to 10 seconds, matching baselines that support only short clips.
- AI-generated human reference images may introduce distribution bias, and intense motion or conflicting inputs can occasionally produce artifacts or blur.
- The ablation studies evaluate Unified Channel-wise Conditioning, Gated Local-Context Attention, and Decoupled-Then-Joint Training.
- Future work includes reinforcement-learning post-training, larger datasets, and greater model capacity to improve dynamics and extreme-condition robustness.
6 Conclusion
The conclusion presents OmniShow as a unified HOIVG framework that combines multimodal control with heterogeneous-data utilization. Experiments on HOIVG-Bench support its reported superiority, while future work targets broader scale and inputs.
- OmniShow orchestrates text, reference image, audio, and pose conditions for precise multimodal control and high-quality video generation.
- Extensive comparisons and ablation studies on HOIVG-Bench verified OmniShow’s superiority.
- The authors envision expanding capabilities with larger datasets, richer inputs, and broader application fields.
Impact Statement
The impact statement recognizes benefits for digital content creation and education alongside risks from generating deceptive or harmful content. It advocates responsible use and continued safeguards.
- The technology may enhance accessibility in digital content creation and education.
- It also carries misuse risks involving deceptive or harmful generated content, motivating responsible use and safeguards.
A Training Data Collection
The data pipeline constructs heterogeneous HOIVG resources by segmenting and filtering videos, applying task-specific processing, and curating approximately 1 million clips totaling around 3500 hours. Validation includes quantitative and qualitative comparison on RAP2V.
- Pipeline overview: The pipeline segments videos into isolated shots, filters them using diverse metric scores, and applies specialized workflows to build heterogeneous datasets.
- Stage 2: Multi-dimensional Filtering: Multi-dimensional filtering evaluates resolution, aesthetics, motion intensity, and OCR-related quality to remove low-quality samples.
- Stage 3: Data-Special Processing: The processed data are divided into R2V, A2V, RA2V, and RAP2V datasets for distinct multimodal training objectives.
- RAP2V validation: On RAP2V, OmniShow outperforms the cascaded VACE [31]+LatentSync [35] baseline across all evaluation metrics.
- RAP2V validation: Qualitatively, OmniShow produces coherent RAP2V videos with precise lip synchronization and fewer artifacts or blur under complex occlusions.
- Dataset scale: O(1m) clips totaling around 3500 hours are curated with video captioning covering subjects, objects, actions, environments, and interactions.
B More Implementation Details
OmniShow is built on a 12B Waver 1.0 MMDiT model trained with Flow Matching. Training progresses from 480p to 720p videos and uses sequence parallelism and offline feature extraction for efficiency.
- OmniShow uses the 12B Waver 1.0 model with a Multimodal Diffusion Transformer architecture and Flow Matching supervision.
- Training begins with 480p videos and then proceeds to a 720p high-resolution phase.
- Ulysses-style sequence parallelism with size 8 supports training samples of up to 241 frames at 24 fps.
- All features are extracted offline before training to improve training efficiency.
C Additional Comparison
Against a cascaded VACE–LatentSync baseline for RAP2V, OmniShow performs better across synchronization, pose, video-quality, and motion-quality metrics, supporting end-to-end condition modeling.
- The RAP2V cascaded baseline sequentially combines VACE for RP2V generation with LatentSync for audio-driven lip synchronization.
- OmniShow achieves a Sync-C of 7.699 compared to 7.016 for the cascaded baseline.
- OmniShow achieves a PCK of 0.478 compared to 0.340 for the cascaded baseline.
- OmniShow achieves a VQ of 11.06 versus 10.05 for the cascaded baseline.
- Qualitative comparisons show more coherent videos and better visual fidelity for OmniShow than for the cascaded baseline.
D Additional Ablation Studies
Additional ablations favor native RoPE indexing for pseudo-frames and a context window of w = 5 for Gated Local-Context Attention, which balances temporal context with fine-grained synchronization cues.
- The R2V ablation compares native, Temporal Shift, and Spatiotemporal Shift RoPE strategies for pseudo-frame positional encoding.
- Gated Local-Context Attention uses a sliding window to capture phonetic context, comparing w = 1, w = 5, and w = 11.
- w = 5 provides the best synchronization, while w = 1 causes jittery phoneme transitions and w = 11 produces over-smoothing.
- The selected w = 5 window preserves sufficient context without obscuring instantaneous audio cues needed for precise lip synchronization.
E More Qualitative Results
Additional qualitative results span R2V, RA2V, RP2V, and RAP2V settings. They depict high-fidelity identity preservation, natural motion, precise synchronization, and coherent generation under fully combined conditioning.
- Figures 11–13 present additional comparisons across diverse multimodal conditioning settings.
- The qualitative results cover R2V, RA2V, RP2V, and RAP2V conditioning scenarios.
- Across these scenarios, OmniShow exhibits high-fidelity reference identity preservation, natural motion dynamics, and precise audio-visual synchronization.
- RAP2V results illustrate OmniShow’s ability to orchestrate all input modalities simultaneously into coherent, temporally smooth, and expressive videos.
- Figures 11–13 are presented as additional qualitative evidence of OmniShow’s state-of-the-art performance across multimodal settings.