PaperlayerView exact revision on arXiv

Source-linked AI summary

UniVidX: A Unified Multimodal Framework for Versatile Video Generation via Diffusion Priors

Houyuan Chen, Hong Li, Xianghao Kong, Tianrui Zhu, Shaocong Xu, Weiqing Xiao, Yuwei Guo, Chongjie Ye, Lvmin Zhang, Hao Zhao, Anyi Rao

arXiv:2605.00658v1cs.CV

TL;DR

Existing methods often isolate multimodal graphics problems into fixed input-output mappings, limiting flexibility and cross-modal correlation modeling. UniVidX instead uses a unified conditional video-generation framework with SCM, DGL, and CMSA, and its two instantiations achieve competitive performance and robust in-the-wild generalization with fewer than 1,000 videos.

  • Problem

    Existing approaches train separate networks for fixed input-output mappings, limiting flexibility and often ignoring correlations shared across visual modalities.

  • Method

    UniVidX formulates multimodal graphics tasks as conditional generation using SCM for omni-directional mappings, DGL for modality adaptation, and CMSA for cross-modal consistency.

  • Results

    UniVidX’s UniVid-Intrinsic and UniVid-Alpha achieve competitive performance across diverse tasks and robust in-the-wild generalization with fewer than 1k training videos.

  • Takeaways & Limitations

    UniVidX provides a unified recipe for aligned multimodal video modeling across diverse visual modalities and conditional-generation paradigms.

  • Takeaways & Limitations

    Intrinsic-related and alpha-related capabilities remain separate because jointly annotated training data for both label types is unavailable.

Abstract

from arXiv · show

Recent progress has shown that video diffusion models (VDMs) can be repurposed for diverse multimodal graphics tasks. However, existing methods often train separate models for each problem setting, which fixes the input-output mapping and limits the modeling of correlations across modalities. We present UniVidX, a unified multimodal framework that leverages VDM priors for versatile video generation. UniVidX formulates pixel-aligned tasks as conditional generation in a shared multimodal space, adapts to modality-specific distributions while preserving the backbone's native priors, and promotes cross-modal consistency during synthesis. It is built on three key designs. Stochastic Condition Masking (SCM) randomly partitions modalities into clean conditions and noisy targets during training, enabling omni-directional conditional generation instead of fixed mappings. Decoupled Gated LoRA (DGL) introduces per-modality LoRAs that are activated when a modality serves as the generation target, preserving the strong priors of the VDM. Cross-Modal Self-Attention (CMSA) shares keys and values across modalities while keeping modality-specific queries, facilitating information exchange and inter-modal alignment. We instantiate UniVidX in two domains: UniVid-Intrinsic, for RGB videos and intrinsic maps including albedo, irradiance, and normal; and UniVid-Alpha, for blended RGB videos and their constituent RGBA layers. Experiments show that both models achieve performance competitive with state-of-the-art methods across distinct tasks and generalize robustly to in-the-wild scenarios, even when trained on fewer than 1,000 videos. Project page: https://houyuanchen111.github.io/UniVidX.github.io/

1 Introduction

UniVidX addresses the rigidity and cross-modal inconsistency of isolated task-specific video models with a unified framework for flexible multimodal generation. Its SCM, DGL, and CMSA designs support diverse conditional mappings while preserving VDM priors and enabling robust, data-efficient generalization.

  • Motivation: Existing approaches train separate networks for fixed input-output mappings, limiting flexibility and often ignoring correlations among visual modalities.These limitations can produce cross-modal inconsistencies in final modality stacks.
  • Motivation: UniVidX asks whether subsets of aligned modalities can flexibly serve as conditions or targets within one generative framework.The goal is flexible generation across visual modalities rather than isolated modality-specific prediction.
  • Framework: SCM randomly partitions modalities into clean conditions and noisy targets, training the T2V backbone for omni-directional generation from text, visual, or hybrid inputs.This design addresses the challenge of mastering diverse task categories within a shared conditional-generation framework.
  • Framework: DGL assigns independent modality-specific LoRAs that activate only for generation targets, adapting to distinct distributions while preventing parameter interference and preserving VDM priors.Condition modalities use deactivated LoRA modules.
  • Framework: CMSA shares keys and values across modalities while retaining modality-specific queries, promoting cross-modal consistency during joint generation.The design enables information exchange while preserving modality-specific query representations.
  • Instantiations and results: The framework instantiates UniVid-Intrinsic and UniVid-Alpha, supports three conditioning paradigms, and collectively covers 15 distinct tasks.Applications include relighting, retexturing, material editing, inpainting, and foreground or background replacement.
  • Instantiations and results: Both models achieve competitive performance across diverse tasks and robust in-the-wild generalization despite training on fewer than 1k videos.The reported results attribute this data efficiency to effective use of VDM priors.

2 Related Work

Prior work has applied video diffusion priors and data-driven methods to visual synthesis, intrinsic decomposition, and alpha-wise processing. UniVidX extends these directions by reformulating alpha-wise tasks as conditional video generation for versatile multimodal capabilities.

  • Visual multimodal generative models: Video diffusion models provide strong priors for simulating real-world dynamics and are increasingly leveraged for visual synthesis beyond RGB.These priors support downstream multimodal graphics tasks spanning perception and generation.
  • Intrinsic decomposition and generation: Intrinsic decomposition separates RGB images into appearance- and geometry-related channels, with methods progressing from physical heuristics to domain-specific data-driven networks.Recent work also uses generative priors to address decomposition’s ill-posed nature.
  • Intrinsic decomposition and generation: Text-to-intrinsic generation has emerged as a direct synthesis paradigm, but prior work remains confined to the image level.UniVid-Intrinsic introduces a video-based instantiation within the paper’s framework.
  • Alpha-wise perception and generation: Alpha-wise processing has advanced from optimization heuristics to data-driven video matting and semantic layer decomposition.These approaches target precise structure disentanglement across alpha-related representations.
  • Alpha-wise perception and generation: Reformulating alpha-wise tasks as conditional video generation unlocks versatile capabilities including video matting, inpainting, and text-to-RGBA generation.UniVid-Alpha serves as a representative instantiation of the unified framework.

3 Method

UniVidX unifies multimodal video generation by dynamically assigning modalities as conditions or targets, adapting each modality while preserving video diffusion priors, and exchanging cross-modal context. Two instantiations cover RGB/intrinsic and RGBA-layer domains across 15 tasks.

  • Stochastic Condition Masking: SCM dynamically partitions modality latents into clean conditions and noisy targets, enabling Text→X, X→X, and Text&X→X generation within one model.Condition latents remain clean, while target latents are corrupted during training and initialized from Gaussian noise at inference.
  • Decoupled Gated LoRA: DGL assigns independent low-rank adapters to modalities and activates each adapter only when its modality is a generation target.Condition modalities bypass adapters, preserving the pretrained VDM’s native encoding capability while avoiding parameter interference.
  • Cross-Modal Self-Attention: CMSA shares keys and values across modalities while retaining modality-specific queries, allowing each stream to use multimodal context.The design addresses the isolation of vanilla self-attention and promotes cross-modal consistency and alignment.
  • Model Instantiations: UniVidX instantiates UniVid-Intrinsic for RGB, albedo, irradiance, and normal maps and UniVid-Alpha for blended RGB, alpha, foreground, and background layers.Both models operate across three paradigms and collectively support 15 distinct tasks.
  • Modality Design: The intrinsic model excludes roughness, metallic, and depth targets because material annotations are scarce or unreliable and normals capture fine local geometry.The framework instead relies on pretrained VDM priors to infer material responses from context.

4 Experiment

The experiments compare UniVidX against baselines through qualitative and quantitative evaluations, ablations, multi-condition perception analyses, downstream task compositions, and limitation studies.

  • Experimental Analysis: The evaluation reports qualitative and quantitative comparisons for text-to-intrinsic, text-to-RGBA, inverse rendering, forward rendering, and albedo estimation.The experimental analysis first defines tasks and compares both models with other baselines.
  • Additional Studies: Ablation studies assess the effectiveness of UniVidX’s core architectural designs.The experiments also examine how multi-condition perception resolves ambiguity and how task composition supports downstream applications.

4.1 Experimental Setup

UniVidX evaluates representative text-to-intrinsic, inverse-rendering, and forward-rendering tasks for UniVid-Intrinsic, including dedicated albedo and normal estimation tests.

  • Text-to-Intrinsic: Text-to-intrinsic jointly generates RGB videos and corresponding intrinsic maps from text prompts.This task is evaluated under the Text→X paradigm.
  • Inverse Rendering: Inverse rendering estimates intrinsic maps from an input RGB video, with dedicated evaluations for albedo and normal estimation.These evaluations belong to the X→X paradigm.
  • Forward Rendering: Forward rendering synthesizes realistic RGB videos from input intrinsic information.This is also evaluated as an X→X task.

4.2 Comparative Evaluation

UniVid-Intrinsic and UniVid-Alpha achieve strong quantitative and qualitative results across generation, rendering, albedo, normal-estimation, and matting tasks. The evaluations also show temporal stability, real-world transfer, and data-efficient generalization.

  • Text-to-X Generation: Across text-to-intrinsic and text-to-RGBA generation, both models surpass representative baselines and obtain higher visual-quality, text-alignment, and modality-consistency ratings.Temporal Flickering scores are consistently close to 1.0, indicating temporally stable outputs.
  • Text-to-X Generation: UniVid-Alpha retains impressive generation quality with 484 videos versus LayerDiffuse’s 1M images and supports shared prompts without VAE fine-tuning.The model also generalizes from human-focused training data to out-of-distribution samples such as animals.
  • Inverse Rendering and Forward Rendering: UniVid-Intrinsic achieves state-of-the-art inverse and forward rendering, with the lowest surface-normal MAE of 11.09°.Its outputs most closely resemble ground truth, including albedo, irradiance, normals, and reconstructed RGB videos.
  • Albedo Estimation: UniVid-Intrinsic transfers from synthetic training to the real-world MAW benchmark, achieving intensity error 0.44 and chromaticity error 3.60.The intensity error is best reported in the benchmark, while chromaticity performance is competitive.
  • Normal Estimation: UniVid-Intrinsic produces temporally coherent normal maps while preserving high-frequency geometric details.Compared with specialized estimators and intrinsic-related baselines, it avoids texture loss and temporal jitter.
  • Video Matting: UniVid-Alpha achieves state-of-the-art video matting, including the lowest MAD of 4.24, while preserving intricate hair details without auxiliary inputs.It outperforms both mask-guided and auxiliary-free competitors and avoids artifacts and background leakage.

4.3 Ablation Study

The ablations show that channel concatenation, parameter sharing, ungated LoRAs, and vanilla attention undermine multimodal generation, while UniVidX’s decoupling, gating, and CMSA designs preserve quality and alignment.

  • Channel Concatenation: Channel concatenation produces corrupted outputs in text-to-intrinsic and text-to-RGBA generation by disrupting the diffusion priors.The UniVid-Intrinsic and UniVid-Alpha models instead produce high-fidelity results.
  • Gating: The full gated model produces robust normal estimates, while disabling gating causes inaccurate background prediction and texture loss.Gating activates modality-specific LoRAs for generation targets and deactivates them for clean conditioning inputs.
  • Attention Analysis: CMSA yields modality-disentangled attention, with BL attending globally, FG attending the subject, and BG attending the background.The w/o Dec. variant produces noisy maps and severe feature leakage across FG and BG.
  • Decoupling: Decoupled parameter spaces enable robust text-to-RGBA generation with both distinct and shared prompts, whereas shared parameters cause foreground-background confusion.LayerDiffuse also fails with a shared prompt under its shared-parameter design.
  • Cross-Modal Self-Attention: CMSA preserves structural consistency across RGB, albedo, irradiance, and normal, whereas vanilla attention produces inter-modal misalignment.The comparison supports sharing keys and values across modalities while retaining modality-specific queries.

4.4 The Value of Multi-Condition Perception Paths

Multi-condition perception paths reduce ambiguity in inverse rendering by adding auxiliary modalities as semantic and structural constraints. In the demonstrated case, RGB plus albedo recovers geometry that RGB alone misses.

  • Motivation: Auxiliary modalities provide semantic cues and structural constraints that guide more physically accurate inverse-rendering predictions.This is especially useful when RGB inputs contain ambiguous regions.
  • Example: RGB alone misinterprets a blurry planet as empty sky, while RGB plus albedo enables accurate recovery of its surface normals.The albedo modality signals the underlying structure that is not reliably inferred from the RGB input.

4.5 Applications

UniVidX composes flexible input-output paths into downstream graphics applications. The demonstrations cover relighting, retexturing, material editing, inpainting, and foreground or background replacement.

  • Video Relighting: Video relighting conditions on albedo and normal maps to preserve surface color and geometry while changing illumination.The model generates a relighted RGB video and corresponding irradiance maps from a target text prompt.
  • Text-driven Video Retexturing: Text-driven video retexturing conditions on irradiance and normal maps to preserve lighting and geometry while modifying the surface.The output includes a new RGB video and albedo map.
  • Material Editing: Material editing modifies albedo and normal maps before forward rendering with the original irradiance to generate updated materials.Albedo changes colors, while normal edits modify texture details.
  • Video Inpainting: Video inpainting conditions on extracted alpha mattes and backgrounds to generate new foreground content while preserving the original context.The output includes the new foreground and corresponding blended RGB video.
  • Layer Replacement: Background replacement generates alpha and foreground components first, then synthesizes a replacement background and final blended RGB video.Foreground replacement instead conditions on an extracted background to jointly generate a new subject, alpha matte, and blended RGB video.

4.6 Limitations and Failure Analysis

UniVidX has scope and data limitations that constrain its current deployment. The model also shows suboptimal behavior on physical corner cases when training data lacks relevant examples.

  • Data limitations: The intrinsic and alpha capabilities remain split across two models because jointly annotated intrinsic and alpha data are unavailable.The authors believe a single model could unify them if such annotations become available.
  • Computational constraints: The 14B Wan2.1-T2V backbone limits UniVidX to four modalities, 21 frames, and 480p resolution.These constraints arise despite using parameter-efficient LoRA tuning because of the backbone’s memory footprint.
  • Data bias and corner cases: Reliance on pretrained priors makes UniVidX susceptible to training-distribution bias and suboptimal performance on specific physical corner cases.The authors characterize these limitations as data-dependent and suggest targeted samples as a remedy.
  • Data bias and corner cases: UniVid-Intrinsic produces spatially inconsistent normal estimates for transparent glass surfaces.The issue is documented in the glass-surface example shown in Fig. 14.

5 Conclusion

UniVidX presents a unified framework for versatile multimodal video generation by combining SCM, DGL, and CMSA to exploit video diffusion priors. Across UniVid-Intrinsic and UniVid-Alpha, it reports strong performance, temporal stability, data efficiency, and in-the-wild generalization with fewer than 1,000 videos.

  • Conclusion: UniVidX combines Stochastic Condition Masking, Decoupled Gated LoRA, and Cross-Modal Self-Attention for unified multimodal video generation.SCM supports flexible conditioning, DGL adapts modality distributions, and CMSA aligns modalities.
  • Conclusion: UniVid-Intrinsic and UniVid-Alpha demonstrate exceptional performance, superior temporal stability, and robust in-the-wild generalization.These outcomes are reported with remarkable data efficiency.
  • Conclusion: Both models achieve these results while being trained on fewer than 1,000 videos.
Loading 2605.00658v1…