PaperlayerView exact revision on arXiv

Source-linked AI summary

V-RGBX: Video Editing with Accurate Controls over Intrinsic Properties

Ye Fang, Tong Wu, Valentin Deschaintre, Duygu Ceylan, Iliyan Georgiev, Chun-Hao Paul Huang, Yiwei Hu, Xuelin Chen, Tuanfeng Yang Wang

arXiv:2512.11799v1cs.CV

TL;DR

Video generation systems lack an explicit intrinsic representation space for controlling properties such as lighting and material and propagating edits consistently across frames. V-RGBX combines intrinsic decomposition, synthesis, and keyframe editing with interleaved conditioning. The paper reports improved fidelity and controllability over appearance-only baselines and demonstrates object appearance editing and scene-level relighting.

  • Problem

    Video generation systems lack explicit intrinsic representations for disentangling properties such as lighting and material and consistently propagating intrinsic edits across frames.

    Source: §1 · p. 2

  • Method

    V-RGBX unifies video inverse rendering into intrinsic channels, photorealistic synthesis from them, and keyframe editing using interleaved conditioning.

    Source: §1 · p. 2

  • Results

    V-RGBX reports improved fidelity and controllability over appearance-only baselines and demonstrates object appearance editing and scene-level relighting.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework supports keyframe-level, multi-modality intrinsic edits and their propagation in temporally coherent video editing.

    Source: §1 · p. 2

    The model is trained only on indoor synthetic datasets and may struggle to generalize to out-of-distribution scenarios such as outdoor scenes.

    Source: §5 · p. 8

Abstract

from arXiv · show

Large-scale video generation models have shown remarkable potential in modeling photorealistic appearance and lighting interactions in real-world scenes. However, a closed-loop framework that jointly understands intrinsic scene properties (e.g., albedo, normal, material, and irradiance), leverages them for video synthesis, and supports editable intrinsic representations remains unexplored. We present V-RGBX, the first end-to-end framework for intrinsic-aware video editing. V-RGBX unifies three key capabilities: (1) video inverse rendering into intrinsic channels, (2) photorealistic video synthesis from these intrinsic representations, and (3) keyframe-based video editing conditioned on intrinsic channels. At the core of V-RGBX is an interleaved conditioning mechanism that enables intuitive, physically grounded video editing through user-selected keyframes, supporting flexible manipulation of any intrinsic modality. Extensive qualitative and quantitative results show that V-RGBX produces temporally consistent, photorealistic videos while propagating keyframe edits across sequences in a physically plausible manner. We demonstrate its effectiveness in diverse applications, including object appearance editing and scene-level relighting, surpassing the performance of prior methods.

1. Introduction

Video generation systems offer controllable appearance editing, but explicit control of intrinsic properties and their consistent propagation across frames remain largely unexplored. V-RGBX combines intrinsic decomposition, synthesis, and keyframe editing in a framework for multi-modality control.

1. Introduction

  • Direct intrinsic-property control remains largely unexplored, despite its relevance to realism, consistency, relighting, material editing, and stylized video generation.The stated goal includes changing an object's shadeless texture while preserving lighting, or changing environmental lighting without altering surface material.Source: §1 · p. 2
  • Prior approaches often condition in pixel space or globally from a prompt or single reference, limiting consistent intrinsic preservation and localized edits across modalities and time.The cited examples include GenProp, VACE, DaS, and AnyV2V.Source: §1 · p. 2
  • V-RGBX supports physically grounded relighting, albedo manipulation, and geometry-aware object insertion with keyframe-level control.The paper reports significant improvements in fidelity and controllability over appearance-only baselines.Source: §1 · p. 2
  • V-RGBX unifies video inverse rendering into intrinsic channels, photorealistic synthesis from those channels, and keyframe-based editing conditioned on them.The framework extracts modalities such as albedo, irradiance, normal, and depth for selective edits on sparse keyframes.Source: §1 · p. 2
  • The framework uses interleaved conditioning to support flexible intrinsic conditioning during synthesis and propagation of keyframe edits.Untouched modalities are randomly provided and preserved during generation.Source: §1 · p. 2

2. Related work

Prior video diffusion and intrinsic-editing approaches provide generation or editing capabilities, but the cited work leaves intrinsic edits difficult to propagate through video. V-RGBX builds on video diffusion with interleaved intrinsic controls across time.

2.1. Video Diffusion Models

  • Recent video diffusion models have shifted from U-Net architectures with temporal modules to diffusion transformers, which the passage associates with stronger temporal coherence and visual quality.V-RGBX builds on the WAN model [46].Source: §2.1 · pp. 2–3

2.2. Intrinsic-Aware Diffusion Models

  • Image-level intrinsic editing methods [13] do not address video propagation; DiffusionRenderer lacks pixelwise edit propagation, while X2Video [22] cannot directly propagate framewise intrinsic edits without a complete edited sequence.V-RGBX instead uses single-frame interleaved conditioning and type-aware embeddings to propagate intrinsic controls across time.Source: §2.2 · p. 3

2.3. Controlled Video Generation and Editing

  • Existing video control and editing work covers camera, structural, action, scene, and general editing signals, while intrinsic-based tasks remain largely unexplored.V-RGBX recovers multiple intrinsic properties, including illumination, and recomposes them through intrinsic-conditioned generation to propagate per-frame edits across time.Source: §2.3 · p. 3

3. Method

V-RGBX decomposes video into explicit intrinsic channels, lets users edit selected keyframes, and synthesizes a video that propagates those edits while preserving untouched properties. Its conditioning pipeline interleaves intrinsic inputs and combines them with keyframe references and temporal-aware embeddings.

3.1. Overview

  • V-RGBX estimates intrinsic channels from an input video, applies keyframe edits, and conditions forward rendering on a streamed intrinsic sequence and the edited keyframes.The estimated channels include albedo, normal, material, and irradiance.Source: §3.1 · p. 3

3.2. Inverse Rendering (RGB→X)

  • The inverse renderer predicts intrinsic channels with a DiT backbone and decodes the denoised latent into a three-channel intrinsic image for the target modality.The denoising process uses a frozen Wan-VAE encoder, while the target modality is supplied as a text prompt.Source: §3.2 · pp. 3–4

3.3. Intrinsic-aware Conditioning

  • The conditioning sampler interleaves edited modalities at edited keyframes with non-conflicted modalities from other frames, avoiding use of neighboring intrinsic channels that could conflict with edits.The paper describes this temporal multiplexing design as lightweight and extensible for diverse attribute combinations and incomplete inputs.Source: §3.3 · p. 4

3.4. Forward Rendering with Intrinsic-aware Conditioning

  • Forward rendering combines the interleaved intrinsic sequence with edited keyframe references; Temporal-aware Intrinsic Embedding packs per-frame modality embeddings into latent chunks while preserving temporal order and modality identity.The model builds on WAN, and its keyframe reference supplies scene information not explicitly represented in the intrinsic channels.Source: §3.4 · p. 5

4. Experiments

V-RGBX is evaluated on synthetic and real-world data across inverse rendering, intrinsic-to-video synthesis, keyframe conditioning, and editing applications. The reported comparisons favor it in cycle consistency, intrinsic estimation, and keyframe-edit propagation.

4.1. Experiment settings

  • Training uses 171K paired frames from 127 synthetic interior scenes; evaluation uses 85 unseen Evermotion and 85 RealEstate10K videos, typically with the first frame as keyframe.PSNR, SSIM, and LPIPS assess rendering, while FVD and VBench smoothness assess video quality and temporal coherence.Source: §4.1 · pp. 5–6

4.2. Inverse rendering (RGB→X)

  • V-RGBX outperforms other baselines in albedo, normal, and irradiance estimation on the synthetic inverse-rendering evaluation.The paper also reports better temporal coherence than other baselines in qualitative intrinsic-channel comparisons.Source: §4.2 · p. 6, Table 1 · p. 8

4.4. Keyframe based video conditioning

  • With an RGB reference, V-RGBX remains quantitatively comparable when albedo or irradiance is omitted or supplied only in the first frame.The first-frame-only setting improves results, indicating propagation of the provided guidance across the sequence.Source: §4.4 · p. 8, Table 4 · p. 8

4.5. Applications

  • In appearance editing and relighting, V-RGBX propagates keyframe edits while preserving untouched intrinsic content; qualitative comparisons show competing methods drifting or disrupting other channels.The applications include object appearance editing and scene relighting.Source: Figure 3 · p. 6, Figure 4 · p. 7, §4.5 · p. 8

4.3. Intrinsic-aware video generation (X→RGB)

  • For X→RGB, V-RGBX surpasses prior approaches in forward-rendering accuracy, video quality, and temporal smoothness; reference conditioning further improves results.Reference images help capture scene style and compensate for information absent from intrinsic channels.Source: §4.3 · p. 7, Table 2 · p. 8
  • V-RGBX achieves the best overall cycle-consistency performance among baselines on synthetic and real datasets, with stronger temporal coherence and frame-wise accuracy.The comparison covers inverse rendering followed by forward rendering.Source: Table 3 · p. 8, §4.3 · p. 8

5. Discussion

The discussion identifies limits in training-data scope, per-frame modality conditioning, and backbone scalability, and points to long-range generation as a future extension. The appendix notes additional details and qualitative results.

5. Discussion

  • The authors identify three constraints: indoor-synthetic training may hinder out-of-distribution generalization, one modality per frame limits complex edits, and the pretrained backbone constrains video length and real-time performance.Source: §5 · p. 8
  • Long-range generation is proposed as a future extension for more persistent keyframe edits across time.Source: §5 · p. 8

A. Appendix Overview

B. Workflow & Implementation Details

V-RGBX decomposes videos and edited keyframes into intrinsic channels, interleaves original and edited cues, then renders the edited sequence. Its cycle workflow evaluates reconstruction, while supplemental ablations show that Intrinsic Type Embedding improves temporal stability and appearance consistency.

B.1. Video editing workflow explanation

  • The editing workflow decomposes input videos and edited keyframes into albedo, irradiance, normal, and material channels, then combines edited cues with preserved channels in an interleaved sequence for forward rendering.The forward renderer also uses edited keyframes as appearance references to synthesize the final RGB sequence.Source: §B.1 · p. 9, Figure S1 · p. 10

B.2. RGB→X→RGB cycle workflow

  • The RGB→X→RGB cycle uses the first predicted intrinsic frame as a keyframe and compares reconstructed videos with inputs to assess fidelity, structural detail, and temporal continuity.The evaluation also examines the stability of intrinsic-based edits propagated across frames.Source: §B.2 · p. 9, Figure S2 · p. 10

B.3. Inference details

  • At inference, classifier-free guidance is applied to the reference branch while the interleaved intrinsic sequence remains a shared condition, modulating reference-driven appearance while retaining intrinsic priors.The guidance scale is set to 1.5.Source: §B.3 · p. 9

C. Additional Experiments

  • The supplemental ablations remove either the Intrinsic Type Embedding or reference condition and retrain each variant from scratch with the same iterations and hyperparameters as V-RGBX.The ITE comparison attributes temporal and modality inconsistencies to confusion between per-frame intrinsic channels when type disambiguation is absent.Source: C. Additional Experiments · p. 9, §C.1 · p. 9

C.1. The Effectiveness of ITE Module

  • Adding Intrinsic Type Embedding improves all metrics and yields more temporally stable, appearance-consistent videos by disentangling intrinsic channels over time.Without it, per-frame interleaving can confuse modality identities and produce flickering and color inconsistencies.Source: §C.1 · p. 9

C.2. The Effectiveness of Reference Condition

The reference-condition ablation finds further gains over ITE alone, with clearer reflections and colors closer to ground truth. It evaluates reference supervision during training, distinct from providing reference images at inference.

C.2. The Effectiveness of Reference Condition

  • Adding the reference condition further improves performance over ITE alone, with clearer, more coherent reflections and object colors and tones closer to ground truth.The comparisons attribute complementary cues to the reference condition, which help correct biases in X→RGB mapping and improve reconstruction fidelity.Source: §C.2 · p. 10, Figure S3 · p. 10
  • The ablation evaluates whether the model is trained with reference supervision, unlike the main-paper evaluation of whether reference images are supplied at inference.The visual comparison includes intrinsic maps and reconstructed RGB frames for the base model, ITE, ITE with reference condition, and ground truth.Source: §C.2 · p. 10, Figure S3 · p. 10
  • The reference condition improves reflection coherence and brings object colors and tones closer to ground truth in the reported visual comparisons.These changes indicate complementary cues for correcting biases in the X→RGB mapping.Source: §C.2 · p. 10, Figure S3 · p. 10

D.1. Additional RGB→X Results

Additional inverse-rendering results show V-RGBX predictions across synthetic and real scenes and compare them with RGBX and DiffusionRenderer. The reported comparisons highlight more stable albedo and normal reconstructions and reliable decompositions under challenging real-world and outdoor lighting.

D.1. Additional RGB→X Results

  • The supplemental section provides qualitative inverse-rendering examples and representative baseline comparisons alongside the quantitative results reported in the main paper.The comparisons are presented in Fig. S4, Fig. S5, and Fig. S6.Source: §D.1 · pp. 10–11
  • V-RGBX produces more stable albedo and normal reconstructions than RGBX, while DiffusionRenderer has reported collapsed normal maps and inaccurate color estimates.The comparisons are shown alongside representative predictions of albedo, normal, material, and irradiance.Source: §D.1 · pp. 10–11
  • V-RGBX produces reliable intrinsic decompositions under challenging real-world and outdoor lighting conditions.Additional examples include both synthetic and real-scene input videos with predicted albedo, normal, material, and irradiance channels.Source: §D.1 · pp. 10–11

D.2. Additional X→RGB Results

Additional X→RGB examples show V-RGBX generating sequences from intrinsic representations with stable shading, reflections, and temporal coherence across complex lighting and geometric structures.

D.2. Additional X→RGB Results

  • V-RGBX handles complex lighting and geometric structures more reliably, producing RGB sequences with more stable shading, reflections, and temporal coherence.The supplemental examples compare X→RGB outputs with baseline methods and are presented as further evidence of robustness when generating videos from intrinsic representations.Source: §D.2 · p. 11

D.3. Additional RGB→X→RGB Results

The RGB→X→RGB cycle evaluation tests whether intrinsic representations retain enough information for reconstruction and edit propagation. V-RGBX shows more stable temporal behavior and better preservation of scene appearance across the full cycle.

  • The RGB→X→RGB cycle evaluates whether intrinsic representations preserve enough information for accurate reconstruction and reliable edit propagation.Source: §D.3 · p. 11
  • V-RGBX achieves more stable temporal behavior across the full cycle than the baseline methods.Source: §D.3 · p. 11
  • V-RGBX better preserves scene appearance across the full cycle than the baseline methods.Source: §D.3 · p. 11

D.4. Keyframe Editing Results

The keyframe-editing results show how modified intrinsic channels condition synthesis, while additional demonstrations span challenging real-world scenes. Across the examples, V-RGBX produces temporally consistent edits and stronger visual results in decomposition, synthesis, and cycle reconstruction.

D.4. Keyframe Editing Results

  • Modified keyframes and extracted albedo, normal, material, or irradiance channels jointly condition synthesis of temporally consistent, intrinsically coherent outputs.The visualizations cover solid-color, texture, material, normal, light-color, and shadow edits.Source: §D.4 · p. 11, Figure S11 · p. 17

D.5. Real-world Challenging Cases

  • Additional demonstrations cover indoor scenes, self-captured videos, general objects, and complex lighting across diverse real-world conditions.Source: §D.5 · p. 11
  • V-RGBX decomposes synthetic indoor videos into albedo, normal, material, and irradiance channels with spatially coherent and temporally stable results.Source: Figure S4 · p. 12
  • On real-world RealEstate10K videos, V-RGBX produces coherent, temporally stable intrinsic decompositions and shows reasonable generalization to outdoor scenes.Source: Figure S5 · p. 13
  • Compared with RGBX and DiffusionRenderer, V-RGBX yields higher-fidelity decompositions, more accurate albedo, and more consistent normals; against RGBX it also gives clearer, more plausible irradiance maps.Source: Figure S6 · p. 14, Figure S7 · p. 15
  • Qualitative X→RGB comparisons show better scene appearance and temporal consistency, while cycle reconstructions more closely match ground truth on synthetic and real-world videos.Source: Figure S8 · p. 15, Figure S9 · p. 16, Figure S10 · p. 16

Related papers

Loading 2512.11799v1…