PaperlayerView exact revision on arXiv

Source-linked AI summary

JointLight: Human-Centric Video Relighting with Asynchronous RGB-Irradiance Joint Modeling

Henglei Lv, Bailin Deng, Xiaoqiang Liu, Yichen Wang, Haoxian Zhang, Pengfei Wan, Lin Gao

arXiv:2610.29780v1cs.CV

TL;DR

Human-centric relighting remains challenging because precise control and diverse, precisely labeled video data are limited, while implicit light-transport modeling complicates control. JointLight jointly denoises RGB and irradiance with asynchronous timesteps and domain matching, using a paired synthetic video dataset. The authors report state-of-the-art relighting quality and fidelity, while noting limits in specular transport and fine-detail preservation.

  • Problem

    Human-centric relighting lacks precise illumination control and diverse, high-quality, precisely labeled video data, while implicit light-transport modeling complicates controllability.

    Source: §1 · p. 2

  • Method

    JointLight jointly denoises RGB video and irradiance maps with separate timesteps, using noise-corrupted irradiance cues and domain matching to address cue artifacts and domain gap.

    Source: §1 · p. 2, §5.2 · p. 5

  • Results

    JointLight achieves state-of-the-art relighting quality and fidelity, and outperforms state-of-the-art realistic video relighting approaches in relighting fidelity and video quality.

    Source: §1 · p. 2, §8 · p. 10

  • Takeaways & Limitations

    The paper contributes a human-centric multi-illumination synthetic video dataset paired with irradiance maps.

    Source: §1 · p. 2

    Irradiance provides diffuse transport only, so the method cannot explicitly or physically control full specular transport; suppressing cue errors can also over-smooth fine facial details.

    Source: §8 · p. 10

Abstract

from arXiv · show

Understanding and modeling light transport in human-centric scenes is a longstanding challenge. While recent progress in diffusion models has boosted research in relighting, precise illumination control still lags, primarily due to the scarcity of diverse, high-quality, precisely labeled video relighting datasets. Existing end-to-end relighting approaches model light transport implicitly through neural networks, which substantially complicates controllability. In contrast, some works resort to path-traced irradiance cues as an explicit blueprint for light transport. However, such constructed irradiance cues are prone to artifacts and hinder generative quality when directly used as control signals. With the key observation that noise corruption removes high-frequency artifacts and preserves low-frequency light transport, we introduce an asynchronous joint-modeling paradigm that facilitates fine-grained illumination control. During two-stage training, we force the model to jointly denoise RGB and ground-truth irradiance frames with separate timesteps scheduling. A domain matching strategy is applied to the second stage to bridge the domain gap between irradiance cues and ground-truth irradiance maps. At inference time, we introduce an asynchronous hijacked generation pipeline that facilitates flexible illumination control. For training data, we develop a multi-illumination high-quality synthetic video dataset with paired irradiance maps using a powerful game engine. Trained exclusively on this dataset, our model exhibits robust generalization to real- world scenarios and outperforms prior approaches in both lighting fidelity and generative quality. The code and dataset will be publicly released.

1 Introduction

Human-centric relighting in diverse indoor and outdoor scenes remains challenging because scene properties are coupled and practical geometry and material capture is difficult. JointLight combines asynchronous RGB–irradiance modeling with a paired multi-illumination dataset, and reports state-of-the-art relighting fidelity and video quality.

1 Introduction

  • Human-centric relighting across diverse indoor and outdoor scenes remains difficult because real-world scene geometry and materials are impractical to capture and input appearance couples geometry, material, and lighting.Existing approaches described here either lack explicit physical light-transport modeling or use synthetic data limited to simple object-centric scenes.Source: §1 · p. 2
  • JointLight jointly denoises RGB video and irradiance maps using separate timesteps, with noise corruption intended to suppress cue artifacts while preserving low-frequency light transport.At inference, the irradiance cue is rendered using estimated geometry and target lighting, then noised and jointly denoised with RGB.Source: §1 · p. 2
  • The paired synthetic dataset contains 68K videos from 6.8K dynamic scenes, rendered under 10 lighting conditions sampled from 821 HDR environment maps.Twenty percent of the lighting conditions use rotating environment maps.Source: §1 · p. 2
  • JointLight outperforms state-of-the-art realistic video relighting approaches in both relighting fidelity and video quality.The paper also reports that ablations verify the effectiveness of its key designs.Source: §1 · p. 2

2 Related Work

Prior human-centric relighting methods vary in scene scope and control interface, while imperfect irradiance cues remain a challenge for joint multimodal diffusion. Existing multi-illumination datasets are described as mainly static or simple-scene, motivating human-centric video synthesis with lighting annotation.

Human-centric scene relighting

  • Many human-centric video methods use degradation pairs and text prompts or environment maps for coarse control, while synthetic relighting systems do not explicitly guide light transport in complex scenes with occlusions.Other systems target static relighting, captured personalized avatars, reference-frame propagation, or text-based atmosphere editing rather than environment-map-controlled general monocular videos.Source: §2 · pp. 2–3

Joint cross-modal modeling

  • Joint diffusion methods commonly assume clean auxiliary modalities, whereas this work addresses imperfect irradiance cues with geometry artifacts through domain matching and hijacked inference.The cited prior asynchronous-timestep methods do not address imperfect auxiliary modalities at inference time.Source: §2 · p. 3

Multi-illumination dataset

  • Existing multi-illumination datasets mainly cover static images in restricted scenarios, and prior synthetic relighting work uses simple rendered scenes.Unreal Engine 5 is presented as enabling high-quality human-centric multi-illumination video synthesis with accurate lighting annotation.Source: §2 · p. 3

3 Preliminary

The preliminary material frames diffusion generation as an ordinary differential equation and describes a rectified-flow path from noise to data. Its conditional flow-matching objective trains a velocity field that is integrated from noise toward a clean sample.

3 Preliminary

  • The generative process maps noise samples toward data samples through an ordinary differential equation whose velocity is parameterized by the diffusion transformer.At inference, integrating the learned ODE from noise toward data produces a clean sample.Source: §3 · p. 4
  • Rectified Flow defines a straight interpolation between a clean latent and Gaussian-noise latent over a continuous noise level from 0 to 1.The endpoints are clean data at t = 0 and noise at t = 1.Source: §3 · p. 4
  • The Conditional Flow Matching objective uses mean squared error to regress the target velocity along the interpolation path.The target velocity is ε − z0; the predicted velocity is vΘ.Source: §3 · p. 4

4 Training Dataset

The training dataset is an automated Unreal Engine 5 collection of multi-illumination human-centric videos paired with irradiance maps. It covers indoor and outdoor environments and uses photorealistic rendered content, which the authors associate with real-world generalization.

4 Training Dataset

  • The dataset contains 68K videos at 832×480 resolution and 94 frames each, paired with irradiance maps and rendered under 10 illuminations per scene.It draws on 30 environments, 60 characters, 600 body animations, and 821 HDR environment maps; 20% of illuminations use rotary lighting.Source: §4 · p. 4
  • The dataset comprises 12 indoor and 18 outdoor environments, and the authors report that their model generalizes well to real-world videos despite training purely on synthetic data.They attribute this generalization to UE5 photorealism, which they say reduces the sim-to-real gap compared with prior Blender-based synthetic datasets.Source: §4 · pp. 4–5
  • The dataset focuses on human-centric indoor and outdoor scenarios and is designed to support precise lighting control over human subjects.The assets include characters, body animations, facial expressions, and HDR environment maps.Source: §4 · p. 4

5 Method

JointLight constructs per-frame irradiance cues and jointly denoises RGB video and irradiance with separate timesteps. Domain matching and hijacked inference let it use imperfect cues while balancing lighting fidelity and generation quality.

5.1 Irradiance cue

  • Per-frame meshes rendered under target lighting produce irradiance cues, but geometry errors, removed boundary triangles, and temporal inconsistency leave a gap from ground-truth irradiance.The method uses monocular point clouds, removes triangles across depth discontinuities, assigns a uniform white diffuse BRDF, and renders with Mitsuba 3.Source: §5.1 · p. 5

5.2 Joint modeling with separate timestep scheduling

  • Gaussian-noise corruption suppresses high-frequency cue artifacts while preserving a low-frequency light-transport prior for joint denoising.Joint denoising learns how much of the corrupted cue to retain; hole filling and blurring may fail around undefined geometry or erase useful structure.Source: §5.2 · p. 5
  • The model jointly denoises relit RGB and irradiance using separate timesteps, concatenating their latents with the input-video latent for shared spatiotemporal attention.This trains the model to handle mismatched noise levels and supports inference with noisy irradiance conditions.Source: §5.2 · p. 5
  • A learnable modality embedding distinguishes aligned RGB, irradiance, and input-video features while sharing their spatial-temporal positions.The embedding is added on top of timestep embeddings to help avoid modality mixture.Source: §5.2 · p. 5
  • Domain matching randomly substitutes noisy irradiance cues during training to address the inference gap between imperfect cues and ground-truth irradiance.The approach is designed for imperfect cues with domain discrepancies, unlike methods that assume reliable modalities or directly feed estimated cues to the generator.Source: §5.2 · p. 5

5.3 Hijacked asynchronous inference strategy

  • Hijacked inference holds noisy irradiance latents fixed before the hijack timestep, then denoises irradiance and RGB together to use the cue prior while removing artifacts.The hijack timestep controls the noise and the tradeoff between generation quality and lighting fidelity; 0.6T works for most cases, with larger values preferred for noisier or less aligned cues.Source: §5.3 · p. 5

6 Experiments

Across synthetic benchmarks, JointLight leads on all SynthDynamic metrics and on two of three SynthStatic metrics; users also prefer its in-the-wild results. It generalizes to HumanOLAT and dynamic illumination, with neural generation the largest runtime component.

6 Experiments

  • Experiments use a synthetic test set of 100 cases, public video-relighting baselines, and HumanOLAT static images for cross-dataset evaluation.The test set comprises 50 static-lighting and 50 rotary-lighting cases; prompt-based methods receive captions, while environment-map methods use the identical target HDR map.Source: §6 · p. 6

6.1 Evaluation

  • Qualitative comparisons report that JointLight controls illumination using path-traced irradiance guidance, while baselines show prompt ambiguity, texture or identity issues, or incorrect light-source interpretation.The comparison covers vertical SynthStatic and HumanVid videos and horizontal images; JointLight also generalizes to a cat example.Source: Figure 5 · p. 6, Figure 6 · p. 7, §6.1 · p. 8
  • JointLight achieves the best PSNR and LPIPS on SynthStatic and all three metrics on SynthDynamic; UniRelight has the best SynthStatic SSIM.The evaluation covers 100 cases split evenly between SynthStatic and SynthDynamic; the larger advantage under rotating illumination indicates improved controllability as target lighting changes.Source: Table 1 · p. 7, §6.1 · p. 8
  • On HumanOLAT, JointLight achieves the best performance among evaluated methods despite being trained without HumanOLAT data.The evaluation uses OLAT captures relit under 100 HDR environment maps, foreground masks, and no per-channel rescaling.Source: §6.1 · p. 8, Table 2 · p. 8
  • In a 13-person study of 15 in-the-wild cases, participants consistently prefer JointLight on visual quality and lighting fidelity.Participants rated both dimensions from 1 to 5.Source: Table 1 · p. 7, §6.1 · p. 8
  • For dynamic scenes and illumination, JointLight follows the sun direction and produces physically plausible lighting effects.Results are sampled every 10 frames.Source: §6.1 · p. 8
  • An 81-frame 832 × 480 video takes 223 s end to end on one NVIDIA A800 GPU, with neural generation accounting for 141.9 s.The pipeline includes depth estimation, mesh conversion, and Mitsuba rendering; its total time is lower than the reported times for UniRelight, DiffusionRenderer, and Light-A-Video under the same setting.Source: §6.1 · p. 8

7 Ablation study

The ablations show that asynchronous joint modeling and domain matching are both needed for physically plausible, pixel-aligned relighting. Hijacked generation and asynchronous scheduling progressively improve fidelity and visual quality, while time-embedding modality cues perform best among the tested embedding strategies.

7 Ablation study

  • Hijacked conditional generation and asynchronous timestep scheduling progressively improve lighting fidelity and visual quality, with domain matching providing the final boost.Domain matching aligns the cue distribution with inference-time inputs.Source: §7 · p. 9
  • Time-embedding modality cues perform best among the tested embedding strategies, likely by globally modulating denoising and reducing ambiguity between RGB appearance and irradiance guidance.Table 4 covers the modality-embedding ablation.Source: §7 · p. 9, Table 4 · p. 9
  • Without asynchronous joint modeling, irradiance cues fail to guide light transport; joint modeling alone leaves a domain gap that domain matching must bridge.The full recipe produces pixel-aligned RGB and irradiance images, while omitting domain matching yields black edges and blurred irradiance details.Source: §7 · pp. 8–9, Figure 10 · p. 10

8 Discussion

JointLight relies on monocular geometry only to render irradiance cues, so framewise errors do not accumulate as global reconstruction drift. Its control and detail remain limited by cue accuracy, diffuse-only irradiance, and occasional smoothing of fine facial features.

8 Discussion

  • Framewise geometry errors do not accumulate as global reconstruction drift, but severely incorrect geometry can omit light transport that noise injection cannot recover.Aggressive subject or camera motion can also produce blur and occlusion errors.Source: §8 · p. 10
  • Irradiance provides diffuse transport only, so polished metal, mirrors, and glass remain difficult to control physically; moderate skin and hair highlights are not guaranteed to follow target illumination.The authors suggest jointly reasoning about geometry, material, diffuse and specular transport, and extending control beyond environment maps to local lights.Source: §8 · p. 10
  • Suppressing high-frequency cue errors can occasionally over-smooth fine facial details around the eyes and mouth.The authors propose detail-preserving or preference-based post-training as a possible direction.Source: §8 · p. 10

Related papers

Loading 2610.29780v1…