PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Latent Proxies for Controllable Single-Image Relighting

Haoze Zheng, Zihao Wang, Xianfeng Wu, Yajing Bai, Yexin Liu, Yun Li, Xiaogang Xu, Harry Yang

arXiv:2603.15555v1cs.CV

TL;DR

Single-image relighting lacks observable geometry and materials, while existing approaches either require dense intrinsic supervision or provide weak physical control. LightCtrl uses few-shot latent physical proxies, lighting-aware spatial guidance, DPO refinement, and the ScaLight dataset, achieving improved controllable relighting across object and scene benchmarks. Its strongest evidence remains in synthetic object-centric settings, with less robust behavior in cluttered real scenes.

  • Problem

    Single-image relighting is under-constrained, and prior intrinsic or latent approaches respectively require dense supervision or lack sufficient physical grounding for fine-grained control.

  • Method

    LightCtrl combines a few-shot latent proxy, lighting-aware mask, DPO-refined PBR encoder, diffusion backbone, and ScaLight dataset for physically guided relighting.

  • Results

    LightCtrl improves relighting performance over intrinsic-based and diffusion baselines on object- and scene-level benchmarks, with state-of-the-art RMSE and PSNR and the highest user-study preference rate.

  • Takeaways & Limitations

    Sparse physical cues can provide fine-grained and physically consistent illumination control without full intrinsic reconstruction or dense supervision.

  • Takeaways & Limitations

    Evidence is strongest in synthetic object-centric settings; cluttered real scenes show less robust control and mixed quality gains under long-range transport and strong specular interactions.

Abstract

from arXiv · show

Single-image relighting is highly under-constrained: small illumination changes can produce large, nonlinear variations in shading, shadows, and specularities, while geometry and materials remain unobserved. Existing diffusion-based approaches either rely on intrinsic or G-buffer pipelines that require dense and fragile supervision, or operate purely in latent space without physical grounding, making fine-grained control of direction, intensity, and color unreliable. We observe that a full intrinsic decomposition is unnecessary and redundant for accurate relighting. Instead, sparse but physically meaningful cues, indicating where illumination should change and how materials should respond, are sufficient to guide a diffusion model. Based on this insight, we introduce LightCtrl that integrates physical priors at two levels: a few-shot latent proxy encoder that extracts compact material-geometry cues from limited PBR supervision, and a lighting-aware mask that identifies sensitive illumination regions and steers the denoiser toward shading relevant pixels. To compensate for scarce PBR data, we refine the proxy branch using a DPO-based objective that enforces physical consistency in the predicted cues. We also present ScaLight, a large-scale object-level dataset with systematically varied illumination and complete camera-light metadata, enabling physically consistent and controllable training. Across object and scene level benchmarks, our method achieves photometrically faithful relighting with accurate continuous control, surpassing prior diffusion and intrinsic-based baselines, including gains of up to +2.4 dB PSNR and 35% lower RMSE under controlled lighting shifts.

1. Introduction

Single-image relighting is ill-posed because geometry and materials are unobserved, making illumination changes difficult to control physically. LightCtrl addresses this with sparse physical guidance, a scalable dataset, and diffusion-based relighting.

  • Single-image relighting is ill-posed because geometry and materials are unobserved while lighting changes can cause nonlinear appearance variations.
  • Existing methods trade off physical modeling, controllability, and supervision through limited generalization, weak disentanglement, or complex intrinsic pipelines.
  • The framework uses sparse physically meaningful cues instead of full intrinsic reconstruction or dense G-buffer supervision.
  • LightCtrl combines a scalable diffusion backbone, ScaLight data, a few-shot latent proxy, a lighting-aware mask, and DPO refinement.
  • LightCtrl preserves material appearance and geometry while enabling fine-grained illumination control across object and scene-level benchmarks.

2. Related work

Prior relighting work spans explicit physical modeling, intrinsic decomposition, and end-to-end generative approaches. LightCtrl occupies a middle ground by combining lightweight physical cues with latent diffusion for more precise lighting control.

  • Explicit inverse-rendering methods estimate geometry, albedo, and illumination but require strong supervision and can struggle with complex high-frequency lighting.
  • Diffusion and foundation models improve data-driven illumination synthesis, while explicit differentiable-rendering methods provide physically based light-source control.
  • Large-scale generative approaches may require complex data construction, whereas simpler latent frameworks often lack precise control over intensity, color, and direction.
  • LightCtrl adopts lightweight few-shot physical guidance to target superior efficiency and precision without redundant two-stage intrinsic processing.

3. Method

LightCtrl conditions diffusion on relative lighting and compact physical proxies rather than full intrinsic reconstructions. A lighting-aware mask focuses denoising on affected regions, while DPO stabilizes sparse proxy supervision.

  • LightCtrl computes a relative lighting representation from source and reference conditions, then synthesizes the source appearance under target illumination.
  • The method avoids dense G-buffer supervision by using sparse physical cues and a compact proxy instead of complete intrinsic decomposition.
  • 3.1. Few-shot Latent Proxy Conditioning: The few-shot encoder predicts albedo, normals, roughness, and metallicity as material–geometry cues, with sparse supervision stabilizing the proxy while diffusion trains on unlabeled data.
  • 3.1. Few-shot Latent Proxy Conditioning: The proxy maps are pooled into a conditioning token injected alongside appearance and lighting tokens to constrain denoising.
  • 3.2. Lighting-Aware Mask Prediction: The lighting-aware mask identifies illumination-sensitive regions and weights reconstruction toward them while preserving illumination-invariant areas.
  • 3.3. Post-Training for Latent Encoder: DPO post-training compares preferred ground-truth PBR maps with lower-reward encoder outputs using albedo, roughness, normals, and metallicity criteria.
  • 3.3. Post-Training for Latent Encoder: DPO strengthens proxy physical consistency and improves downstream relighting stability and controllability without modifying the diffusion backbone.

4. Dataset

ScaLight is a large-scale synthetic object dataset designed for controllable, physically consistent relighting. It varies cameras and lighting systematically while retaining fixed geometry and materials and recording complete metadata.

  • ScaLight renders diverse objects under systematically varied directional, point, and environment lighting using a physically based automated pipeline.
  • Multiple camera viewpoints and light configurations are sampled while each frame records camera pose and illumination parameters.
  • A small subset includes material annotations for weak supervision, supporting few-shot intrinsic-cue learning in the proxy encoder.
  • ScaLight supports relighting-pair construction with known illumination differences between source and target renderings.
  • ScaLight contains over 300K controllable 3D objects and more than 1M rendered images while preserving fixed geometry and materials across lighting variations.

5. Experiment

LightCtrl is evaluated on controlled synthetic objects, real indoor scenes, and in-the-wild images using quantitative, qualitative, preference, and ablation studies. Results show accurate, physically consistent relighting, strong generalization, and complementary contributions from its proxy, mask, and DPO components.

  • Evaluation protocol: LightCtrl is evaluated with RMSE, SSIM, and PSNR across diverse lighting changes, alongside qualitative comparisons and component ablations.The controlled evaluation uses ScaLight and examines temperature, position, and energy variations.
  • Object-level evaluation: On 1.5K unseen ScaLight objects, LightCtrl preserves shading discontinuities, specular structure, fine material cues, and consistent albedo under varied illumination.Compared with intrinsic-based and latent-only diffusion methods, it exhibits fewer color shifts, less oversmoothing, and more stable high-frequency detail.
  • Scene-level evaluation: LightCtrl generalizes from synthetic object training to real indoor scenes, producing coherent global illumination, shadows, and highlights across complex geometry and mixed materials.Scene-level evaluation on MIIW reports consistent superiority over intrinsic-based and diffusion baselines in relighting accuracy and perceptual quality.
  • User preference: 81.45% and 55.73% are LightCtrl’s highest preference rates in controlled object-level and complex scene-level evaluations, respectively.The user study evaluates preference separately on ScaLight objects and MIIW/RWR scenes with 35 participants.
  • Ablation study: Removing the latent proxy degrades RMSE and PSNR across lighting variations, especially for color-temperature and intensity shifts, while SSIM also falls for temperature and energy.The proxy supplies material- and geometry-dependent cues that stabilize shading and reflectance as illumination changes.
  • Ablation study: Removing the lighting-aware mask lowers RMSE, SSIM, and PSNR across all variations, while disabling DPO causes the largest ablation loss with noisier albedo and distorted normals.The mask limits edits to illumination-sensitive pixels, whereas DPO stabilizes intrinsic cues under sparse supervision.
  • Ablation study: The proxy, mask, and DPO address distinct challenges—material awareness, spatial selectivity, and intrinsic stability—and contribute synergistically to relighting performance.This conclusion is supported by the component-wise ablation trends.

6. Limitations

LightCtrl’s strongest evidence is concentrated in synthetic, object-centric settings, while real-scene performance is less robust under complex light transport and specular interactions. Its conditioning also struggles with distant cast shadows and high-frequency geometry or highlights, motivating richer geometry and occlusion supervision.

  • Real-scene scope: Real-scene transfer is less robust and produces more mixed quality gains than synthetic, object-centric evaluation under complex global illumination.The model often preserves stable appearance cues but remains sensitive to long-range light transport and strong specular interactions.
  • Shadow limitations: Global cast-shadow recasting is a primary failure mode, especially for distant objects and large viewpoint-dependent lighting changes.The sparse single-view proxy lacks dense multi-view geometry and explicit occlusion reasoning, so the denoiser may produce smooth ambient adjustments instead of precise shadow intersections.
  • High-frequency detail: Strong illumination contrast and concentrated highlights can oversmooth fine structural details and partially wash out local textures.The proxy may not provide enough high-frequency constraints to separate geometry variation from illumination variation, causing flattened relief around bright reflections.
  • Residual strengths: Color consistency remains comparatively stable despite limitations in global shadows and highlight-heavy regions.The proxy still anchors albedo better than purely latent editing baselines, according to the supplied discussion.
  • Future directions: Future work targets global light-transport reasoning, richer geometry and occlusion supervision, and diagnostics for shadow, structure, and highlight fidelity.These directions are intended to narrow the gap between object-centric training and reliable real-scene controllability.

C. Conditioning Architecture and Tokens

LightCtrl conditions a U-Net diffusion backbone with appearance, illumination, and proxy tokens through cross-attention, while a spatial mask concentrates conditioning on illumination-sensitive regions. This design preserves stable appearance cues while guiding shading, specularity, and global illumination changes.

  • Architecture: LightCtrl injects appearance, illumination, and proxy cues into a U-Net through a unified token-based cross-attention interface.Each token type is encoded into a fixed-dimensional embedding and consumed during denoising.
  • Appearance token: The appearance token anchors object identity using coarse geometry and view-dependent texture cues that should remain invariant under relighting.It is produced by a frozen CLIP-ViT variant vision encoder.
  • Lighting token: The lighting token represents directional, energetic, and chromatic changes and determines intended shading and highlight appearance during denoising.It is obtained by mapping the relative illumination encoding through a lightweight MLP.
  • Proxy token: The proxy token adds soft material and geometry priors inferred by the PBR encoder from sparse PBR supervision.This compensates for the under-constrained nature of single-image relighting.
  • Cross-attention: Cross-attention increasingly relies on lighting and proxy tokens in deeper layers, while early layers prioritize appearance to preserve object identity.The mechanism dynamically selects conditioning streams at each spatial location.
  • Mask-weighted fusion: Mask-weighted fusion makes stable regions rely primarily on appearance cues while illumination-variant regions receive full conditioning.This prevents unnecessary texture distortion and preserves material fidelity under large lighting edits.

D. Training Details

LightCtrl is trained with mixed-precision AdamW optimization on four H800 GPUs for 400K diffusion steps, using ScaLight validation selection. Inference uses approximately 1.2B parameters and generates 512 × 512 images in about 0.84 seconds over 50 steps.

  • Optimization: LightCtrl trains on 4×NVIDIA H800 GPUs with fp16 mixed precision, AdamW, a 1 × 10^-4 learning rate, and batch size 64 total.The optimizer uses β1=0.9, β2=0.999, and weight decay 1×10^-2.
  • Training schedule: The model trains for 400K diffusion steps with 5K warm-up steps, a DDIM scheduler with 1000 timesteps, and the standard v-prediction objective.Models are selected using best validation performance on the held-out ScaLight split.
  • Efficiency: ~1.2B parameters generate a 512 × 512 image in ~0.84s using 50 steps, with peak memory usage of 2.53GB on one H800.This measurement uses bf16 precision optimization on an NVIDIA H800 with CUDA 12.2.

E. Implementation Details of the DPO Mechanism

The DPO stage robustifies the latent proxy under extreme PBR-label sparsity by contrasting ground-truth PBR maps with the encoder’s own artifact-prone predictions. This directly penalizes entangled shading failures that standard supervised fine-tuning can memorize.

  • Sparse supervision: Only ~3% of training data has explicit PBR annotations, so post-training uses sparse supervision to robustify the latent proxy encoder.Joint training first uses large-scale relighting pairs without ground-truth intrinsic supervision.
  • Preference construction: DPO replaces SFT by teaching physical consistency through preferred ground-truth PBR maps and rejected predictions from the current model.The dynamically generated negatives contain the artifacts and entangled shading the network naturally produces.
  • Motivation: SFT can overfit by memorizing lighting artifacts such as baking cast shadows into predicted albedo.DPO instead explicitly penalizes these model-generated failure modes.

F. Impact of PBR Supervision Scale

Increasing PBR supervision improves shading-transfer fidelity and relighting accuracy, but gains diminish beyond roughly 9K samples. The results indicate that sparse supervision can provide stable material cues without dense intrinsic annotations.

  • Sparse PBR supervision provides meaningful benefits for relighting tasks.
  • PBR-supervised examples improve shading-transfer fidelity and relighting accuracy when increased.
  • Beyond roughly 9K supervised samples, performance gains show clear marginal returns.

G. Additional Qualitative Results

Additional qualitative results cover controlled changes in light position, intensity, and color temperature across ScaLight and MIIW. These examples show smooth, photometrically consistent transitions across continuous lighting variations.

  • Additional Qualitative Results: Additional figures evaluate controlled edits of light position, illumination intensity, and color temperature.
  • Additional Qualitative Results: The additional samples demonstrate smooth, photometrically consistent transitions across continuous lighting variations.
  • Additional Qualitative Results: The supplementary material also details relative illumination encoding, lighting-aware mask construction, conditioning-token integration, training schedules, and hardware.
  • Additional Qualitative Results: ScaLight results include qualitative comparisons under varying light positions, intensity, and temperature, with reference, ground-truth, and LightCtrl prediction views shown for selected figures.
  • Additional Qualitative Results: MIIW results provide additional qualitative relighting examples with reference, ground-truth, and LightCtrl prediction views.
Loading 2603.15555v1…