PaperlayerView exact revision on arXiv

Source-linked AI summary

RelightFormer: Feed-forward Generative Transformer for Multiview Object Relighting

Hejun Wang, Jinxi Li, Junwei Jiang, Shiwei Mao, Hu Cheng, Shouwang Huang, Bo Yang

arXiv:2609.07414v1cs.CVcs.AIcs.GRcs.LGcs.MM

TL;DR

Image relighting remains difficult because inverse-rendering pipelines are ill-posed and existing generative methods may miss multi-view cues. RelightFormer uses a feed-forward Transformer with latent illumination injection, permutation-invariant multi-view encoding, and large-scale training data to directly relight images. The paper reports strong performance across single-view, multi-view, and novel-view relighting, while novel-view relighting additionally requires reconstructing unseen viewpoints.

  • Problem

    Image relighting requires changing illumination while preserving geometry and materials, but inverse rendering is ill-posed and single-image generative models can ignore multi-view cues.

  • Method

    RelightFormer directly generates relit single- and multi-view images using latent illumination cross-attention, permutation-invariant positional encoding, and the LOD dataset.

  • Results

    RelightFormer clearly outperforms baselines across multiple benchmarks and demonstrates state-of-the-art visual quality in single-view and multi-view object relighting.

  • Takeaways & Limitations

    The approach provides a feed-forward alternative to explicit inverse rendering for relighting from single- or multi-view inputs.

  • Takeaways & Limitations

    Novel-view relighting additionally requires reconstructing unseen viewpoints, whereas the model is primarily designed for input-view relighting.

Abstract

from arXiv · show

Image relighting is traditionally tackled via complex inverse rendering pipelines, which suffer from ill-posed optimization, or single-image generative models that ignore crucial multi-view cues necessary for understanding 3D geometry and material interactions. To address these limitations, we introduce a feed-forward generative Transformer for direct single- and multi-view image relighting that entirely bypasses explicit intrinsic property estimation. Adapted from a video foundation model, our architecture features a latent illumination module that dynamically injects target environment maps into spatial features via cross-attention. Furthermore, we employ permutation-invariant positional encodings to symmetrically process unordered multi-view inputs without sequential bias. To train this robust data-driven model, we construct the massive Laval Objaverse Dataset (LOD), comprising 90K objects and 39K unique illuminations. Extensive experiments demonstrate state-of-the-art visual quality, photorealistic relighting quality, and strong zero-shot generalization across single-view, multi-view, and novel-view relighting tasks.

1 Introduction

RelightFormer addresses the ill-posedness and missing multi-view reasoning of prior relighting methods with a feed-forward Transformer that directly conditions relit outputs on images and target illumination. Its design combines latent illumination injection, permutation-invariant multi-view encoding, and a large diverse training dataset.

  • Motivation: Prior inverse-rendering methods estimate intrinsic properties through per-scene optimization, while single-image generative methods overlook multi-view cues for 3D geometry and material interactions.Recovering geometry and materials from images is ill-posed, so incorrect estimates can produce artifacts under new illumination.
  • RelightFormer: RelightFormer directly synthesizes relit single- or multi-view images from input images and a target illumination without explicitly estimating intrinsic properties.The model extends direct generative relighting to fuse multi-view images during training and handle an arbitrary number of views at test time.
  • Architecture: A latent illumination module maps target environment maps into latent codes and injects them into image features through cross-attention layers.This mechanism adapts a video foundation model for illumination-conditioned image relighting.
  • Architecture: Permutation-invariant positional encoding treats multi-view inputs symmetrically, avoiding sequential bias when fusing global illumination across views.The encoding is designed so that all input views receive equal treatment.
  • Training data: LOD contains 90,545 3D objects and 39,008 illuminations, supporting the model’s data-driven relighting approach.The dataset is rendered from Objaverse objects and illuminations from the Laval Database.
  • Results: RelightFormer demonstrates state-of-the-art single-view and multi-view relighting performance, outperforming existing baselines in visual quality.The reported contributions frame this result across direct image relighting settings.

2 Related Works

Prior relighting and inverse-rendering methods rely on optimization or domain restrictions to manage an ill-posed problem. Generative approaches broaden the formulation, but existing pipelines still face generalization and visual-effect challenges.

  • Inverse Rendering: Inverse-rendering methods estimate geometry, materials, and lighting but typically require time-consuming optimization and handcrafted regularization.These pipelines minimize rendering losses while often relying on geometry or lighting priors.
  • Inverse Rendering: These inverse-rendering pipelines struggle to generalize to novel objects and challenging visual effects such as specular highlights.The limitation follows from the optimization-heavy and regularized nature of the pipelines described in the related work.
  • Image Relighting: Single-image relighting is severely ill-posed, leading prior methods to restrict their scope to domains such as portraits, human bodies, or outdoor scenes.Domain restriction is used to make the relighting problem more tractable.
  • Generative Relighting: Diffusion-based methods formulate relighting as a generative task and directly produce relit images conditioned on illumination, but most focus on limited input settings.The supplied related-work passage introduces generative relighting as a response to inverse-rendering difficulty.

3 RelightFormer

RelightFormer adapts a latent video diffusion Transformer for direct relighting from reference images and target environment maps, combining geometry-aware multi-view reasoning with learned illumination injection. Its positional encoding treats unordered views symmetrically by deriving rotations from camera configurations rather than sequence order.

  • 3.1 Preliminaries: Latent Diffusion Transformers: RelightFormer encodes reference images, target illumination, and noisy target latents before jointly processing their tokens in a latent Transformer.Reference and illumination representations receive ray embeddings, while noisy target latents are sampled from a standard Gaussian process.
  • 3.1 Preliminaries: Latent Diffusion Transformers: The flow-matching process linearly interpolates clean latents with Gaussian noise, then learns a velocity field whose ODE transports noise samples toward the data distribution.The forward path uses z_t = (1 − t)z_0 + tε, and generation solves the learned reverse dynamics.
  • 3.2 Latent Illumination Module: Within each Transformer block, multi-view self-attention aggregates intra- and cross-view information while illumination cross-attention injects environment-map cues into image features.Image tokens query illumination tokens that encode incident lighting, and the two attention outputs are combined before feed-forward processing.
  • 3.2 Latent Illumination Module: After the final Transformer layer, reference tokens are discarded and the updated noise tokens predict the flow-matching velocity for relit-image synthesis.This preserves the target-view token stream as the output pathway while using reference views as conditioning information.
  • 3.2 Latent Illumination Module: The illumination module learns a discrete, weighted aggregation of incident lighting contributions, with image features querying ray-embedded illumination representations at each spatial location.This design is motivated by the structural similarity between attention and the rendering integral involving geometry and material-dependent light contributions.
  • 3.3 Permutation-Invariant Multi-view Encoding: PRope makes multi-view attention permutation-invariant by deriving token transformations from physical camera configurations and patch geometry instead of arbitrary frame indices.Consequently, the same view set can be processed without sequential bias from input ordering.

4 Laval Objaverse Dataset

The Laval Objaverse Dataset addresses the lack of large-scale multi-view relighting data by rendering diverse Objaverse objects under a broad set of HDR environment-map conditions.

  • 4 Laval Objaverse Dataset: The dataset is designed to fill the gap in public large-scale datasets for multi-view object relighting.Its construction combines diverse 3D objects with abundant lighting conditions.
  • 4 Laval Objaverse Dataset: LOD renders multi-view images from 90,545 high-quality Objaverse objects curated by Neural Gaffer.Its object collection supplies the 3D assets used to construct the relighting dataset.
  • 4 Laval Objaverse Dataset: LOD derives illumination conditions from Laval Indoor and Outdoor HDR datasets and expands them through uniformly sampled horizontal rotations.The rotations increase the illumination pool by 16×, yielding 39,008 unique illumination conditions.

5 Experiments

Experiments evaluate RelightFormer across single-image, multi-view, real-world zero-shot, material-specific, and novel-view relighting settings. Results show robust performance and efficient novel-view inference, while rapidly varying specular illumination and unseen-view reconstruction remain challenging.

  • 5.1 Single Image Relighting: RelightFormer generates photorealistic single-image relit results despite missing multi-view correspondences, leveraging priors learned from large-scale training data.LightSwitch and Reli3D degrade in ambiguous cases caused by absent correspondence-based geometry and material estimation.
  • 5.2 Multi-view Image Relighting: RelightFormer consistently outperforms all baselines in multi-view relighting, with performance improving monotonically as the number of reference views increases.The evaluation includes 16-to-16 and 32-to-32 settings, each containing 7,248 multi-view image pairs.
  • 5.3 Real-world Generalization and Material Analysis: RelightFormer achieves competitive environment-map results and the best PSNR and SSIM scores for rotating point-light relighting on real-world OLATverse objects.Neural Gaffer obtains slightly better sPSNR and LPIPS under rotating point lights, leaving rapidly varying specular details challenging.
  • 5.3 Real-world Generalization and Material Analysis: RelightFormer performs best on most material categories, including glossy, translucent, and furry objects, while Neural Gaffer achieves higher PSNR on plastic and stone.The authors attribute remaining ambiguity to the inherently ill-posed nature of image-based relighting for challenging materials.
  • 5.4 Novel View Relighting: RelightFormer-Post achieves on-par or slightly better performance than Reli3D across all metrics at comparable inference time for novel-view relighting.Optimization-based methods require one to twenty hours, whereas the latent-space pipeline completes inference within minutes; 3DGS+RelightFormer offers an intermediate quality–efficiency trade-off.

6 Ablation Analysis

The ablations examine illumination conditioning, viewpoint flexibility, and permutation-invariant processing for multi-view relighting. Results indicate that cross-attention and order-agnostic view processing are important design choices.

  • Illumination Conditioning Strategy: Cross-attention injects illumination tokens into image features, addressing the domain mismatch between spherical environment maps and image-space pixels.The comparison contrasts channel-wise concatenation with the proposed cross-attention mechanism.
  • Viewpoint Flexibility during Training: Training with varying numbers of reference views compares fixed single-view input against ranges of 1–8 and 1–16 views.The 1–16-view variant is the default setting.
  • Permutation-Invariant Design: Permutation-invariant positional encoding replaces standard RoPE so multi-view inputs are processed in an order-agnostic manner.The ablation evaluates the effect of removing this design.
  • Permutation-Invariant Design: Removing permutation invariance causes a notable performance drop in the ablation results.This supports treating unordered input views symmetrically during multi-view relighting.

7 Conclusion

RelightFormer is a feed-forward model for single- and multi-view object relighting that bypasses explicit inverse rendering. It combines latent illumination integration with a large diverse dataset, and experiments show strong performance across benchmarks.

  • Conclusion: RelightFormer relights objects from single- and multi-view images without explicit inverse rendering.Its architecture is built on a video generation model and includes a latent illumination module.
  • Conclusion: The model integrates lighting cues into generation and is trained with a large-scale dataset spanning diverse objects and illumination conditions.The paper identifies extension to scene-level and dynamic video relighting as future work.
  • Conclusion: Extensive experiments show that RelightFormer outperforms baselines across multiple benchmarks.The conclusion reports this as the overall experimental finding.

A More Details in Methodology

The methodology aligns reference images and environment panoramas in a shared latent formulation, represents their rays explicitly, and learns ray embeddings jointly with the relighting model.

  • Latent Encoding: A frozen Wan2.1 VAE projects reference images and environment panoramas into compact latent representations.The same first-frame encoding pathway is applied symmetrically to all input modalities.
  • Ray Representation: Perspective-image rays use direction and camera-dependent position, while environment-map rays originate at the scene center and have zero moment.This yields a compact Plücker representation for incident-light directions.
  • Ray Embedding: The RaysEmbedding module projects Plücker ray representations into structured latent features using a shallow Conv2D-based sinusoidal network.Its parameters are optimized jointly with the full model rather than pretrained separately.
  • Ray Embedding: The RaysEmbedding output dimension is 1536 to match the Transformer setting.The architectural specification is reported in Table 6.

B More Details of Laval Objaverse Dataset

LOD is an open-source multi-view relighting dataset built from filtered Objaverse objects, diverse HDR illuminations, controlled rendering, and leakage-preventing splits.

  • Dataset Overview: LOD is presented as a large-scale open-source multi-view relighting dataset, unlike alternatives limited by licensing or scale and diversity.The paper states that its materials allow full public release.
  • Object Filtering: The dataset uses Objaverse objects filtered to exclude poor geometry or materials.Object selection follows criteria from the relitObjaverse dataset.
  • Lighting Selection: LOD combines open-source Laval indoor and outdoor HDR datasets, rotating each illumination map 16 times and sampling balanced indoor and outdoor maps per object.Each object receives 16 environment maps in total.
  • Rendering Protocol: Objects are unit-scaled and placed at the origin, with cameras distributed on surrounding spheres at distances of 1.8 or 2.2 units.Cameras are oriented toward the scene center.
  • Dataset Partitioning: Training, validation, and test sets use an 8:1:1 split, excluding training objects, camera poses, and illumination conditions from evaluation sets.The dataset contains 39,008 environment maps divided into training and test sets.

C More Details of Experiments

The experiments detail qualitative and quantitative evaluations across Laval Objaverse, OLATverse, TensoIR, and Stanford-ORB, using controlled test-pair construction and foreground-focused metrics. Additional details cover baseline implementation, training configurations, and material-stratified results.

  • Results and Implementation: Qualitative results are reported for Laval Objaverse and real-world OLATverse, with material-stratified OLATverse results and TensoIR quantitative results provided in tables.Table 8 reports OLATverse metrics, while Table 9 reports 16-image relighting results on TensoIR.
  • Evaluation Protocol: 7,248 testing data pairs were selected for each N-to-N relighting sub-task with N∈{1,16,32}.Each source illumination was paired with 16 target illuminations for objects in the test split.
  • Evaluation Protocol: 5,082 lighting mappings were formed for OLATverse environment-map relighting across 42 objects and 11 environment maps.The mappings combine all 11 × 11 source-target lighting combinations per object.
  • Evaluation Protocol: Evaluation pairs were fixed with a predetermined random seed, so all baselines and RelightFormer used identical test pairs.The protocol fixes viewpoint sampling, illumination assignment, and cropping parameters.
  • Generalization and Metrics: TensoIR tests zero-shot generalization because its 14 target illuminations and environment maps are absent from training.Each mapping uses 16 independently sampled viewpoints to construct the relighting reference set.
  • Generalization and Metrics: sPSNR, PSNR, SSIM, and LPIPS were computed on foreground-masked images, excluding background discrepancies.PSNR and sPSNR use foreground-only MSE, while SSIM and LPIPS operate on foreground-masked images.
  • Results and Implementation: Neural Gaffer was fine-tuned for 20K iterations on 8 NVIDIA RTX 3090 GPUs with a global batch size of 64.The fine-tuned model retained white-background compositing for training and inference, while a validation variant used 2,989 objects.
  • Results and Implementation: All variants were trained to convergence with a 1 × 10^-4 learning rate and cosine decay schedule.
Loading 2609.07414v1…