Source-linked AI summary
RGB$\leftrightarrow$X: Image decomposition and synthesis using material- and lighting-aware diffusion models
Zheng Zeng, Valentin Deschaintre, Iliyan Georgiev, Yannick Hold-Geoffroy, Yiwei Hu, Fujun Luan, Ling-Qi Yan, Miloš Hašan
TL;DR
The paper addresses the separation between intrinsic image decomposition, forward rendering, and generative synthesis. It introduces a unified diffusion framework for estimating scene channels and synthesizing realistic interior images from full or partial channels, achieving realistic outputs while supporting flexible appearance control.
Problem
Intrinsic-channel estimation is under-constrained, while traditional rendering requires detailed scene descriptions and generative models are difficult to control precisely.
Method
The paper develops an improved diffusion RGB→X model with lighting estimation and an X→RGB diffusion model trained to synthesize images from arbitrary subsets of intrinsic channels.
Results
The models extract scene properties at least as well as previous specialized methods and synthesize realistic interior images, including outputs that closely match path-traced references under suitable inputs.
Takeaways & Limitations
Joint decomposition and synthesis enable applications such as material editing and object insertion while allowing unspecified appearance properties to be plausibly generated.
Takeaways & Limitations
The intrinsic channels are imprecise and cannot directly represent glass, while roughness and metallicity require extra care because training data are unreliable and properties are ambiguous.
Abstract
from arXiv · showhide
The three areas of realistic forward rendering, per-pixel inverse rendering, and generative image synthesis may seem like separate and unrelated sub-fields of graphics and vision. However, recent work has demonstrated improved estimation of per-pixel intrinsic channels (albedo, roughness, metallicity) based on a diffusion architecture; we call this the RGB$\rightarrow$X problem. We further show that the reverse problem of synthesizing realistic images given intrinsic channels, X$\rightarrow$RGB, can also be addressed in a diffusion framework. Focusing on the image domain of interior scenes, we introduce an improved diffusion model for RGB$\rightarrow$X, which also estimates lighting, as well as the first diffusion X$\rightarrow$RGB model capable of synthesizing realistic images from (full or partial) intrinsic channels. Our X$\rightarrow$RGB model explores a middle ground between traditional rendering and generative models: we can specify only certain appearance properties that should be followed, and give freedom to the model to hallucinate a plausible version of the rest. This flexibility makes it possible to use a mix of heterogeneous training datasets, which differ in the available channels. We use multiple existing datasets and extend them with our own synthetic and real data, resulting in a model capable of extracting scene properties better than previous work and of generating highly realistic images of interior scenes.
1 INTRODUCTION
The paper unifies intrinsic-channel estimation and realistic image synthesis in diffusion models for interior scenes. Its models support partial appearance specification, heterogeneous training data, and downstream editing applications.
- 1 INTRODUCTION: The RGB→X model improves prior work by adding lighting estimation and using more training data from heterogeneous datasets.The datasets differ in their available channels.
- 1 INTRODUCTION: The X→RGB model balances generative flexibility with appearance control by enforcing selected properties while hallucinating plausible unspecified content.This contrasts with traditional rendering's need for full scene specification and generative models' limited precise control.
- 1 INTRODUCTION: The X→RGB model synthesizes realistic images from full or partial intrinsic channels, optionally guided by text prompts.Channel dropout enables conditioning on arbitrary subsets of channels.
- 1 INTRODUCTION: The framework jointly addresses RGB→X intrinsic-channel estimation and X→RGB realistic image synthesis with diffusion models for indoor scenes.Intrinsic channels describe geometric, material, and lighting information.
- 1 INTRODUCTION: The unified framework is intended to support material editing, relighting, realistic rendering, and rendering from simple or under-specified scene definitions.The paper presents this as a first step toward unified decomposition and synthesis frameworks.
2 RELATED WORK
Related work spans diffusion-based image generation, intrinsic decomposition, normal estimation, relighting, and neural synthesis from decomposed representations. The paper positions its approach as jointly handling decomposition and synthesis without requiring ray tracing.
- Generative models for images: Diffusion models emerged as a recent approach to realistic image generation after GAN-based methods, whose training can be challenged by mode collapse.Diffusion models iteratively denoise images from pure noise.
- Intrinsic decomposition: Intrinsic decomposition has expanded from albedo and shading to per-pixel roughness, metallicity, lighting, and other intrinsic channels for interior scenes.Recent diffusion-based work leverages image-generation priors for per-pixel inverse rendering.
- Neural image synthesis from decompositions: Earlier neural synthesis methods learned screen-space shading, global illumination, or ray-traced effects from intrinsic channels.These approaches include CNNs, conditional GANs, and screen-space ray tracing.
- Datasets: The paper combines heterogeneous datasets with different channel availability to support its decomposition and synthesis models.Table 1 marks channels as available, unavailable, or not fully reliable across four datasets.
- Relighting: Single-image relighting methods have used explicit or implicit representations, but prior approaches were limited to simple lighting or specialized settings.Related work includes per-pixel inverse rendering for relighting interior scenes.
3 INTRINSIC CHANNELS AND DATASETS
The models use per-pixel geometric, material, and lighting channels and train across datasets that provide complementary or estimated annotations. These representations are useful but remain less precise than traditional material systems.
- 3.1 Intrinsic channels: The intrinsic representation contains camera-space normals, albedo, roughness, metallicity, and diffuse irradiance.Albedo covers diffuse dielectric color and specular metallic color; irradiance represents lighting reaching surfaces.
- 3.1 Intrinsic channels: Depth was omitted because normals can provide depth estimates while retaining more high-frequency local variation.The paper treats normals as more informative for local geometric detail.
- 3.1 Intrinsic channels: The intrinsic channels are imprecise compared with traditional material systems and cannot represent glass directly.Glass is assigned zero roughness and metallicity, with context used to plausibly infer objects or content behind it.
- 3.2 Datasets: No existing dataset provides all required channels, so the authors combine datasets with partial information and construct additional data.The desired data include RGB images, normals, albedo, roughness, metallicity, irradiance, and captions.
- 3.2 Datasets: InteriorVerse supplies over 50,000 synthetic images with normals, albedo, roughness, metallicity, and RGB, but contains noise, questionable roughness and metallicity, and synthetic-style biases.The authors denoise its images and avoid using its roughness and metallicity values.
- 3.2 Datasets: Hypersim contributes over 70,000 photorealistic renders with normals, albedo, irradiance, and greater appearance variety, but lacks several material channels and remains insufficient alone for highly realistic synthesis.Evermotion adds 17,000 images from 85 scenes and provides the only reliable roughness and metallicity source.
4 THE RGB→X MODEL
The RGB→X model estimates intrinsic channels from an input RGB image by fine-tuning Stable Diffusion 2.1 in latent space. A prompt-controlled single-channel output strategy supports heterogeneous datasets while retaining one network.
- The model fine-tunes Stable Diffusion 2.1 to estimate intrinsic channels X from an input RGB image.Its output includes albedo, normal, roughness, metallicity, and lighting channels.
- The model encodes the input image and concatenates it with a noisy diffusion latent before iterative denoising.A frozen pre-trained encoder is used for the image and intrinsic-image encodings.
- The RGB→X training objective uses v-prediction, which the authors find performs better than noise prediction.The noise amount is sampled as a training time step, and the noisy latent is formed by adding Gaussian noise to the target latent.
- Handling multiple output channels: A repurposed text prompt acts as a switch selecting one output channel at a time across normal, albedo, roughness, metallicity, and irradiance.The switch is used because datasets may contain heterogeneous subsets of intrinsic channels.
- The prompt-switching approach performs similarly to separate modality-specific models while requiring fine-tuning and storing only one network’s weights.
5 THE X→RGB MODEL
The X→RGB model synthesizes RGB images from intrinsic channels using a diffusion model conditioned on encoded scene properties. Optional conditions, channel dropout, and coarse lighting inputs let it use incomplete information and heterogeneous datasets.
- The X→RGB model fine-tunes Stable Diffusion 2.1 to synthesize realistic RGB images from intrinsic channels X.
- The model encodes intrinsic-channel inputs, concatenates them with a noisy RGB latent, and uses text embeddings as additional cross-attention control.The RGB target latent directly encodes the image, while additional input convolution weights accommodate conditional channels.
- Handling heterogeneous data: Intrinsic-channel dropout enables joint training on heterogeneous datasets and image generation with any subset of available conditions.At inference, omitted channels are handled by the model’s learned prior.
- When albedo or lighting is omitted, the model generates plausible images by compensating for missing information with its prior.
- Low-resolution lighting: Lighting is downsampled to latent resolution so it serves as a coarse hint rather than precise per-pixel control when editing detailed normals.The authors motivate this design because high-resolution irradiance follows geometry and normals closely.
- Fine-tuning for inpainting: Inpainting support is added by concatenating downsampled mask and masked-image channels to the conditioning latent.
6 RESULTS
The experiments evaluate RGB→X on synthetic and real inputs, X→RGB against path tracing and partial-channel controls, and applications including material replacement and object insertion. Results show strong intrinsic estimation and realistic, controllable image synthesis, with variability for difficult material properties.
- RGB→X on synthetic and real inputs: Our RGB→X model removes reflections, highlights, shadows, and color cast while producing flatter constant albedo regions than prior methods.On challenging real images, it can outperform previous methods, although it may remove floor wear learned from synthetic materials.
- RGB→X on synthetic and real inputs: Our model produces diffuse irradiance estimates that closely match synthetic ground truth and remain realistic on real inputs.The estimates preserve intricate shadow patterns, avoid material-property leakage, and shift plausibly under colored lighting.
- RGB→X on synthetic and real inputs: Our RGB→X model generates more plausible roughness and metallicity than previous publicly available methods, but these channels have higher sampling variance.Reliable training data are limited, and the properties are difficult to infer without suitable high-frequency illumination.
- RGB→X on synthetic and real inputs: Our model estimates normals plausibly on synthetic and real inputs, including high-frequency geometry and flat surfaces with texture or high-frequency lighting.The authors report outperforming Zhu et al. and slightly improving on PVT-normal, without claiming general improvement across images.
- RGB→X on synthetic and real inputs: Our RGB→X model achieves the best PSNR and LPIPS values on all evaluated channels except irradiance, which lacks existing comparison methods.Albedo, normal, and irradiance use the Hypersim test set, while roughness and metallicity use Evermotion.
- X→RGB model results: X→RGB closely matches a path-traced reference on an unseen synthetic kitchen and can synthesize plausible images from partial channels with text-controlled lighting or object colors.Text control works better with few objects, while material replacement and object insertion produce edited scenes with updated appearance, lighting, color bleeding, and shadows.
7 CONCLUSION
The paper presents a unified diffusion framework for intrinsic-channel estimation and realistic image synthesis. It demonstrates realistic rendering from partial appearance properties and applications including material editing and object insertion.
- The framework jointly supports intrinsic-channel estimation from images and realistic image synthesis from those channels.
- The RGB→X model matches or exceeds previous methods across intrinsic channels including albedo, normals, roughness, metallicity, and lighting.
- The X→RGB model synthesizes realistic images while allowing unspecified appearance properties to be generated plausibly.
- Combining both models enables applications such as material editing and object insertion.
A NOTE ABOUT TRADITIONAL RENDERING FOR X→RGB
Traditional rendering cannot easily produce realistic images from intrinsic channels alone because it requires fuller scene specifications. The generative X→RGB model can still synthesize reasonable results from imperfect or partial channels, optionally guided by text.
- Traditional rendering requires full 3D geometry and explicit light and material definitions beyond intrinsic channels.
- Intrinsic channels alone lack sufficient information for traditional rendering to produce realistic images.
- When intrinsic channels are imperfect or partial, traditional rendering is not viable, whereas the generative model can still produce reasonable results.
- Appropriate text prompts can provide additional control over results from imperfect or partial intrinsic channels.
B IMPLEMENTATION
Both models are fine-tuned from Stable Diffusion 2.1 using multiple interior-scene datasets, with X→RGB receiving additional ImageDecomp data. Inference uses DDIM sampling and supports resolutions larger than the training crops.
- Both models are fine-tuned from pre-trained Stable Diffusion 2.1 on InteriorVerse, Hypersim, and Evermotion.
- The X→RGB model is additionally trained on ImageDecomp, constructed from RGB images using the RGB→X model.
- Training uses batch size 256, AdamW, learning rate 1e-5, and random 512 × 512 crops.
- Inference uses the DDIM sampler with 50 steps, and test images can be processed at larger resolutions such as 1080p.
- Classifier-free guidance is used for X→RGB but omitted for RGB→X because it impairs quality without providing benefits.
C DISCUSSION AND LIMITATIONS
The models face limitations from imperfect synthetic data, dataset bias, incomplete conditioning control, and reduced quality above 2K resolution. Channel-specific training choices also affect performance, with larger multi-channel latent outputs proving difficult to train.
- Discussion and Limitations: Synthetic datasets contain shading errors, unreliable material parameters, noise, and aliasing artifacts that limit quality.
- Discussion and Limitations: Interior-scene datasets encode biases that can limit editing freedom, while source-selection and dropout probabilities influence preferred inputs.
- Discussion and Limitations: Classifier-free guidance does not work in this context, leaving the diversity-versus-condition-adherence trade-off without an equivalent control mechanism.
- Discussion and Limitations: Image quality degrades beyond 2K resolution because the networks are trained on 512 × 512 resolutions.
- Additional Results: A five-channel latent output remains difficult to train and performs poorly after 100 epochs.
- Additional Results: A four-channel latent output yields reasonable albedo after 100 epochs but performs poorly on the other channels.
- Additional Results: A three-channel latent output produces clean albedo and decent roughness and metallicity, but introduces albedo color shifts and roughness distortions.