PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Models: What Do They Know? Do They Know Things? Let's Find Out!

Xiaodan Du, Nicholas Kolkin, Greg Shakhnarovich, Anand Bhattad

arXiv:2311.17137v3cs.CVcs.AIcs.GRcs.LG

TL;DR

The paper asks whether generative models encode recoverable scene intrinsics and whether this knowledge can be extracted across architectures with little training. It uses LoRA to recover intrinsic maps, finding recovery across model types and a positive association between generator quality and intrinsic accuracy.

  • Problem

    It remains unclear whether generative models encode scene intrinsics such as depth, normals, albedo, and shading in recoverable form.

    Source: §1 · p. 2

  • Method

    The authors use small LoRA adaptations to recover intrinsic maps across generative models, retaining the generator’s output head and using limited labeled data.

    Source: §1 · pp. 2–3

  • Results

    Generative models across architectures encode recoverable intrinsics, and higher-quality generators tend to produce more accurate intrinsic representations.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Recovered intrinsic knowledge could support relighting, object compositing, and image editing without large labeled datasets or extensive retraining.

    Source: §1 · p. 2

    How generative models use intrinsic information during image generation remains ambiguous.

    Source: §6 · p. 11

Abstract

from arXiv · show

Generative models excel at mimicking real scenes, suggesting they might inherently encode important intrinsic scene properties. In this paper, we aim to explore the following key questions: (1) What intrinsic knowledge do generative models like GANs, Autoregressive models, and Diffusion models encode? (2) Can we establish a general framework to recover intrinsic representations from these models, regardless of their architecture or model type? (3) How minimal can the required learnable parameters and labeled data be to successfully recover this knowledge? (4) Is there a direct link between the quality of a generative model and the accuracy of the recovered scene intrinsics? Our findings indicate that a small Low-Rank Adaptators (LoRA) can recover intrinsic images-depth, normals, albedo and shading-across different generators (Autoregressive, GANs and Diffusion) while using the same decoder head that generates the image. As LoRA is lightweight, we introduce very few learnable parameters (as few as 0.04% of Stable Diffusion model weights for a rank of 2), and we find that as few as 250 labeled images are enough to generate intrinsic images with these LoRA modules. Finally, we also show a positive correlation between the generative model's quality and the accuracy of the recovered intrinsics through control experiments.

1 INTRODUCTION

The paper asks whether generative models encode recoverable scene intrinsics and whether a general, lightweight method can reveal them. It argues that LoRA recovers intrinsic images across model families and that recovery quality correlates positively with generator quality.

1 INTRODUCTION

  • The open question is whether realistic image generation reflects encoded scene properties such as depth, normals, albedo, and shading that can be explicitly recovered.The paper also asks whether generation instead relies on abstract representations of the world.Source: §1 · p. 2
  • Recovering embedded intrinsic knowledge could support relighting, object compositing, and image editing without large labeled datasets or extensive model retraining.Source: §1 · p. 2
  • Prior findings cover specific models, while the ubiquity of intrinsic representations across generative architectures remains unresolved.The paper motivates examining architectures beyond diffusion because GANs and autoregressive models can also produce high-quality images.Source: §1 · p. 2
  • LoRA offers a general recovery approach across architectures with minimal parameter updates and labeled data, while other methods could also meet these criteria.The authors use limited fine-tuning and data to reduce the chance of introducing new knowledge during recovery.Source: §1 · p. 2
  • Across GAN, autoregressive, and diffusion models, LoRA recovers intrinsic images with the original output head, and better generative models tend to yield more accurate intrinsics.The authors describe this as a possible alternative to improving models solely by scaling data and parameters.Source: §1 · p. 3, Table 1 · p. 3, Figure 2 · p. 3

2 RELATED WORK

Related work studies generative models as sources of scene knowledge, but often examines particular architectures or uses them for downstream tasks. The paper positions its approach as architecture-spanning recovery of intrinsic representations without a specialized decoder or full-model retraining.

2 RELATED WORK

  • Prior work explores GAN interpretability and intrinsics, diffusion geometry and shadows, and autoregressive image synthesis, but the latter models’ intrinsic representations remain uninvestigated.Source: §2 · pp. 3–4
  • Image-based intrinsic recovery has used labeled data, while albedo and shading labels are scarce; these methods do not recover intrinsics from generative models.Source: §2 · p. 4
  • Other work uses generative models as feature extractors, fine-tunes them, or adds decoders; this paper instead targets intrinsic knowledge already captured by the models.Source: §2 · p. 4
  • Studies report 3D face information and hierarchical semantics in StyleGAN, intrinsic representations and depth in diffusion activations, and emergent correspondences in image diffusion models.Source: §2 · p. 4
  • For SD v1-5, the paper reports LoRA uses 1.59M parameters versus 8.5M for Luo et al.’s readout networks, approximately five times greater parameter efficiency.The cited readout work targets generation control, whereas this paper investigates encoded intrinsic knowledge.Source: §2 · p. 4
  • A concurrent LoRA-like approach focuses on dense prediction in restricted domains and does not cover the broad model range or diverse real-image contexts studied here.Source: §2 · p. 4

3 APPROACH

The approach adapts a generative model with a small number of parameters to predict intrinsic maps from labeled examples or pseudo-ground-truth targets. It uses the generator’s existing image-producing architecture rather than a specialized decoder, with architecture-specific LoRA placements and a single-step diffusion path.

3 APPROACH

  • A small parameter set adapts the generator to produce image-like maps with up to three channels, such as surface normals, using the same architecture as image generation.Source: §3 · pp. 4–5
  • Training minimizes the distance between adapted-model outputs and pseudo-ground-truth intrinsic predictions on generated samples.The distance metric depends on the intrinsic being learned.Source: §3 · pp. 4–5
  • For diffusion, the model takes a real or generated image as input and predicts its intrinsic map directly, rather than iteratively denoising.The training objective compares the adapted model’s output on image x with the predictor’s intrinsic estimate for x.Source: §3 · p. 5, Figure 3 · p. 5
  • LoRA adds a low-rank weight update, factored into smaller matrices, to the original projection; its update matrix is initialized to preserve model behavior.Source: §3 · p. 5
  • LoRA is placed in attention layers for diffusion, GAN affine layers, and VQGAN decoder attention layers; distinct adapters target different intrinsics.For diffusion, the UNet acts as a dense predictor, with textual input specifying the intrinsic of interest.Source: §3 · p. 5, Figure 3 · p. 5
  • The illustrations show recovered intrinsics for VQGAN, StyleGAN-v2, and StyleGAN-XL trained on FFHQ, and StyleGAN-v2 trained on LSUN bedrooms.The FFHQ comparison includes synthetic images, four intrinsic predictions from a non-generative model, and LoRA-recovered intrinsics.Source: Figure 4 · p. 6, Figure 5 · p. 6

4 EXPERIMENTS

The experiments test LoRA-based intrinsic recovery across generator types, with small parameter and data requirements, and examine how recovery quality relates to generator quality. They also compare LoRA with alternative recovery methods.

4.1 FINDING 1: INTRINSIC IMAGES ARE ENCODED ACROSS GENERATIVE MODELS, AND LORA IS A GENERAL APPROACH FOR RECOVERING THEM

  • LoRA recovers intrinsic images from almost all tested models, with StyleGAN-XL on ImageNet the notable qualitative failure.The study covers GAN, autoregressive, and diffusion models; generated-image comparisons use pseudo-ground truths, so results depend on the selected targets.Source: §4.1 · p. 7
  • On DIODE real images, single-step Stable Diffusion UNet recovery matches or exceeds Omnidata and ZoeDepth on several metrics, including normal median error and depth RMSE.It uses pseudo-ground-truth models as its training signal, but evaluation is against actual DIODE ground truth.Source: §4.1 · p. 8, Table 3 · p. 8
  • DINOv2 also yields quantitative recovery results comparable to Stable Diffusion with a 0.26% parameter increase, though its intrinsics show visible discontinuities.This extends the recovery approach to a self-supervised, non-generative model.Source: §4.1 · p. 8

4.2 FINDING 2: TINY NEW PARAMETERS & DATA ARE ENOUGH FOR INTRINSIC RECOVERY

  • Rank 8 gives the highest normal-recovery accuracy, while rank 2 adds 0.4M parameters, or 0.04%, and still performs well.Across models, rank 8 adapters add 0.17% to 0.57% of parameters.Source: §4.2 · p. 8, Figure 8 · p. 8

4.3 FINDING 3: BETTER THE GENERATOR BETTER IS INTRINSIC IMAGE RECOVERY

  • Higher-quality generators correlate with more accurate recovered intrinsics across Stable Diffusion versions and face generators, including in the randomly initialized UNet control.The control performs poorly, while Stable Diffusion improvements and lower FID scores in the face-generator experiments align with better intrinsic predictions.Source: Figure 10 · p. 9, §4.3 · p. 9

4.2 FINDING 2: TINY NEW PARAMETERS & DATA ARE ENOUGH FOR INTRINSIC RECOVERY

  • Credible surface-normal predictions appear with 250 labeled samples, while peak performance uses 4,000 training examples.The figure compares these results with a supervised model trained on 12M+ labeled samples.Source: Figure 9 · p. 8, §4.2 · p. 9

4.4 FINDING 4: LORA RECOVERS BETTER INTRINSIC IMAGES THAN OTHER APPROACHES

  • With 250 to 16,000 training samples, LoRA significantly outperforms linear probing and full-model fine-tuning in low-data regimes.The methods use the same number of epochs and have converged; Table 4 summarizes comparisons across sample sizes.Source: §4.4 · p. 9, Table 4 · p. 9

5 TOWARDS IMPROVED INTRINSIC IMAGES RECOVERY

Multi-step diffusion can sharpen recovered intrinsics but introduces input misalignment and color shifts. The proposed SDAUG modification reduces these issues, though its quantitative results remain below the single-step SD-UNet approach.

5 TOWARDS IMPROVED INTRINSIC IMAGES RECOVERY

  • Naive multi-step diffusion improves sharpness but yields intrinsics misaligned with the input and shifts output color relative to ground truth.The figure contrasts these outputs with SDAUG’s corrected layout and shows detailed recovery for domain-specific bedroom training.Source: §5 · pp. 9–10, Figure 12 · p. 10
  • SDAUG resolves input misalignment and significantly reduces color shifts, producing sharp intrinsic images with improved quantitative accuracy, but still underperforms single-step SD-UNet quantitatively.The method augments UNet noise input with the input image’s latent encoding and uses SDv2-1 with the prior learning protocol.Source: §5 · p. 10
  • SDAUG uses an input-image latent encoding in the UNet noise input to address misalignment, and replaces SDv1-5 with SDv2-1 for the Zero SNR strategy.The Zero SNR strategy requires a v-prediction objective, which SDv1-5 lacks and SDv2-1 employs.Source: §5 · p. 10

6 DISCUSSIONS AND LIMITATIONS

The paper reports that generative models implicitly learn intrinsic images and that more powerful generators produce more accurate intrinsic images. It also identifies unresolved questions about how generators use this information and how much adaptation recovery requires.

6 DISCUSSIONS AND LIMITATIONS

  • More powerful generative models produce more accurate intrinsic images, supporting the paper’s hypothesis that intrinsic knowledge is a byproduct of image-generation training.The reported relation is positive correlation, not a demonstrated causal mechanism.Source: §6 · pp. 10–11
  • How generators use intrinsic information when producing images remains ambiguous, and the framework may admit further parameter and label reductions or a parameter-free alternative.The paper also reports that SDAUG produces sharper results but remains quantitatively behind its single-step counterpart.Source: §6 · p. 11

A ADDITIONAL ABLATION STUDIES

The ablations find little quality change beyond 10 diffusion steps, while 25 steps are more stable across intrinsics; CFG=3.0 gives the best overall quality and minimizes color-shift artifacts.

A.1 NUMBER OF DIFFUSION STEPS

  • 25 diffusion steps are used in the main experiments because they are more stable across different intrinsics.Source: Figure 14 · p. 16, §A.1 · p. 16
  • 10 steps are sufficient for better surface normals, with no significant quality gain from increasing the step count.The experiments use 25 steps because results are more stable across different intrinsics.Source: §A.1 · p. 16, Figure 14 · p. 16

A.2 CFG SCALES

  • CFG=3.0 produces the best overall quality and minimizes color-shift artifacts in multi-step SDAUG.Source: §A.2 · p. 16

B OTHER ABLATIONS AND BASELINES

The ablations favor applying LoRA to both cross- and self-attention, while alternatives and direct SDEdit produce poor or artifact-ridden intrinsic predictions. The section also reports a comparison with Bhattad et al. and lists the hyperparameters used.

B OTHER ABLATIONS AND BASELINES

C HYPER-PARAMETERS

D GENERATED IMAGES USED FOR QUANTITATIVE ANALYSIS

The quantitative analysis uses synthetic images from GANs, autoregressive models, and Stable Diffusion, with pseudo-ground truth produced by off-the-shelf models.

D GENERATED IMAGES USED FOR QUANTITATIVE ANALYSIS

E ADDITIONAL QUALITATIVE RESULTS

The appendix adds qualitative examples of recovered intrinsics across SDAUG variants, datasets, and generator architectures.

E ADDITIONAL QUALITATIVE RESULTS

F RESULTS ON 10242 SYNTHETIC IMAGES

SDAUG models trained on 512^2 DIODE images successfully extract intrinsics from unseen 1024^2 synthetic images generated by Stable Diffusion XL. Additional examples span other generators and datasets, and improved diffusion techniques reduce color-shift artifacts.

F RESULTS ON 10242 SYNTHETIC IMAGES

  • Multi-step SDAUG models trained exclusively on 512^2 DIODE images successfully extracted intrinsic images from unseen 1024^2 high-resolution synthetic images generated by Stable Diffusion XL, as shown across Figures 25–34.Figure 25 shows the original image at left, the extracted result (“ours”) in the middle, and pseudo ground truth at right.Source: F RESULTS ON 10242 SYNTHETIC IMAGES · pp. 18–19, Figure 25 · p. 23
  • SDAUG significantly reduces color-shift artifacts observed in SD1-5AUG when extracting detailed scene intrinsics.The reported improvement uses enhanced diffusion techniques.Source: Figure 20 · p. 20
  • Additional examples cover VQGAN and StyleGAN variants trained on FFHQ, StyleGAN-v2 on LSUN bedrooms, and single-step Stable Diffusion UNet extraction.These figures show qualitative results across the listed generators and settings.Source: Figure 21 · p. 21, Figure 22 · p. 21, Figure 23 · p. 22
  • StyleGAN-XL trained on ImageNet does not produce image intrinsics; the authors suggest this may reflect its inability to generate high-quality plausible images.The proposed explanation is tentative.Source: Figure 24 · p. 22
Loading 2311.17137v3…