PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision

Ayush Tewari, Tianwei Yin, George Cazenavette, Semon Rezchikov, Joshua B. Tenenbaum, Frédo Durand, William T. Freeman, Vincent Sitzmann

arXiv:2306.11719v2cs.CVcs.GRcs.LG

TL;DR

Reconstructing a 3D scene from one image is ambiguous because many scenes can explain the partial view, and diffusion training is constrained when target signals are never observed. This paper integrates differentiable forward models into conditional diffusion and demonstrates the approach on three computer-vision tasks, including inverse graphics, motion prediction, and GAN inversion.

  • Problem

    Reconstructing a 3D scene from a single image is ill-posed because many scenes can produce the same partial view, while common data-completion methods rely on ground-truth outputs.

    Source: §1 · p. 1

  • Method

    The framework integrates a differentiable forward model with conditional diffusion to learn signal distributions from partial observations without directly observing the signals.

    Source: §1 · pp. 1–2

  • Results

    The approach is demonstrated on three computer-vision tasks: inverse graphics, single-image motion prediction, and GAN inversion.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    In inverse graphics, the method samples real-world 3D scenes consistent with a single image and produces multi-view-consistent novel views with diverse unseen geometry and appearance.

    Source: §5 · p. 10

    Sampling can be expensive, ranging from seconds for the GAN application to around 100 minutes for 360-degree autoregressive sampling for Co3D.

    Source: §3 · p. 19

Abstract

from arXiv · show

Denoising diffusion models are a powerful type of generative models used to capture complex distributions of real-world signals. However, their applicability is limited to scenarios where training samples are readily available, which is not always the case in real-world applications. For example, in inverse graphics, the goal is to generate samples from a distribution of 3D scenes that align with a given image, but ground-truth 3D scenes are unavailable and only 2D images are accessible. To address this limitation, we propose a novel class of denoising diffusion probabilistic models that learn to sample from distributions of signals that are never directly observed. Instead, these signals are measured indirectly through a known differentiable forward model, which produces partial observations of the unknown signal. Our approach involves integrating the forward model directly into the denoising process. This integration effectively connects the generative modeling of observations with the generative modeling of the underlying signals, allowing for end-to-end training of a conditional generative model over signals. During inference, our approach enables sampling from the distribution of underlying signals that are consistent with a given partial observation. We demonstrate the effectiveness of our method on three challenging computer vision tasks. For instance, in the context of inverse graphics, our model enables direct sampling from the distribution of 3D scenes that align with a single 2D input image.

1 Introduction

The paper introduces a diffusion framework that uses differentiable forward models to learn distributions over unobserved signals from partial observations, without paired signal-observation training data. It applies the framework to inverse graphics and other structured inverse problems.

1 Introduction

  • Reconstructing a 3D scene from one image is ill-posed, with infinitely many plausible scenes, while collecting the ground-truth 3D datasets required by conventional completion methods is challenging.The aim is to sample plausible scenes consistent with the image, including unseen parts.Source: §1 · p. 1
  • The framework integrates a differentiable, potentially nonlinear non-invertible forward model with conditional diffusion to learn signal distributions from partial observations without directly observing signals or paired observations.In inverse graphics, training uses images without paired 3D scenes.Source: §1 · pp. 1–2
  • Prior approaches either model insufficiently multimodal distributions, require ground-truth outputs or two-stage scene reconstruction, or model novel views rather than 3D scenes directly –.RenderDiffusion learns from monocular images but is limited to simple distributions such as single objects in canonical orientations.Source: §1 · p. 2
  • For inverse graphics, the method directly learns image-conditional 3D radiance-field generation from 2D images, using a novel 3D-structured denoising step and differentiable rendering for conditioning and the forward model.It yields diverse 3D scenes that can be rendered from novel views with guaranteed multi-view consistency.Source: §1 · p. 2
  • Under natural assumptions, as observations per training signal approach infinity, the model maximizes both observation likelihood and the likelihood of unobserved signals.Source: §1 · p. 2
  • The paper also demonstrates the framework on single-image motion prediction with a warping forward model and GAN inversion with a pretrained StyleGAN generator.The approach is trained end-to-end and avoids first reconstructing a large signal dataset.Source: §1 · p. 2

2 Method

The method learns p(S|O, ϕ) from multiple observations of signals that are never directly observed, by integrating a differentiable forward model into conditional diffusion. Under stated reconstruction and data conditions, its learned conditional signal distribution converges to the true one.

2 Method

  • The approach requires a differentiable forward model and multiple observations per signal, but no direct signal samples or prior distribution over signals.Its goal is to model the conditional distribution of signals given a previously unseen observation and its forward-model parameters.Source: §2 · p. 3

2.1 Background: Denoising Diffusion Probabilistic Models and their Limitation

  • Existing diffusion models cannot learn p(S|O, ϕ) without observed signals, because their training requires adding noise to those signals.Without signals, the forward Markovian noising process cannot be computed.Source: §2.1 · p. 4

2.2 Integrating Denoising Diffusion with Differentiable Forward Models

  • The denoiser predicts an underlying signal from noisy target and context observations, then applies the differentiable forward model to produce the denoised target observation.Training uses context and target observations of the same signal; inference iteratively samples a signal conditioned on the context observation.Source: §2.2 · p. 4
  • The two training losses approximate a total-observation loss that maximizes the likelihood of all possible observations of the signal.The losses use the target observation and a separate novel forward-model parameter.Source: §2.2 · p. 4
  • With infinitely many known observations per signal and full reconstruction from all possible observations, minimizing the stated losses yields the true conditional signal distribution.The proposition states that the learned p(S|O, ϕ) agrees with the true conditional distribution under these conditions.Source: §2.2 · p. 5

3 Prior Work on Latent Variable Models for Inverse Problems

Prior approaches use latent-variable or unconditional generative models with differentiable forward models, or signal-trained diffusion priors for inverse problems. Their limitations include restricted modeling quality or a need for directly supervised signal datasets.

3 Prior Work on Latent Variable Models for Inverse Problems

  • Latent-variable models combined with forward models sample unobserved signals from observations, but have struggled to model complex distributions and have so far been limited to synthetic 3D scenes in inverse graphics.This category includes Variational Autoencoders, normalizing flows, conditional and attentive neural processes.Source: §3 · p. 5
  • GANs and concurrent diffusion models [20] use differentiable forward models for unconditional generation of unobserved signals, but unconditional models are limited to tight distributions and comparable-quality conditional generation has not been demonstrated.Source: §3 · p. 5
  • Signal-trained diffusion priors have been applied to superresolution [21] [22] [23], inpainting [21] [23] [25] and medical imaging [26] through a versatile plug-and-play approach [29], but rely on large datasets of directly supervised signals.They integrate the diffusion model with a forward measurement process according to Bayes’ rule and can adapt to new inverse problems without retraining.Source: §3 · p. 5

4 Applications

The paper applies its framework to three stochastic inverse problems—3D inverse graphics, single-image motion prediction, and GAN inversion—where signals are inferred from indirect observations.

4 Applications

  • The framework is applied to three computer-vision inverse problems: inverse graphics, single-image motion prediction, and GAN inversion.For each, the paper describes the forward model, dataset, baselines, and prior work.Source: §4 · p. 5

4.1 Inverse Graphics

  • For inverse graphics, the method combines differentiable rendering with a novel 3D-structured denoiser to sample plausible scenes consistent with a single image.Training uses at least two image observations and their camera parameters per scene; visible regions are expected to be reconstructed faithfully while unobserved regions are generated plausibly.Source: §4.1 · p. 6

4.1.1 Results

  • Our method outperforms all baselines on Co3D LPIPS, FID, and KID, but has slightly lower PSNR than pixelNeRF.Qualitatively, it samples diverse plausible 3D scenes from one image; autoregressive sampling extends Co3D results to a 360-degree trajectory, while SparseFusion’s score-distillation is costly and produces unnatural-looking results.Source: Figure 3 · p. 6, Figure 4 · p. 7, §4.1.1 · p. 8

4.2 Single-Image Motion Prediction

  • For single-image motion prediction, the model learns diverse possible pixel motions from video-frame pairs without observing motion correspondences; interpolating motion magnitude yields short videos.A deterministic baseline collapses to near-zero motion regardless of the input and fails to learn meaningful image features.Source: §4.2 · pp. 8–9

4.3 GAN Inversion

  • For GAN inversion, the method models diverse StyleGAN2 latents consistent with a visible image patch, training on patch/full-image pairs without supervised image-latent pairs.It achieves significantly better FID [53] and KID [54] than a deterministic baseline, which matches the patch but does not model output diversity.Source: §4.3 · pp. 9–10, Table 1 · p. 9

5 Discussion

The discussion concludes that integrating differentiable forward models with conditional diffusion enables sampling signals from observation-only training, demonstrated across inverse graphics, motion synthesis, and GAN inversion. It also identifies slow 3D sampling, multi-view training requirements, and limited dataset scale as limitations.

Limitations

  • 3D scene sampling is slow because denoising and volume rendering are expensive; inverse graphics training needs multi-view observations, and limited dataset scale restricts out-of-distribution generalization.Source: §5 · p. 10

Conclusion

  • The method integrates differentiable forward models with conditional diffusion to learn signal distributions from observations, enabling single-image-consistent 3D scene sampling with diverse unseen geometry and appearance.The resulting scenes support multi-view-consistent novel-view rendering.Source: §5 · p. 10
  • The approach also learns 2D motion distributions conditioned on a single image and samples GAN-latent-space images consistent with a given patch.Source: §5 · p. 10

1 Proposition · 2 Details of the Method

The supplied outline lists four topics under “2 Details of the Method,” but provides no substantive details about them or the preceding proposition.

2.1 Inverse Graphics

  • The method-details outline includes inverse graphics.Source: §2 · p. 15

2.2 Single-Image Motion Prediction

  • The method-details outline includes single-image motion prediction.Source: §2 · p. 15

2.3 GAN Inversion

  • The method-details outline includes GAN inversion.Source: §2 · p. 15

2.4 Sampling

  • The method-details outline includes sampling.Source: §2 · p. 15

3 Limitations · 1 Proposition

The proposition gives a consistency guarantee: under reconstructibility and infinitely many known observations per signal, minimizing the stated losses recovers the true conditional signal distribution. The model’s observation support is restricted to observations generated by signals.

1 Proposition

  • With infinitely many known observations per signal and reconstructibility from all observations, minimizing Ltrgt_θ + Lnovel yields the true conditional signal distribution.The proof connects the loss minimum to the desired distributional equality.Source: §1 · pp. 15, 17
  • The guarantee assumes the scattering map is injective, so the complete set of possible observations uniquely determines each signal.This assumption is stated for the class of signals under consideration.Source: §1 · p. 16

3 Limitations

  • The model is limited to observations generated by some signal, a consistency property that many light-field approaches do not satisfy.Target observations are slices of total observations produced by predicted signals.Source: §1 · p. 16

1 Proposition

  • Because total observations uniquely identify signals, the distribution over signals can be recovered from the model’s distribution over total observations.Total observations that do not arise from any signal have probability zero in this recovery argument.Source: §1 · p. 16
  • Adding Lnovel with stochastically sampled forward-model parameters makes the training loss enforce the required total-observation likelihood minimum.The argument applies even though the denoised-signal estimate depends only on the context and noised target observations.Source: §1 · pp. 16–17
  • Although predictions may depend on the auxiliary target forward-model parameter, Lnovel makes the learned signal distribution independent of it in the infinite-observation limit.In that limit, the learned distribution agrees with the true distribution, which does not depend on the auxiliary parameter.Source: §1 · p. 17
  • The proof does not require computing Scatter−1, which may be inefficient, to establish agreement with the true signal density.It assumes an inverse scattering map exists but does not need to evaluate it.Source: §1 · p. 17

2 Details of the Method

The method adapts denoising models to inverse graphics, motion prediction, and GAN inversion with task-specific forward models, and uses 50 DDIM denoising steps across applications.

2.1 Inverse Graphics

  • Depth smoothness and distortion losses regularize 3D reconstructions, selecting preferred natural scenes when multiple scenes explain the limited observations.The authors heuristically state that these regularizers make the scene-to-observation map invertible on the impoverished 2D training data.Source: §2.1 · pp. 17–18
  • Inverse-graphics training renders random 24×24 image patches to reduce volume-rendering costs, and uses DiT for the image backbone with time-conditioned pixelNeRF MLPs.Training proceeds on 8 A100 GPUs, first at 64×64 and then fine-tuned at 128×128.Source: §2.1 · p. 18

2.2 Single-Image Motion Prediction

  • For motion prediction, a DiT denoiser predicts a pixel-aligned motion field from concatenated clean context and noisy target images, then SoftSplat warps the context image.The motion field uses edge-aware smoothness regularization equivalent to the depth smoothness loss in Sec. 2.1.Source: §2.2 · p. 18

2.3 GAN Inversion

  • For GAN inversion, a DiT denoiser predicts a latent w code from a masked face patch and noise, then the generator produces the denoised target image.Training uses 64×64 images, while test-time output can retain the GAN’s 256×256 resolution by skipping final downsampling.Source: §2.3 · pp. 18–19

2.4 Sampling

  • Sampling uses 50 DDIM denoising steps for every application; only the RealEstate10k inverse-graphics model uses classifier-free guidance, with weight 2.That model is trained unconditionally on 10% of iterations by zeroing the conditioning image; other models use no classifier-free guidance.Source: §2.4 · p. 19

3 Limitations

The approach has costly sampling and training, limited out-of-distribution generalization, and only preliminary investigation beyond inverse graphics.

3 Limitations

  • Sampling ranges from a few seconds for the GAN application to around 100 minutes for 360-degree autoregressive Co3D sampling.The cost reflects both iterative denoising and rendering 3D reconstructions with volume rendering.Source: §3 · p. 19
  • Training requires substantial memory, preventing training on smaller GPUs.The authors identify making these models easier to train as future work.Source: §3 · p. 19
  • Because the models were not trained on very large-scale datasets, they cannot generalize to out-of-distribution data.Source: §3 · p. 19
  • Applications outside inverse graphics receive only preliminary investigation, although the authors hope the experiments support future exploration.Source: §3 · p. 19

Related papers

Loading 2306.11719v2…