PaperlayerView exact revision on arXiv

Source-linked AI summary

LOLNeRF: Learn from One Look

Daniel Rebain, Mark Matthews, Kwang Moo Yi, Dmitry Lagun, Andrea Tagliasacchi

arXiv:2111.09996v2cs.CV

TL;DR

Learning 3D structure from images is challenging when only one view per object is available and no depth or geometry supervision is provided. LOLNeRF aligns images approximately to a canonical pose and conditions a shared radiance-field model on learned latent codes. It reports improved novel-view reconstruction and depth-prediction results, with performance on depth below the best unsupervised method but above several supervised and unsupervised alternatives.

  • Problem

    Recovering 3D structure from images is difficult when training data provides only single views and no explicit geometric information.

    Source: §1 · p. 1

  • Method

    LOLNeRF aligns images to an approximate canonical pose and trains an auto-decoder NeRF conditioned on per-image latent codes to reconstruct them.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    LOLNeRF achieves significantly better novel-view reconstruction than π-GAN [9] and outperforms several supervised and unsupervised depth-prediction methods, though not the best-performing unsupervised method [70].

    Source: §4.2 · p. 6, §4.3 · p. 7

  • Takeaways & Limitations

    The experiments show that single-view, in-the-wild datasets can support learning spaces of 3D shape and appearance without full-image discriminators.

    Source: §5.3 · p. 7

    The method depends on external semantic information such as landmarks, and incorrect pose estimates or segmentations can cause failures.

    Source: §5.1 · p. 7

Abstract

from arXiv · show

We present a method for learning a generative 3D model based on neural radiance fields, trained solely from data with only single views of each object. While generating realistic images is no longer a difficult task, producing the corresponding 3D structure such that they can be rendered from different views is non-trivial. We show that, unlike existing methods, one does not need multi-view data to achieve this goal. Specifically, we show that by reconstructing many images aligned to an approximate canonical pose with a single network conditioned on a shared latent space, you can learn a space of radiance fields that models shape and appearance for a class of objects. We demonstrate this by training models to reconstruct object categories using datasets that contain only one view of each subject without depth or geometry information. Our experiments show that we achieve state-of-the-art results in novel view synthesis and high-quality results for monocular depth prediction.

1. Introduction

Recovering 3D geometry from real-world images remains challenging, and existing approaches often rely on multiple views or geometric supervision. LOLNeRF learns object-class radiance fields from single-view images using approximate canonical alignment and a shared latent space, with pixelwise training that scales to arbitrary image sizes.

1. Introduction

  • 3D geometry recovery remains challenging, while prior work has relied on multiple views or geometric supervision such as depth or point clouds.NeRF methods focused on learning a single scene from multiple views.Source: §1 · p. 1
  • Existing NeRFs can collapse to flat scene representations without multi-view supervision, while GAN-based alternatives use discriminators to enforce multi-view consistency.Multiple-view data is described as hard to acquire.Source: §1 · p. 1
  • Single-view images aligned to an approximate canonical pose can train a shared generative NeRF without adversarial supervision, provided approximate camera poses are available.The method uses predicted 2D landmarks for alignment and an auto-decoder framework for image-specific latent codes.Source: §1 · p. 2
  • Pixelwise reconstruction lets training use arbitrary image sizes without increasing memory requirements, unlike GAN discriminator supervision on inter-pixel relationships.The method need not render entire images or patches during training.Source: §1 · p. 2
  • The approach targets category-level shape and appearance, including geometry prediction without geometric supervision and reconstruction of held-out images and novel views.Separate foreground and background models address inconsistent backgrounds; a solid-surface prior further improves predicted shapes.Source: §1 · p. 2

2. Related Work

Related work spans geometric mesh models, neural implicit representations, single-view shape estimation, and GAN-based generative shape spaces. LOLNeRF builds on radiance fields and auto-decoders while reconstructing images directly rather than relying on discriminator-based training.

2. Related Work

  • Classic 3D morphable models often use accurate scans or depth sensors, but mesh representations reproduce phenomena such as hair poorly for novel-view synthesis.LOLNeRF instead derives volumetric representations from images taken in uncontrolled environments.Source: §2 · p. 2
  • Neural implicit representations include distance, occupancy, scene-feature, and light fields; NeRFs use density and radiance fields to learn 3D scene structure from images.Extensions include foreground-background decompositions, camera co-optimization, and hard-surface priors.Source: §2 · p. 2
  • Single-view shape estimation has used structural priors as supervision, while Wu et al. [70] infer shape for approximately symmetric object categories.Source: §2 · pp. 2–3
  • GAN-based shape methods [9] [47] [57] enforce an implicit multi-view constraint through realistic renders, but discriminator networks are inefficient with volumetric representations.LOLNeRF reconstructs images directly using stochastic sampling.Source: §2 · p. 3
  • Auto-decoders learn latent codes directly without an encoder or discriminator, with a decoder mapping each code to an output.The codes are co-optimized with model parameters.Source: §2 · p. 3
  • Figure 4 contrasts π-GAN [9] using a sampled training-distribution code with LOLNeRF using a learned code that reconstructs a training image.The caption attributes LOLNeRF's sharper detail to higher-resolution training images.Source: Figure 4 · p. 3

3. Method

LOLNeRF extends NeRF with a shared latent space and approximate camera alignment to learn radiance fields from single-view images. Its method combines per-image codes with foreground/background rendering, surface and camera constraints, and latent-code fitting or sampling for generation.

3. Method

  • A shared NeRF is conditioned on per-image latent codes, jointly optimized with the code table using rays associated with each training image.The auto-decoder extends single-scene NeRF without requiring an encoder model.Source: §3 · pp. 3–4

3.1. Foreground-Background Decomposition

  • A lower-capacity background model is composited with the foreground using NeRF transparency; when available, a pre-trained segmenter supplies an additional transparency-consistency loss.For face datasets, the method uses MediaPipe Selfie Segmentation with λmask=1.0.Source: §3.1 · p. 4

3.2. Hard Surfaces

  • Because single-view supervision can blur surfaces along the viewing direction, a mixture-of-Laplacians prior discourages intermediate rendering weights.The prior encourages sparse weights near zero or one; the density constraint depends on sampling density.Source: §3.2 · p. 4

3.3. Camera Parameters

  • For single-view images, 2D landmarks are fit to canonical 3D landmark projections by least squares to estimate rough camera parameters.The method uses a pre-trained network to extract class-consistent 2D landmarks.Source: §3.3 · pp. 4–5

3.4. Conditional Generation

  • To reconstruct an unseen training-set image, a new latent-table row is initialized to the table mean and optimized using the model’s losses.The fitting procedure uses the same optimizer as main-model training.Source: §3.4 · p. 5

3.5. Unconditional Generation

  • For unconditional generation, latent codes are sampled from a Gaussian fit to the learned code table, with truncation controlling the diversity-quality trade-off.The Gaussian mean and covariance are estimated by PCA on the table rows.Source: §3.5 · p. 5

4. Results

LOLNeRF is evaluated on image reconstruction, novel-view synthesis, depth prediction, and high-resolution synthesis. The reported results include better novel-view reconstruction than π-GAN, competitive depth prediction, and improved reproduction of high-resolution detail.

4.1. Image Reconstruction – Table 1 and Table 2

  • On 200 training-set images, LOLNeRF achieves significantly higher reconstruction quality than π-GAN whether trained on FFHQ or CelebA-HQ.The evaluation reports PSNR, SSIM, and LPIPS; π-GAN image fitting uses test-time latent optimization with camera fitting for profile views.Source: Table 1 · p. 5, §4.1 · p. 5
  • Table 2 also evaluates reconstruction on 200 held-out FFHQ images using models trained on CelebA or CelebA-HQ.The section describes this held-out image-fitting evaluation, but the supplied passages do not state its outcome.Source: Table 2 · p. 5, §4.1 · p. 5

4.2. Novel View Synthesis – Table 3

  • LOLNeRF achieves significantly better reconstruction from novel views than π-GAN, indicating a more generalizable face shape space.The evaluation fits a latent representation from one HUMBI view and renders from another view of the same person.Source: §4.2 · p. 6, Table 3 · p. 6

4.3. Depth Prediction

  • The model outperforms several supervised and unsupervised depth-prediction methods, though its score is below the best-performing unsupervised method [70].Depth correlation is evaluated on 3DFAW keypoint locations.Source: §4.3 · p. 7, Table 4 · p. 7

4.4. High Resolution Image Synthesis

  • At 256×256 training resolution, LOLNeRF reproduces high-resolution detail better than π-GAN trained at 128×128, despite both methods supporting “infinite resolution” rendering in theory.The comparison uses high-resolution renders from 512×512 test images.Source: Table 2 · p. 5, §4.4 · p. 7

5. Discussion

The discussion identifies reliance on semantic landmarks and the auto-decoder’s limited ability to maximize image plausibility as limitations. It concludes that LOLNeRF learns 3D shape and appearance from single-view datasets and reports ethical limits on model release.

5.1. Limitations and Future Work

  • The method relies on external semantic information such as landmarks, and incorrect pose or segmentation estimates can cause failures; alignment without prior knowledge remains open.Figure 9 illustrates segmentation errors and camera estimates that distort reconstructions.Source: §5.1 · p. 7, Figure 9 · p. 7
  • The auto-decoder cannot maximize rendered-image plausibility as GANs do, which can result in lost detail; adversarial training is proposed as future work.The proposed augmentation aims to improve perceptual quality for images rendered from novel latent codes.Source: §5.1 · p. 7

5.2. Ethical Considerations

  • The authors will release code for reproducibility but not trained generative models, while acknowledging potential misuse.They describe fairness and privacy as possible socially beneficial applications when models are developed correctly.Source: §5.2 · p. 7

5.3. Conclusions

  • LOLNeRF learns spaces of 3D shape and appearance from single-view datasets and is reported to avoid the high cost of a full-image discriminator and adversarial mode dropping.The conclusion says it learns effectively from unstructured, in-the-wild data.Source: §5.3 · p. 7

A. Qualitative Results – Figures 11, 12

Supplementary qualitative comparisons show novel views at high angles and high-resolution renders. The camera-fitting procedure estimates camera parameters by minimizing landmark reprojection error, with dataset-specific choices for intrinsics and canonical positions.

B. Camera Fitting Procedure

  • The camera fit minimizes reprojection error between predicted 2D landmarks and canonical 3D positions, estimating extrinsics and optionally intrinsics.For faces, canonical positions are predetermined; for AFHQ, positions are optimized jointly across images under a symmetry constraint.Source: B. Camera Fitting Procedure · p. 11
  • Camera intrinsics are predicted for human faces, fixed for AFHQ, and taken from the dataset for SRN cars.The authors note that AFHQ landmarks constrain focal length less effectively; semantically consistent car landmarkers exist.Source: B. Camera Fitting Procedure · p. 11

A. Qualitative Results – Figures 11, 12

B. Camera Fitting Procedure

  • Table 5 reports CelebA perceptual image-distribution quality metrics for LOLNeRF and baselines, with HoloGAN and GRAF results taken from [9].Source: Table 5 · p. 11

C. Dataset Size Ablation – Figure 13

The dataset-size ablation shows a trade-off between training-image reconstruction and the quality of learned 3D structure: more training images improve the consistency and geometric reasonableness of novel views.

C. Dataset Size Ablation – Figure 13

  • As dataset size increases, training-image reconstruction accuracy slowly decreases while novel-view structure becomes more consistent and geometrically reasonable.Very small datasets reconstruct training images accurately but produce unreasonable geometry and novel views.Source: C. Dataset Size Ablation – Figure 13 · p. 11

D. Unconditional Sampling – Figure 14

Unconditional-sample metrics are mixed: the method’s Inception Score is competitive, while FID and KID favor different methods. The figures also examine high-resolution reconstruction and how reconstruction and novel-view renderings change with training-set size.

D. Unconditional Sampling – Figure 14

  • Inception Score is competitive with other 3D-aware GAN methods, while the method does worse in FID but better in KID than π-GAN [9].The authors note that FID may be sensitive to noise and that peripheral noise-like artifacts appear in their generated images.Source: D. Unconditional Sampling – Figure 14 · p. 11
  • High-resolution reconstructions retain sharp details, whereas π-GAN [9] lacks detail beyond its training resolution and shows visible artifacts.Source: Figure 12 · p. 12
  • The dataset-size ablation varies training images from ten to ten thousand, comparing predicted-view reconstructions with novel-view renders using lower-capacity networks.Rows alternate between predicted-view reconstructions and novel-view renders; the ablation uses 256-neuron-per-layer networks.Source: Figure 13 · p. 12

E. Architecture Details

The architecture adds a shared latent code to a standard NeRF backbone, with larger networks and latents for high-resolution models. It also uses a separate background network and produces unconditional samples with method-specific visual differences.

E. Architecture Details

  • The network concatenates a latent code with positional encoding in a standard NeRF backbone [44].Source: E. Architecture Details · p. 12
  • SRN cars and AFHQ use 256-neuron networks with 256-dimensional latents, while CelebA-HQ and FFHQ use 1024 neurons and 2048-dimensional latents.Source: E. Architecture Details · p. 12
  • Both methods generate samples resembling the training distribution; ours has sharper central-face details, while π-GAN [9] has more plausible hair and backgrounds.The caption attributes π-GAN’s hair and background result to its adversarial training.Source: Figure 14 · p. 12
  • The background model is a five-layer, 256-neuron ReLU MLP, and volume rendering uses 128 samples per ray without hierarchical sampling.Source: E. Architecture Details · p. 12

F. Training Details

Training uses batches of pixels sampled across many images rather than rendering whole frames. The reported schedule uses exponential learning-rate decay and takes about 36 hours for high-resolution models.

F. Training Details

  • Each training batch contains 4096 images with 32 pixels per image; with 256² images, the same compute budget would allow only two images per batch for a GAN-based method.Models are trained for 500k iterations.Source: F. Training Details · p. 12
  • The ADAM [31] learning rate decays exponentially from 5 × 10^-4 to 1 × 10^-4, and high-resolution training takes approximately 36 hours on 64 v4 TPU chips.Source: F. Training Details · p. 12

Related papers

Loading 2111.09996v2…