PaperlayerView exact revision on arXiv

Source-linked AI summary

Pix2NeRF: Unsupervised Conditional $\pi$-GAN for Single Image to Neural Radiance Fields Translation

Shengqu Cai, Anton Obukhov, Dengxin Dai, Luc Van Gool

arXiv:2202.13162v1cs.CVcs.AIcs.GRcs.LG

TL;DR

Single-image NeRF reconstruction is challenging because a single view contains incomplete scene information, while NeRF commonly requires multiple views for training. Pix2NeRF combines an encoder with π-GAN and adversarial and reconstruction objectives to translate a category image into a NeRF without 3D, multi-view, or pose supervision. The authors report novel-view synthesis comparable to state-of-the-art generative NeRF models and superiority over naive GAN inversion.

  • Problem

    Single-image NeRF reconstruction is challenging because one image provides incomplete scene information and conventional NeRF training requires multiple views.

    Source: §1 · pp. 1–2

  • Method

    Pix2NeRF couples an image encoder with π-GAN’s generator and jointly optimizes adversarial and reconstruction objectives to produce a NeRF without 3D, multi-view, or pose supervision.

    Source: §1 · p. 2

  • Results

    The authors report novel-view synthesis comparable to state-of-the-art generative NeRF models and superiority over naive GAN inversion.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Pix2NeRF can also generate novel scenes by sampling content and pose and passing them through the generator.

    Source: §5 · p. 8

    The current setting is limited to one category per dataset and cannot directly generalize beyond the chosen category.

    Source: §5 · p. 8

Abstract

from arXiv · show

We propose a pipeline to generate Neural Radiance Fields~(NeRF) of an object or a scene of a specific class, conditioned on a single input image. This is a challenging task, as training NeRF requires multiple views of the same scene, coupled with corresponding poses, which are hard to obtain. Our method is based on $\pi$-GAN, a generative model for unconditional 3D-aware image synthesis, which maps random latent codes to radiance fields of a class of objects. We jointly optimize (1) the $\pi$-GAN objective to utilize its high-fidelity 3D-aware generation and (2) a carefully designed reconstruction objective. The latter includes an encoder coupled with $\pi$-GAN generator to form an auto-encoder. Unlike previous few-shot NeRF approaches, our pipeline is unsupervised, capable of being trained with independent images without 3D, multi-view, or pose supervision. Applications of our pipeline include 3d avatar generation, object-centric novel view synthesis with a single input image, and 3d-aware super-resolution, to name a few.

1. Introduction

Pix2NeRF targets single-image NeRF reconstruction, where missing scene content and NeRF’s usual multi-view training needs make novel-view synthesis difficult. It combines π-GAN with an encoder and adversarial and reconstruction objectives to produce radiance fields without 3D, multi-view, or pose supervision.

1. Introduction

  • Single-image novel-view synthesis is difficult because one view leaves scene content incomplete, while conventional NeRF training requires multiple views.Prior single-shot approaches may rely on multi-view images during training or inference, and symmetry-based reconstruction does not address the general case.Source: §1 · pp. 1–2
  • Pix2NeRF translates a category-specific input image into a NeRF using an encoder coupled to π-GAN’s generator, without 3D, multi-view, or pose supervision.The pipeline is trained end-to-end on natural images without pretraining, annotation, or fine-tuning.Source: §1 · p. 2
  • The framework jointly trains an encoder and generator with adversarial and reconstruction objectives; adaptation is important because π-GAN inversion alone is insufficient.The encoder maps images to latent codes, while adaptation calibrates the learned representations of the encoder and generator.Source: §1 · p. 2
  • The authors report novel-view synthesis comparable to state-of-the-art generative NeRF models, superiority over naive GAN inversion, and ablations of their design choices.The stated comparison provides no quantitative result in these passages.Source: §1 · p. 2

2. Related works

The related work places Pix2NeRF at the intersection of neural scene representations, NeRF-based generative models, and few-shot NeRF. It identifies single-shot NeRF without additional supervision as an under-explored gap and positions Pix2NeRF as a π-GAN-based approach to it.

Neural scene representations

  • Pix2NeRF is a category-specific 3D-aware novel-view-synthesis method based on NeRF and π-GAN.Source: §2 · p. 2
  • NeRF represents scene color and volume density as a neural function of spatial coordinates and viewing direction, then uses volume rendering to produce views.Source: §2 · p. 2

NeRF-based GANs

  • GRAF and π-GAN generate category-specific radiance fields, with π-GAN conditioning on one latent code and using FiLM SIREN layers.Source: §2 · p. 3

Few-shot NeRF

  • Few-shot methods such as PixelNeRF and GRF condition NeRFs on CNN-extracted pixel-aligned features, while CodeNeRF uses learnable latent codes and test-time optimization.These methods still require multi-view supervision during training.Source: §2 · p. 3
  • Single-shot NeRF without additional supervision remains under-explored; Pix2NeRF adds an auto-encoder to π-GAN, whose NeRF-based design provides stronger 3D consistency than HoloGAN.The paper says merely applying is insufficient for accurate image-to-latent mapping with π-GAN.Source: §2 · p. 3

3. Method

Pix2NeRF combines a π-GAN generator and discriminator with an encoder that separates image content from pose. Its training objectives support generation, inversion, reconstruction, and conditional view realism, with a warm-up and alternating updates.

3. Method

  • Pix2NeRF uses a generator, discriminator, and encoder; the encoder maps images to latent content and pose, which can be varied to render novel views.The method assumes known latent and pose priors representative of real images.Source: §3 · p. 3

3.1. GAN generator objective

  • The generator learns realistic images from sampled latent codes and poses, while pose-prediction MSE encourages generated images to match their sampled poses.The pose objective can help when real-data poses are noisy, such as in CelebA.Source: §3.1 · p. 4

3.2. GAN discriminator objective

  • The discriminator learns to distinguish generated images from real images and predicts pose, with supervision using known generated poses.Source: §3.2 · p. 4

3.3. GAN inversion objective

  • GAN inversion trains the encoder to recover sampled content and pose from generated images, promoting consistency between sampled and encoded latent variables.These objectives alone do not produce a reasonable mapping, motivating additional reconstruction and 3D-consistency objectives.Source: §3.3 · p. 5

3.4. Reconstruction objective

  • Reconstruction encodes a real image and renders it at its predicted pose, using SSIM and VGG perceptual losses alongside reconstruction loss.Source: §3.4 · p. 5

3.5. Conditional adversarial objective

  • Conditional adversarial training renders an encoded real image from random poses to address the risk that single-view reconstruction yields trivial poses or unrealistic other views.Source: Figure 2 · p. 4, §3.5 · p. 5

3.6. Encoder warm-up

  • During the first half of training, the encoder is frozen while the generator learns reconstruction and conditional adversarial objectives; the encoder is then unfrozen.The warm-up is intended to reduce early reconstruction-loss imbalance and help the generator learn correspondence with encoder outputs.Source: §3.6 · p. 5

3.7. Training and Inference

  • Because of GPU memory constraints, objectives are updated in alternating iterations; at inference, one image is encoded and rendered at selected poses, or a latent can be sampled for π-GAN-like synthesis.The discriminator and inversion objectives run every iteration, the generator objective on even iterations, and reconstruction plus conditional adversarial objectives on odd iterations.Source: §3.7 · p. 5

4. Experiments

Pix2NeRF is evaluated on in-the-wild and ShapeNet-SRN data, with quantitative comparisons and qualitative novel-view results. Ablations show how removing components or changing warm-up training affects reconstruction, detail, and 3D consistency.

4.1. Evaluation

  • Conditional synthesis is weaker than π-GAN, especially on CelebA, but is on par with prior 3D view-generation methods.CelebA results are less consistent, which the authors attribute to dataset noise that encourages GANs to converge toward the mean at the expense of variation.Source: §4.1 · p. 7
  • On CARLA and ShapeNet-SRN, the generator outperforms π-GAN on most generative metrics; CelebA results are less consistent amid dataset noise.The authors connect the reduced diversity under conditional sampling to encoded real images occupying a small portion of the latent prior.Source: §4.1 · p. 7
  • Pix2NeRF performs well on in-the-wild CARLA and CelebA views, including extreme poses and artifacts, where cited few-shot NeRF methods cannot train without multi-view supervision.Figure 3 also includes reconstructions and novel views for ShapeNet-SRN chairs.Source: Figure 3 · p. 6, §4.1 · p. 7

4.2. Ablation studies

  • Naive GAN inversion maps images approximately to latent codes, but without joint distillation the reconstruction differs from the input.Using π-GAN as an auto-encoder yields decent reconstruction but visible 3D inconsistency; removing GAN inversion produces blurry, uncanny results.Source: §4.2 · p. 8
  • Removing the conditional adversarial objective makes renderings incomplete and artifact-ridden, while significantly degrading 3D consistency.Source: §4.2 · p. 8
  • Without warm-up, the model overfits the input view; keeping the encoder frozen throughout weakens distillation and reduces fine detail.With lower reconstruction weight instead of warm-up, the reconstruction and adversarial objectives become unbalanced, causing mode collapse in novel-view synthesis.Source: §4.2 · p. 8

5. Conclusions

Pix2NeRF is an unsupervised single-shot framework for translating an input scene image into a NeRF, with reported high reconstruction quality and 3D consistency. Its current setting is limited to one category per dataset, and the authors note potential misuse for misinformation or targeted attacks.

5. Conclusions

  • Pix2NeRF translates a single scene image into a NeRF for novel-view synthesis, using jointly trained encoder and generative NeRF components to represent content and pose.The framework also supports novel scene generation by sampling random content and pose.Source: §5 · p. 8
  • The authors report high reconstruction quality and 3D consistency, on par with or better than previous works.Source: §5 · p. 8
  • The current setting covers one category per dataset and cannot directly generalize beyond the chosen category.The authors identify local conditional fields as a possible direction for generalizing to unseen categories, multiple instances, and real-world scenes.Source: §5 · p. 8
  • Pix2NeRF could be misused to generate content for misinformation or targeted attacks, with deepfake celebrity accounts cited as a growing example.Source: §5 · p. 8

Supplementary Material

The supplementary material includes additional ShapeNet-SRN quantitative results and tables covering ablations and input-view reconstruction alongside novel-view synthesis.

Supplementary Material

  • Table 4 reports additional quantitative results on ShapeNet-SRN, as identified in its caption; the cited passage does not specify metric values, resolutions, or reporting conditions.Source: Table 4 · p. 11
  • Table 5 presents quantitative ablation results on CelebA and ShapeNet-SRN.Source: Table 5 · p. 11
  • Table 6 covers test-set input-view reconstruction and novel-view synthesis using separate metric groups.The caption names PSNR and SSIM for input-view reconstruction and FID, KID × 100, and IS for novel-view synthesis.Source: Table 6 · p. 11

A. Additional qualitative results

Additional qualitative results show Pix2NeRF outputs across CelebA, ShapeNet-SRN chairs, and CARLA.

A. Additional qualitative results

B. Additional quantitative results

Table 4 reports additional quantitative results on ShapeNet-SRN, with generative and reconstruction metrics evaluated at different resolutions; PSNR and SSIM are not reported for CelebA because ground-truth novel views are unavailable.

B. Additional quantitative results

C. Additional ablation study

The additional ablations examine design choices and compare input reconstruction and hybrid latent optimization; naive GAN inversion failed from poor initialization, while input-view optimization produced artifacts for π-GAN.

C. Additional ablation study

  • The ablation study reports FID, KID, and IS on CelebA, plus PSNR and SSIM on ShapeNet-SRN, measured after inference at 64 × 64 resolution.Source: C. Additional ablation study · p. 11
  • Naive GAN inversion failed because the encoder output was not constrained to follow p_z, so the study used a generator warmed up for 300k iterations.Source: C. Additional ablation study · p. 11

D. Input reconstruction and hybrid optimization

  • π-GAN latent optimization used 700 iterations, while optimization initialized from Pix2NeRF used 200 iterations with its generator frozen.The comparison includes input reconstructions and novel views; π-GAN showed strong artifacts and required searching for an optimal iteration count, while Pix2NeRF did not overfit the input view after 1000 optimization iterations.Source: D. Input reconstruction and hybrid… · p. 11

E. Necessity of generator distilling

A frozen pretrained π-GAN generator struggled to capture details without joint fine-tuning. The paper also interpolates latent codes and poses from five input images to produce intermediate views.

E. Necessity of generator distilling

F. Linear interpolation

  • Linear interpolation between predicted latent codes and poses produces intermediate views between input images.The reported interpolation uses five images.Source: F. Linear interpolation · p. 11
  • The CelebA comparison presents input images, reconstructions, and novel-view synthesis results.Source: Figure 5 · p. 12

G. Limitations and failure cases

Pix2NeRF can reconstruct from one image without pose or 3D supervision, but a global latent code can struggle with fine detail, noisy data, and out-of-distribution inputs.

G. Limitations and failure cases

Related papers

Loading 2202.13162v1…