ShaRF: Shape-conditioned Radiance Fields from a Single View

1. Introduction

• Deep learning has enabled estimating 3D scene properties from one image and using implicit representations that avoid some limitations of voxels and textured meshes.The passage contrasts voxel space discretization and the need for accurate geometry in textured meshes with implicit representations.Source: §1 · p. 1
• The method maps a shape code to a voxelized scaffold and conditions a radiance-field network on that scaffold and a separate appearance code.At test time, it optimizes the codes and fine-tunes network parameters so the rendered image matches the input.Source: §1 · p. 1
• The scaffold guides appearance reconstruction toward the object surface, compensating for the unavailable multi-view consistency during single-image inference.The paper notes that empty space around an object provides no useful information for appearance reconstruction.Source: §1 · p. 1
• On ShapeNet-SRN, the method outperforms prior work on PSNR and SSIM except concurrent pixelNeRF, and it generalizes to realistic renderings and Pix3D photographs.On Pix3D, it reports better rendering quality than pixelNeRF; on two ShapeNet classes, its 3D reconstruction performance is on par with Mescheder et al. (2019).Source: §1 · p. 2
• The paper contributes a single-image object-class model, a voxel-conditioned radiance-field representation, and inference-time optimization and fine-tuning strategies.Source: §1 · p. 2

Scene Representation

• Scene representations include point clouds, meshes, voxel grids, implicit surfaces, and learned latent spaces; ShaRF predicts voxel grids as scaffolds for appearance estimation.Voxel grids are naturally processed by CNNs but have resolution limits from memory requirements.Source: §2 · p. 2

Neural Rendering

• Neural rendering uses neural networks in rendering pipelines, while NeRF uses volumetric rendering for realistic novel views; ShaRF conditions a radiance field on voxelized geometry.Geometry-guided synthesis in Riegler and Koltun (2020; 2021) requires several calibrated images for each new scene, unlike ShaRF’s single-view inference.Source: §2 · p. 2

3D Reconstruction

• Learning-based methods reconstruct objects from single RGB images, and ShaRF does so by optimizing shape and appearance codes while fine-tuning its neural renderer.Source: §2 · p. 2

Concurrent Work

• GRF and pixelNeRF use image features to compute radiance fields, whereas ShaRF uses a latent shape scaffold to separate shape and appearance.IBRNet requires multiple views of a scene; the cited DeVries et al. work does not demonstrate radiance-field estimation from one image.Source: §2 · pp. 2–3

Radiance fields

• A radiance field maps a 3D point and viewing direction to color and volume density, providing implicit 3D structure and arbitrary-resolution scene representation.Source: §3 · p. 3
• NeRF estimates the ray integrals by sampling points between the bounds and applying numerical quadrature to estimate ray color.The samples are stratified, and the distance between adjacent ray samples is denoted δk.Source: §3 · p. 3

NeRF background

• The described NeRF setup trains a neural network on multiple calibrated images, and each scene-specific field requires many images for accurate novel-view rendering.A separate model must be trained for every new scene.Source: §3 · p. 3

Rendering a new view

• Rendering a view casts a ray through each pixel and integrates field color and density between scene-dependent near and far bounds.The camera parameters specify the new view, and ray points and direction are derived from that camera geometry.Source: §3 · p. 3

4.1. Generative Neural Rendering

• A shape code generates a voxelized object scaffold, while an appearance code conditions its radiance field; the differentiable pipeline renders the object from new views.Conditioning appearance on the scaffold focuses radiance-field estimation on the object surface and structurally separates shape and appearance codes.Source: §4.1 · p. 4

4.2. Shape Network

• The 128^3 voxel grid provides interpolated occupancy values and surface sampling; shape training combines voxel, symmetry, and silhouette-projection losses.The authors describe the resolution as balancing geometric detail and memory, and note that voxel shapes for real training data can be estimated from multiple views.Source: §4.2 · pp. 4–5

4.3. Appearance Network.

• The appearance network predicts color and density from occupancy, viewing direction, and an appearance code, and is trained on rendered rays with shared per-object codes.Pretraining with ground-truth voxels before fine-tuning with predicted voxels improves rendering quality.Source: §4.3 · p. 5

4.4. Inference on a test image

• At inference, the method fits latent codes to the test image and uses a two-stage procedure to optimize shape and then appearance, without requiring test-time annotations beyond camera parameters.The first stage keeps the appearance network fixed while optimizing shape; the second keeps the resulting shape fixed while optimizing appearance.Source: §4.4 · pp. 5–6
• When test images differ substantially from training data, appearance codes may not capture their colors and predicted shapes may miss thin details, producing blurry renderings.Fine-tuning both networks alongside the latent codes yields more detailed shape reconstructions and more accurate renderings, including for images from a different domain.Source: §4.4 · p. 5
• Fine-tuning the shape network is necessary for accurate reconstruction; optimizing only the shape code produces approximate shapes with missing geometric details.Figure 3 compares shape-code-only optimization with optimization that also fine-tunes the shape network.Source: §4.4 · p. 6, Figure 3 · p. 6
• Fine-tuning the appearance network improves input-image reconstruction, especially outside the training domain; the updated representation renders specified new views using class, symmetry, and scaffold priors.The scaffold conditions appearance estimation to focus on the object's surface.Source: §4.4 · p. 6, Figure 4 · p. 6

5.1. Variants of our method

• The four method variants differ in how they estimate shape and connect it to appearance, ranging from Conditional NeRF without explicit geometry to ShapeFromGT with ground-truth shape.ShapeFromNR fits both latent codes to the test image; ShapeFromMask estimates the scaffold from a segmentation mask before appearance fitting.Source: §5.1 · pp. 6–7

5.2. Novel View Synthesis on ShapeNet-SRN

• On ShapeNet-SRN Chairs, all three variants with a shape scaffold outperform Conditional NeRF; ShapeFromGT performs best, while ShapeFromNR beats ShapeFromMask.The authors attribute ShapeFromNR’s advantage to training and test images having the same distribution.Source: Table 1 · p. 7, §5.2 · p. 8
• ShapeFromNR outperforms previous methods on both ShapeNet-SRN classes and both metrics, is on par with pixelNeRF on Chairs, and is somewhat below it on Cars.The comparison uses numbers from the original papers.Source: Table 2 · p. 7, §5.2 · p. 8

5.3. Novel View Synthesis on ShapeNet-Realistic

• For ShapeNet-Realistic, ShapeFromMask achieves LPIPS 0.109 versus Conditional NeRF’s 0.115, and code-plus-network optimization substantially improves all variants over code-only.ShapeFromMask also outperforms ShapeFromNR in this setting; the authors attribute this to the segmentation mask providing a better shape-reconstruction signal under domain shift.Source: §5.3 · p. 8

5.4. Novel View Synthesis on Pix3D

• ShapeFromMask applied to real Pix3D photographs uses masks as input and a model trained on simple ShapeNet-SRN renderings, demonstrating generalization to a new domain.Source: §5.4 · p. 9

5.5. Comparison to pixelNeRF

• At test time, camera initialization from the closest HOG-matched training render followed by joint optimization yields 13.8° rotation error and 0.14 Blender units translation error; on Pix3D, pixelNeRF renders are blurrier and show thin-structure ghosting and perspective artifacts.The authors note that their canonical-space camera parameters let users specify a novel view directly, unlike pixelNeRF’s relative view-space control.Source: §5.5 · p. 9

5.6. Using symmetry.

• Using symmetry at test time slightly improves ShapeFromMask on ShapeNet-Realistic, from 0.90/23.26 to 0.91/23.51 SSIM/PSNR; Tables 1–3 do not use symmetry.Source: §5.6 · p. 9

5.7. Shape reconstruction

• ShapeFromMask’s shape network performs on par with Occupancy Networks for Chair IoU (0.49 vs 0.50) and better for Car IoU (0.77 vs 0.74), though the comparison is approximate because the method is class-specific.The authors frame this evaluation as an indication of reconstruction quality, while focusing on reconstructions as a means to accurate renderings.Source: §5.7 · p. 9

• The method estimates a radiance field from a single image using explicit and implicit representations, building a geometric scaffold that guides field estimation.Source: §6 · p. 9
• Inverting the generative process recovers explicit and implicit parameters used to synthesize novel views.Source: §6 · p. 9
• The authors report state-of-the-art results on a standard novel-view-synthesis dataset and generalization to images that differ significantly from training data.Source: §6 · p. 9