Source-linked AI summary
CodeNeRF: Disentangled Neural Radiance Fields for Object Categories
Wonbong Jang, Lourdes Agapito
TL;DR
The paper addresses novel-view synthesis and reconstruction of unseen object instances from sparse views or a single image, where geometry, appearance, and viewpoint must be represented and controlled. CodeNeRF learns category-level neural radiance fields with separate shape and texture codes, then optimizes those codes and camera pose from one unposed image. It demonstrates one-shot reconstruction, novel-view synthesis, and shape/texture editing on the SRN benchmark and real-world images.
Problem
Novel-view synthesis of unseen objects from sparse views or a single image requires accurate geometry and appearance together with control over viewpoint, shape, and texture.
Method
CodeNeRF learns separate shape and texture embeddings and an MLP radiance field from posed images, then jointly optimizes latent codes and camera pose for a single unposed input image.
Results
CodeNeRF demonstrates single-view reconstruction, novel-view synthesis, and shape/texture editing on the SRN benchmark and real-world images, with performance close to methods using known pose when estimated pose outliers are removed.
Takeaways & Limitations
The disentangled representation provides explicit control over object shape and texture while supporting reconstruction of unseen objects from a single image.
Takeaways & Limitations
The related-work discussion identifies prior representations in which camera pose estimates are required at inference and shape and appearance are entangled.
Abstract
from arXiv · showhide
CodeNeRF is an implicit 3D neural representation that learns the variation of object shapes and textures across a category and can be trained, from a set of posed images, to synthesize novel views of unseen objects. Unlike the original NeRF, which is scene specific, CodeNeRF learns to disentangle shape and texture by learning separate embeddings. At test time, given a single unposed image of an unseen object, CodeNeRF jointly estimates camera viewpoint, and shape and appearance codes via optimization. Unseen objects can be reconstructed from a single image, and then rendered from new viewpoints or their shape and texture edited by varying the latent codes. We conduct experiments on the SRN benchmark, which show that CodeNeRF generalises well to unseen objects and achieves on-par performance with methods that require known camera pose at test time. Our results on real-world images demonstrate that CodeNeRF can bridge the sim-to-real gap. Project page: \url{https://github.com/wayne1123/code-nerf}
1. Introduction
CodeNeRF addresses novel-view synthesis for unseen objects by learning category-level, disentangled shape and appearance representations. It estimates camera pose and latent codes from a single unposed image, enabling reconstruction, rendering, and editing.
- Novel-view synthesis from sparse views or a single image requires accurate 3D geometry and appearance plus control over viewpoint, shape, and texture.
- During training, posed images supervise joint learning of shape and texture codes and an MLP predicting volume density and view-dependent radiance through multi-view photometric consistency.
- At inference, CodeNeRF jointly optimizes camera pose, shape code, and texture code from a single unposed image of an unseen object.
- The disentangled representation supports single-view reconstruction, novel-view synthesis, and explicit shape or texture editing on SRN and real-world images.
- CodeNeRF learns separate shape and texture embeddings with a neural radiance field that represents category-level object variation.Unlike scene-specific NeRF and pose-dependent SRN, it targets object categories and disentangles geometry from appearance.
2. Related Work
Related work includes discrete and continuous neural representations for 3D reconstruction and view synthesis, often requiring 3D supervision, posed inputs, or scene-specific optimization. CodeNeRF is positioned as a 2D-supervised category representation that supports disentangled editing and single-image inference.
- Voxel, mesh, point-cloud, SDF, and occupancy methods provide learned 3D representations but commonly face resolution, surface-detail, preprocessing, or differentiability limitations.
- Differentiable-rendering approaches such as DVR and SRN learn from image-based supervision and enable test-time optimization, while prior limitations include pose requirements or entangled shape and appearance.
- GRAF allows shape and appearance modification through latent embeddings but cannot estimate shape, texture, or camera pose from an input image for single-view reconstruction.
- CodeNeRF combines category-level neural radiance fields with 2D supervision, separate shape and texture representations, and test-time inference from an unposed image.
3. Methodology
CodeNeRF learns a category-level neural radiance field with separate shape and texture embeddings, then optimizes those codes and camera pose from a single unposed image.
- 3. Methodology: CodeNeRF jointly learns a decoder and separate latent embeddings that model shape deformations and appearance variations across an object category.The embeddings are trained from posed images using image-based supervision rather than 3D supervision.
- 3. Methodology: The disentangled field maps positional encodings, shape and texture codes to volume density and RGB color.Its architecture separates geometry processing from appearance processing: the first network predicts density and intermediate features, while the second predicts color.
- 3. Methodology: Volume density depends on the 3D point and shape code, whereas RGB color additionally depends on viewing direction and texture code.This design choice makes shape and texture separate controls for synthesis and editing.
- 3.1. Training CodeNeRF: Training samples 4094 rays per iteration, evaluates 64 points per ray, and optimizes photometric loss plus latent-vector regularization through volume rendering.The training images provide camera intrinsics and poses for constructing the sampled rays.
- 3.2. Inference Optimization: At inference, decoder weights remain fixed while shape and texture codes are optimized from photometric loss, jointly with azimuth, elevation, and radius camera parameters.Camera pose is represented through spherical parameters and optimized using gradients.
- 3.2. Inference Optimization: The disentanglement design outperforms alternative architectures qualitatively: CodeNeRF edits texture without unwanted shape changes, unlike M1 and M2.M1 fails to synthesize the correct texture, while M2 introduces unwanted shape changes.
4. Experimental Evaluation
CodeNeRF is evaluated on synthetic ShapeNet-SRN objects and real Stanford-Car and Pix3D images, including one-shot reconstruction, pose estimation, novel-view synthesis, editing, and mesh reconstruction. Results show competitive synthesis with unknown test-time pose, convergence from distant initializations, and controllable shape and texture manipulation.
- Quantitative Evaluation on SRN Benchmark: The ShapeNet-SRN evaluation uses separate car and chair models trained on 3,514 cars and 6,591 chairs, with one-view and two-view reconstruction tasks.The benchmark provides 251 test images per object on an Archimedeal spiral.
- Quantitative Evaluation on SRN Benchmark: CodeNeRF (GT pose) outperforms PixelNeRF for cars and comes close on chairs, while joint pose-and-code optimization causes little performance degradation.Removing pose-estimation outliers brings performance close to PixelNeRF.
- Quantitative Evaluation on SRN Benchmark: Optimization from initializations far from ground truth converges to good camera-pose and latent-code estimates, although ground-truth pose produces sharper renderings.The reported optimization sequence includes initialization and iterations 5, 10, 50, and 299.
- Qualitative Comparisons: Varying the corresponding latent codes enables independent shape and texture editing while preserving geometric and appearance consistency in occluded regions.The purple-chair comparison shows CodeNeRF synthesizing the correct combination of shape and texture.
- Results on Real World Datasets: On Stanford-Car and Pix3D, single-image optimization jointly estimates camera pose and latent codes, then produces convincing novel-view renderings and preserves distinctive object style after editing.The learned priors complete shape and texture information absent from the input image.
- Results on Real World Datasets: CodeNeRF also supports explicit mesh reconstruction by querying volume densities on a voxel grid, applying marching cubes, and associating colors to mesh vertices.Examples include a real Stanford-Car image and a synthetic ShapeNet car.
5. Conclusion
CodeNeRF learns separate shape and appearance embeddings with an MLP that maps continuous locations and ray directions to density and color. From one input image, it estimates camera pose and latent codes, with experiments demonstrating one-shot reconstruction and controllable synthesis on synthetic and real images.
- CodeNeRF learns disentangled shape and appearance embeddings together with an MLP that predicts density and color from continuous locations and ray directions.
- At inference, CodeNeRF estimates camera pose and latent codes from a single input image.
- Experiments on SRN, Stanford-Cars, and Pix3D demonstrate one-shot reconstruction and full control over the synthesis process.