PaperlayerView exact revision on arXiv

Source-linked AI summary

Single-view 3D Scene Reconstruction with High-fidelity Shape and Texture

Yixin Chen, Junfeng Ni, Nan Jiang, Yaowei Zhang, Yixin Zhu, Siyuan Huang

arXiv:2311.00457v1cs.CV

TL;DR

Single-view reconstruction must recover detailed geometry and appearance from limited image evidence, while existing approaches often overlook texture and fine shape detail. The paper combines implicit shape and radiance fields with a two-stage 3D/2D supervision curriculum; it reports improved reconstruction on 3D-FRONT and Pix3D and supports novel-view rendering.

  • Problem

    Single-view reconstruction must infer scene geometry and appearance from one image, despite uncertainty about visible-region shape and plausible unseen parts.

    Source: §1 · p. 1

  • Method

    The SSR framework predicts shape and color with implicit networks, combining 3D shape supervision and rendered color, depth, and normal images in a two-stage curriculum.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    27.7% and 11.6%: the method outperforms state-of-the-art methods on 3D-FRONT and Pix3D, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The model renders images from novel viewpoints and supports holistic scene understanding and 3D scene editing through object composition.

    Source: §1 · p. 2

    The authors identify potential limitations in reconstructing novel object categories and recovering textures for unseen object parts.

    Source: §5 · p. 8

Abstract

from arXiv · show

Reconstructing detailed 3D scenes from single-view images remains a challenging task due to limitations in existing approaches, which primarily focus on geometric shape recovery, overlooking object appearances and fine shape details. To address these challenges, we propose a novel framework for simultaneous high-fidelity recovery of object shapes and textures from single-view images. Our approach utilizes the proposed Single-view neural implicit Shape and Radiance field (SSR) representations to leverage both explicit 3D shape supervision and volume rendering of color, depth, and surface normal images. To overcome shape-appearance ambiguity under partial observations, we introduce a two-stage learning curriculum incorporating both 3D and 2D supervisions. A distinctive feature of our framework is its ability to generate fine-grained textured meshes while seamlessly integrating rendering capabilities into the single-view 3D reconstruction model. This integration enables not only improved textured 3D object reconstruction by 27.7% and 11.6% on the 3D-FRONT and Pix3D datasets, respectively, but also supports the rendering of images from novel viewpoints. Beyond individual objects, our approach facilitates composing object-level representations into flexible scene representations, thereby enabling applications such as holistic scene understanding and 3D scene editing. We conduct extensive experiments to demonstrate the effectiveness of our method.

1. Introduction

Single-view reconstruction must infer complete geometry and appearance from limited image evidence, while prior approaches often neglected texture and fine geometric detail. The framework combines neural implicit shape and radiance representations with a two-stage curriculum to recover textured objects and support novel-view rendering and scene editing.

1. Introduction

  • Single-view reconstruction must infer accurate visible-region shape and plausible unseen parts from the uncertainty of a single image.The task aims to recover scene geometry and appearance from one monocular image.Source: §1 · p. 1
  • Earlier shape-focused methods often neglected texture and direct textural supervision, limiting fine, instance-specific geometric detail.The introduction notes these limitations can lead to category mean shapes and difficulty generating smooth details.Source: §1 · p. 2
  • The framework uses Single-view neural implicit Shape and Radiance field representations to predict SDF and color values, trained with 3D shape supervision and rendered color, depth, and normal images.It expresses volume density as a function of the SDF.Source: §1 · p. 2
  • A two-stage curriculum balances rendering and reconstruction losses to address shape-appearance ambiguity under partial observations.The curriculum aims to learn a 3D object prior for unseen parts while capturing pixel-level image details.Source: §1 · p. 2
  • 27.7% and 11.6%: the method outperforms state-of-the-art methods on 3D-FRONT and Pix3D, respectively.The introduction does not specify the metric associated with these dataset-level gains.Source: §1 · p. 2
  • The model renders images from novel viewpoints, with depth and normal quality comparable to existing estimators, and supports holistic scene understanding and 3D scene editing.The paper describes composing object-level representations into scenes.Source: §1 · p. 2

2. Related work

Prior single-view reconstruction methods have used boxes, retrieved CAD models, and explicit or implicit shape representations, but often lacked instance-specific geometry or texture detail. This work jointly models shape and texture and coordinates 3D supervision with volume rendering through a two-stage curriculum.

2. Related work

  • Earlier bounding-box and CAD-model approaches lacked instance-specific 3D object geometry, while newer explicit and implicit representations addressed some of these limitations.The paper identifies texture as an important omission for tasks requiring pixel-level detail.Source: §2 · p. 2
  • Generative approaches use 2D priors for novel-view generation and multi-view reconstruction, or 3D-text pairs for conditional generation; this method instead uses 3D and 2D supervision discriminatively.The paper says its joint explicit modeling captures high-fidelity structures and details.Source: §2 · pp. 2–3
  • Implicit representations offer continuous, high-resolution geometry with constant memory usage, but existing single-view methods may miss fine details or higher-order relationships between 3D points and image details.The proposed shape and radiance representation learns geometry and appearance details jointly using shape supervision and volume rendering.Source: §2 · p. 3
  • Rendering-based surface methods can struggle to recover geometry under sparse views because photo-realistic losses admit multiple shape-appearance explanations.The paper introduces a two-stage curriculum that incrementally increases rendering-loss weights to coordinate 3D and 2D supervision.Source: §2 · p. 3

3. Method

The method combines SDF-based implicit surfaces with volume rendering to reconstruct object geometry and appearance from single-view images. It adds rendering supervision in stages and composes object representations into scenes.

3.1. Background

  • An SDF represents geometry as signed distance to the closest surface, with its zero-level set defining the surface; converting SDF values to density enables differentiable volume rendering.Along camera rays, sampled points yield SDF and color predictions used to render color, depth, and surface normals.Source: §3.1 · p. 3

3.2. 3D object reconstruction

  • Pixel-aligned and instance-aligned image features condition an implicit network to predict SDF and geometry features, while a rendering network uses point normals and viewing direction to predict color.The framework overview specifies that SDF is transformed to volume density for rendering.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3. Supervision and learning curriculum

  • Training first uses 3D SDF supervision, then linearly adds RGB, depth, and normal rendering losses to address shape-appearance ambiguity while learning overall shape before fine detail.Depth and normal consistency losses directly supervise SDF prediction without passing through the rendering network; depth, normal, and segmentation are not required at inference.Source: §3.3 · p. 5

3.4. 3D scene composition

  • Object-level implicit representations are composed using 2D and 3D bounding boxes to recover scene geometry and render scene images.The distance-aware integration of samples from different objects along a ray reflects visibility, and the composition supports novel-view synthesis and 3D scene editing.Source: §3.4 · p. 5

4. Experiment

The experiments evaluate object reconstruction on synthetic 3D-FRONT and real Pix3D, then examine rendering and scene-level applications. Results cover reconstruction quality, novel-view rendering, depth and normal estimation, holistic scene understanding, and scene editing.

4.1. Indoor object reconstruction

  • On 3D-FRONT, the model gains 27.7% in mean CD and 16.4% in F-Score, with the best NC across object categories; it also outperforms prior methods on Pix3D.The comparisons include MGN, LIEN, and InstPIFu on 3D-FRONT, and the Pix3D non-overlapped split.Source: §4.1 · p. 5, Table 1 · p. 6, Table 2 · p. 6
  • Color, depth, and normal supervision improve reconstruction details, while the curriculum performs best when 2D loss weights increase after 3D shape-prior learning.Increasing 2D loss weights can harm 3D reconstruction; Stage Two begins at epoch λ0 = 150, outperforming early injection at λ0 = 0 or 70.Source: §4.1 · pp. 5–6, Table 3 · p. 7, Figure 4 · p. 7

4.2. Rendering capability

  • The model renders plausible color, depth, and normal images from novel views, and PixelNeRF struggles outside the vicinity of its input view.The reported qualitative renderings remain plausible and consistent with viewing-angle changes of about 40°; depth and normal maps are also evaluated on novel views.Source: §4.2 · pp. 7–8, Figure 6 · p. 7, Figure 7 · p. 8

4.3. Applications

  • Composing object-level implicit representations supports detailed scene reconstruction with cross-domain generalization on SUNRGB-D and rendering of corresponding color, depth, and normal images.The application uses existing 3D object detectors.Source: §4.3 · p. 8, Figure 8 · p. 8
  • Scene editing supports object translation, rotation, duplication, and composition, producing both 3D geometry and rendered images for edited scenes.Examples include duplicating chairs, rotating a sofa, and composing the edits.Source: §4.3 · p. 8, Figure 9 · p. 8

5. Conclusion

The conclusion presents a single-view reconstruction framework that combines neural implicit shape and radiance representations with color, depth, and normal supervision. It reports improved textured reconstruction and rendering capabilities, while identifying novel categories and unseen-part textures as limitations.

5. Conclusion

  • The framework combines neural implicit shape and radiance fields to reconstruct object shapes and textures from single-view images.The conclusion identifies integrating color, depth, and normal supervision through a designed curriculum as pivotal to improved performance.Source: §5 · p. 8
  • The model demonstrates rendering, single-view depth and normal estimation, holistic scene understanding, and 3D scene editing capabilities.The stated editing applications include translation, rotation, and composition of objects in 3D space.Source: §5 · p. 8
  • Potential limitations include reconstruction of novel object categories and texture recovery for unseen parts; large-scale 2D or 3D priors are proposed as future work.Source: §5 · p. 8

Supplementary Material

The supplementary material covers dataset preparation, implementation and training details, and additional experimental results and failure cases.

Supplementary Material

  • Section S1 describes preparation of the 3D-FRONT and Pix3D training data.Source: Supplementary Material · p. 13
  • Section S2 provides implementation details, including model architecture, learning curriculum, and training strategies.Source: Supplementary Material · p. 13
  • Section S3 reports further experimental details, evaluation metrics, results, and failure cases.The authors also recommend the supplementary video for visualizations and animations.Source: Supplementary Material · p. 13

S1. Training data preparation

The section describes the datasets and preprocessing used for training and evaluation, along with the 3D and monocular-cue supervision prepared for learning.

S1.1. Datasets and splits

  • 3D-FRONT uses about 20K scene images for training and validation and 2,000 images across eight categories for evaluation; Pix3D evaluation uses a split without overlapping objects.Pix3D supplies aligned real-world furniture images and models; 3D-FRONT supplies synthetic indoor scenes with textured models.Source: S1.1. Datasets and splits · p. 13

S1.2. Fixing CAD models

  • Non-watertight 3D-FRONT models are remeshed, and 1,734 beds and sofas are manually repaired to restore missing backs or undersides.These steps address SDF-learning difficulties and incomplete geometry that can confuse object-prior learning.Source: S1.2. Fixing CAD models · p. 13

S1.3. SDF supervision

  • 3D supervision uses SDF targets from a 64-resolution voxelization, with positive values outside the mesh and negative values inside.Training query-point targets are obtained by trilinear interpolation.Source: S1.3. SDF supervision · p. 13

S1.4. Monocular cues rendering

  • Monocular depth, normal, and segmentation cues guide training only; 3D-FRONT cues are rendered from scene data, while Pix3D depth and normals use estimator-generated pseudo-ground truth.Pix3D provides instance segmentation but lacks depth and normal images.Source: S1.4. Monocular cues rendering · p. 13

S2. Technical details

The technical details cover the implicit and rendering networks, a staged balance of shape and image supervision, and the training procedure.

S2.1. Model architecture

  • The implicit network is an 8-layer, 256-hidden-dimension MLP, while the rendering network is a 2-layer MLP with the same hidden dimension.A pretrained ResNet34 encodes the image; the networks use Softplus and Sigmoid activations, respectively.Source: S2.1. Model architecture · p. 13

S2.2. Learning curriculum

  • Training first uses only 3D supervision, then increases color, depth, and normal loss weights while keeping the 3D loss weight constant.The curriculum sets λ0 to 150 near SDF-loss convergence; substantially lower values such as 0 or 70 degrade performance.Source: S2.2. Learning curriculum · pp. 13–14

S2.3. Training strategy

  • Training samples 64 rendering rays per iteration and adds 30,000 near-surface points for 3D supervision; the model runs for 400 epochs on four A100 GPUs.The batch size is 96, and the learning rate decays at epochs 330 and 370.Source: S2.3. Training strategy · p. 14

S3. More experiment details and results

The experiments report reconstruction comparisons, qualitative outcomes, failure cases, baseline protocols, and novel-view rendering results.

S3.1. Evaluation metrics

  • Evaluation uses CD, F-Score, and normal consistency for shape reconstruction, and L1 depth error plus L1 and angular errors for normals.CD and F-Score are computed after mesh alignment; normal consistency measures agreement between corresponding surface normals.Source: S3.1. Evaluation metrics · p. 14

S3.2.1 Experiments on 3D-FRONT and Pix3D

  • Qualitative results show finer, smoother surfaces with high-fidelity textures, and reproduced results are reported as comparable to the original results.The section presents additional examples for both 3D-FRONT and Pix3D.Source: S3.2.1 Experiments on 3D-FRONT and Pix3D · p. 14

S3.2.2 Failure cases

  • Occlusions distort recovered shape and appearance in hidden areas, while intricate thin geometry challenges the SDF representation.The examples include occlusion and intricate geometry; the authors suggest unsigned distance fields with volume rendering as a future direction.Source: S3.2.2 Failure cases · pp. 14–15, Figure S1 · p. 15

S3.2.1 Experiments on 3D-FRONT and Pix3D

  • On 3D-FRONT, the model achieves best mean CD and F-Score and best NC across categories; on Pix3D, it significantly outperforms state-of-the-art methods.The 3D-FRONT comparison names MGN, LIEN, and InstPIFu; the Pix3D results use the non-overlapped split.Source: Table S1 · p. 14, Table S3 · p. 15

S3.2.3 Comparison with prior-guided models

  • The prior-guided comparison uses 100 3D-FRONT test samples; Zero-1-to-3 and Shap-E receive segmented, background-free images, while this model uses original images.Shap-E also receives the ground-truth object category as its text prompt.Source: S3.2.3 Comparison with prior-guided… · p. 15

S3.3.1 Novel view synthesis

Related papers

Loading 2311.00457v1…