Dense 3D Object Reconstruction from a Single Depth View

1 INTRODUCTION

• Traditional depth-based reconstruction often requires multiple viewing angles, which may be infeasible to capture and costly to process when real-time performance matters.Incomplete scans can leave occluded regions and large holes.Source: §1 · p. 1
• Single-view reconstruction is ambiguous because one depth image can correspond to infinitely many 3D models, while interpolation methods recover only small holes or gaps.The cited approaches include plane fitting, Laplacian hole filling, and Poisson surface estimation.Source: §1 · p. 1
• Earlier single-view deep-learning methods typically produced 32^3 voxel grids, limiting geometric detail, while denser outputs pose a challenge for learning high-resolution geometry.Octree representations were introduced to generate higher-resolution objects efficiently.Source: §1 · p. 1
• 3D-RecGAN++ combines a skip-connected 3D encoder-decoder with conditional adversarial learning to reconstruct fine-grained 3D structure from one 2.5D view.The encoder-decoder estimates a shape, while adversarial training refines it; skip connections preserve high-frequency information.Source: §1 · pp. 1–2
• The method takes an occupancy grid from a single arbitrary depth view without class labels or annotations and predicts a 256^3 voxel grid.The approach is described as end-to-end trainable and general.Source: §1 · p. 2
• The paper reports that 3D-RecGAN++ outperforms the state of the art in single- and multi-category reconstruction, generalizes to unseen categories, and performs robustly on real-world data after synthetic-only training.The authors also report extensive experiments across both single-category and multi-category reconstruction.Source: §1 · p. 2
• The authors collected and released a real-world testing dataset because, to their knowledge, suitable datasets with ground truth for occluded parts and holes were unavailable.Source: §1 · p. 2

2 RELATED WORK

• Geometry-based completion uses fitting, symmetry, or database priors, but large missing regions challenge some methods and database retrieval assumes a similar model is available.The paper notes that such assumptions limit generalization to novel objects or categories.Source: §2 · p. 2
• Multi-view RGB or depth pipelines combine images through feature matching, reprojection, or iterative closest point, while learned occupancy reconstructions are often limited to 32^3 [4].Source: §2 · p. 2
• Single-RGB methods use category templates, multiple-image weak supervision, shape priors, or specialized dense-shape techniques, including Octrees and IDCT.Source: §2 · p. 2
• Earlier single-depth methods generally produced fewer than 40^3 voxels, while later approaches had limitations such as database-dependent upsampling, voxel annotations, inpainting objectives, or coarse primitives.The reviewed methods include approaches designed for denoising, scene completion, or shape inpainting rather than direct full-shape reconstruction.Source: §2 · pp. 2–3
• Although VAEs and GANs have been applied to generate low-resolution 3D structures, adversarial learning for high-resolution shapes remains difficult because of high-dimensional complex distributions and training instability.Source: §2 · p. 3

3.1 Overview

• A 643 partial occupancy view is mapped to an aligned 2563 probabilistic voxel shape, with adversarial refinement during training and the generator alone used at test time.The model learns from paired partial views and complete shapes; training pairs are generated by virtually scanning ShapeNet objects.Source: §3.1 · pp. 3–4

3.2 Architecture

• The generator combines a skip-connected 3D encoder-decoder with an up-sampling module, while a conditional discriminator evaluates paired partial views and generated or real shapes.The up-sampling module raises output resolution to 2563; skip-connections preserve local input structure.Source: §3.2 · p. 4

3.3 Mean Feature for Discriminator

• The discriminator uses a mean vector feature for discrimination, which the authors propose to stabilize adversarial training; they apply WGAN-GP directly rather than balancing separate mean-feature and GAN losses.The cited comparison says Bao et al. use an L2 mean-feature loss alongside Jensen-Shannon loss, requiring hyper-parameter tuning.Source: §3.3 · p. 4

3.4 Objectives

• The generator combines reconstruction and conditional GAN objectives, with a modified cross-entropy loss that penalizes false positives more heavily than false negatives.The reconstruction loss assigns weight α to false positives and 1-α to false negatives and averages over the voxel grid.Source: §3.4 · p. 4

3.5 Training

• Generator and discriminator are trained end-to-end by alternating one discriminator gradient step with one generator step.The authors set λ to 10 for gradient penalty, α to 0.85 for modified cross-entropy, and β to 0.2 for the generator's joint loss.Source: §3.5 · p. 5

3.6 Data Synthesis

• ShapeNet scans provide paired depth-view and complete-shape data, while a Kinect dataset supplies real-world evaluation examples.The synthetic data includes about 26K training pairs per category and separate same-viewing and cross-viewing test sets; the real dataset comprises 20 instances across four classes, with ground truth described as an approximate reconstruction affected by sensor noise, quantization, and SLAM inaccuracy.Source: §3.6 · pp. 5–6, Figure 5 · p. 6

4.1 Metrics

• IoU measures overlap with ground truth, while CE measures prediction confidence; CD and EMD were excluded because high-resolution calculations are costly and downsampling loses geometric detail.Computing CD or EMD between two 256^3 shapes took nearly two minutes on a Titan X GPU.Source: §4.1 · pp. 6–7

4.2 Competing Approaches

• 3D-EPN and Varley et al. outputs were upsampled before scoring, but their lower-resolution training makes this comparison not strictly fair.3D-EPN outputs up to 32^3 and Varley et al. outputs 40^3 grids; both were upsampled to 256^3 for IoU and CE evaluation.Source: §4.2 · p. 7
• The comparison also includes Han et al.'s high-resolution completion network and 3D-RecAE, the proposed generator without its discriminator.Han et al.'s method was not designed for single-depth-view reconstruction and uses LSTMs; 3D-RecAE is used to assess adversarial learning.Source: §4.2 · p. 7

4.3 Single-category Results

• Across four categories, both 3D-RecGAN++ and 3D-RecAE significantly outperform competitors on same- and cross-view 256^3 tests in IoU and CE.Tables 1 and 2 report per-category results for same- and cross-view test sets, respectively.Source: §4.3 · p. 7, Table 1 · p. 7, Table 2 · p. 7
• 3D-RecGAN++ improves IoU and CE modestly over 3D-RecAE; finer details occupy small object parts, as illustrated in Fig. 6a's fourth row.The paper describes this advantage as consistent but not significant across the remaining experiments.Source: §4.3 · p. 7, Figure 6 · p. 8

• Jointly trained 3D-RecGAN++ and 3D-RecAE significantly outperform the state of the art across categories by a large margin.Source: §4.4 · p. 8
• Multi-category training does not notably degrade performance compared with training on individual categories.Source: §4.4 · p. 8
• The authors interpret this result as evidence that the network can learn diverse features across categories.Source: §4.4 · p. 8

• Models trained on bench, chair, couch, and table are tested on car, faucet, firearm, guitar, monitor, and plane, with same- and cross-viewing-angle evaluations.Figure 8 presents qualitative results for the unseen categories under both viewing-angle settings.Source: §4.5 · pp. 8–9, Figure 8 · p. 10
• A separate experiment trains 3D-RecGAN++ on one of four categories and tests it on the other three, with same-viewing-angle results reported on 256^3 voxel grids.Source: §4.5 · pp. 9–10, Table 9 · p. 11
• On six unseen categories, 3D-RecGAN++ achieves higher IoU and smaller CE loss than competing approaches.Source: §4.5 · p. 10
• The authors observe that the bench-trained model tends to generalize better, noting that bench features such as legs, seats, or backs are shared with some other categories.Source: §4.5 · p. 10
• They propose that generalization improves when categories share more low-level geometric features and high-level shape components.Source: §4.5 · p. 10

• Despite noisy and incomplete Kinect inputs, 3D-RecGAN++ reconstructs reasonable dense shapes, while competing methods such as Varley et al. are not robust to real-world noise.Source: §4.6 · p. 11
• Real-world IoU is lower partly because ElasticFusion ground-tr truth objects are less solid than synthetic objects, so dense solid predictions may mismatch in their interiors.Source: §4.6 · p. 11
• Noisy depth views, including views that capture only part of an object, also lead to inferior reconstructions and lower IoU; Table 10 compares real-world IoU and CE loss.Source: §4.6 · p. 11, Table 10 · p. 11

• 3D-RecGAN++ consistently has higher precision than 3D-RecAE, indicating fewer false-positive estimates; Table 11 reports multi-category precision and recall.Source: §4.7 · p. 12, Table 11 · p. 12
• Both 3D-RecGAN++ and 3D-RecAE achieve recall above 0.8, indicating that both estimate major object shapes without many false negatives.Source: §4.7 · p. 12
• The authors describe 3D-RecAE estimates as fatter and bigger, while adversarial learning tends to remove overestimated parts and expose thinner, more detailed shapes.Source: §4.7 · p. 12

4.8 Computation Analysis

• 3D-RecGAN++ has 167.1 million parameters, on a similar scale to VGG-19 at 144 million.Han et al. has the fewest parameters because it shares most parameters across object blocks.Source: §4.8 · p. 12
• Around 40 milliseconds is sufficient for 3D-RecGAN++ to recover a dense 256^3 object.3D-EPN takes the shortest time to predict a 32^3 object; those low-resolution outputs are upsampled separately.Source: §4.8 · p. 12
• Upsampling a low-resolution object to 256^3 takes around 7 seconds on a CPU server.The reported upsampling uses the SciPy library.Source: §4.8 · p. 12
• Table 12 compares model parameter counts and average time to reconstruct a single object.The timing comparison uses the same hardware and software environment for all models.Source: Table 12 · p. 12, §4.8 · p. 12

5 DISCUSSION

• The network takes a volumetric representation of a depth view, so raw depth images require preprocessing.The authors note this preprocessing may be straightforward given depth-camera parameters in scenarios such as robot grasping.Source: §5 · p. 12
• The input depth view must contain clean object information without a cluttered background.Source: §5 · p. 12
• The authors suggest using an existing segmentation algorithm such as Mask-RCNN to isolate the target object instance.Source: §5 · p. 12

6 CONCLUSION

• 3D-RecGAN++ predicts dense, accurate 3D structures with fine details and outperforms the state of the art in single-view shape completion for individual categories.Source: §6 · p. 12
• The network predicts precise shapes across multiple categories without object class labels during training or testing.Source: §6 · p. 12
• The approach also predicts satisfactory 3D structures for unseen categories.Source: §6 · p. 12
• The model, trained purely on synthesized data, infers accurate shapes from noisy real-world data; the authors say this indicates learned general 3D latent features rather than simple fitting to training datasets.The authors also state that adversarial learning adds geometric details to estimated shapes.Source: §6 · p. 12