PaperlayerView exact revision on arXiv

Source-linked AI summary

Matryoshka Networks: Predicting 3D Geometry via Nested Shape Layers

Stephan R. Richter, Stefan Roth

arXiv:1804.10975v1cs.CV

TL;DR

Single-image 3D reconstruction is challenged by volumetric memory limits, octree complexity, and view-based methods’ inability to handle large self-occlusions. This paper predicts voxel tubes as a 2D task and scales reconstruction with nested shape layers. The proposed networks outperform voxel-based and octree-based approaches in the reported settings, and support shape reconstruction from IDs or similarity and shape sampling.

  • Problem

    Volumetric decoders face memory limits at high resolutions, octrees require sophisticated structures and feature propagation, and view-based methods cannot handle large self-occlusions.

    Source: §1 · pp. 1–2

  • Method

    The paper predicts whole voxel tubes at reference-view pixels using 2D networks, then scales to high resolutions with nested shape layers represented by six depth maps per layer.

    Source: §1 · p. 2, §3.2 · p. 3, §3.3 · p. 3

  • Results

    The proposed networks outperform dense voxel-based approaches at low resolutions and octree-based decoder architectures at high resolutions, despite using standard network layers.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Applications to shape reconstruction from IDs or shape similarity and to shape sampling demonstrate the approach’s broad applicability.

    Source: §1 · p. 2, §5 · p. 8

    Experiments fix the maximum number of shape components; predicting that number recursively for arbitrarily complex objects without retraining remains future work.

    Source: §5 · p. 8

Abstract

from arXiv · show

In this paper, we develop novel, efficient 2D encodings for 3D geometry, which enable reconstructing full 3D shapes from a single image at high resolution. The key idea is to pose 3D shape reconstruction as a 2D prediction problem. To that end, we first develop a simple baseline network that predicts entire voxel tubes at each pixel of a reference view. By leveraging well-proven architectures for 2D pixel-prediction tasks, we attain state-of-the-art results, clearly outperforming purely voxel-based approaches. We scale this baseline to higher resolutions by proposing a memory-efficient shape encoding, which recursively decomposes a 3D shape into nested shape layers, similar to the pieces of a Matryoshka doll. This allows reconstructing highly detailed shapes with complex topology, as demonstrated in extensive experiments; we clearly outperform previous octree-based approaches despite having a much simpler architecture using standard network components. Our Matryoshka networks further enable reconstructing shapes from IDs or shape similarity, as well as shape sampling.

1. Introduction

The introduction frames single-image 3D reconstruction as a way to infer detailed surfaces despite missing stereo and motion cues. It proposes 2D voxel-tube prediction and nested shape layers to address resolution, memory, and representation challenges.

1. Introduction

  • Single-image shape reconstruction aims to infer detailed surfaces or plausibly complete hidden parts despite the absence of stereo or motion cues.Source: §1 · p. 1
  • Volumetric decoders face GPU-memory limits at high resolutions, while octree encodings require sophisticated structures and feature-propagation mechanisms.View-based reconstructions can preserve detail but cannot represent shapes with significant self-occlusion.Source: §1 · pp. 1–2
  • The voxel-tube baseline casts 3D occupancy reconstruction as 2D prediction and attains state-of-the-art accuracy, outperforming purely voxel-based approaches.It predicts whole voxel tubes at every pixel of a reference view and uses a structured loss.Source: §1 · p. 2
  • Nested shape layers enable detailed reconstructions with complex topology, including self-occlusions, using a memory-efficient encoding represented by six depth maps per layer.The approach outperforms octree-based architectures despite using a simpler architecture, and supports reconstruction from IDs or shape similarity as well as shape sampling.Source: §1 · p. 2

2. Prior Work

Prior work represents shapes as voxel occupancy, sparse octrees, point clouds, or view-based images, each with different resolution, memory, topology, or fusion challenges. Other approaches use geometry images, volumetric primitives, and depth-based representations.

2. Prior Work

  • Voxel occupancy maps make shape estimation a voxel-classification task, but memory grows cubically with resolution and output is usually limited to 32 voxels per side.Source: §2 · p. 2
  • Octree methods address high-resolution memory demands, but may require a known tree or custom layers, and sparse structure complicates feature propagation [22] [27].The proposed method instead uses standard network layers and 2D convolutions.Source: §2 · p. 2
  • Point-cloud prediction addresses sparsity in 3D volumes and shows low-resolution results, but its scalability to higher resolutions remains unestablished [8].Source: §2 · p. 2
  • View-based methods can generate high-resolution shapes but may produce noisy estimates requiring fusion and cannot handle large self-occlusions [18] [26] [17].Prior approaches fuse views through point-cloud merging, registration and optimization, or differentiable depth-map rendering.Source: §2 · p. 2
  • Geometry images are memory-efficient but require topology handling and yield nonuniform detail from projection; nested layers generalize n-layer heightmaps as six fused overlapping heightmaps [24].Volumetric primitives [35] and depth peeling also inform alternative representations.Source: §2 · pp. 2–3
  • Concurrent work predicts multi-channel depth maps to reconstruct aligned shapes from sketches, exploiting additional projective constraints.Source: §2 · p. 3

3. Formulation

The formulation replaces dense voxel decoding with voxel-tube image prediction, then compresses shapes into fused depth-map layers that can be nested to represent occluded geometry. It also describes losses and implementation choices for these representations.

3.1. Standard voxel decoder

  • A standard voxel decoder progressively upsamples a shape code while reducing features until it produces a dense binary occupancy map.The decoder transforms a four-dimensional tensor into a three-dimensional occupancy tensor.Source: §3.1 · p. 3

3.2. Predicting voxel tubes

  • Voxel tube prediction shares features across a tube of voxels, producing an image-like tensor that can use standard 2D network components.The decoder upsamples spatial resolution while reducing features until the feature count equals the output resolution.Source: §3.2 · p. 3, Figure 2 · p. 3

3.3. Shape layers: Learning to compress voxel tubes

  • Each shape layer uses six depth maps; fusing the three axis-view shapes by intersection removes outliers caused by noisy predictions near silhouettes.A single axis-view shape cannot recover geometry such as the car wheels because of self-occlusion.Source: §3.3 · pp. 3–4, Figure 3 · p. 4

3.4. Nested shape layers: Recovering occluded parts

  • Nested layers add and subtract fused depth-map shapes in turn, allowing complex geometries and parts hidden from all three view axes to be represented.The encoding is trained jointly by a single network; 94.8% of ShapeNet-core shapes at 32^3 fit one layer, compared with only 2.6% of ShapeNet-cars shapes at 128^3.Source: §3.4 · pp. 4–5

3.5. Loss functions for dense and sparse prediction

  • For shape-layer prediction, the method uses a robust L1 loss that separately averages foreground and background losses and allows background predictions below a margin.A modified L2 version yielded significantly worse reconstructions.Source: §3.5 · p. 5

3.6. Implementation details

  • At 32^3 resolution, the networks outperform voxel-decoder baselines and are competitive with the more complex PSGN on ShapeNet-core.Table 2 reports mean IoU (%) per category and the average IoU over all categories.Source: Table 2 · p. 6
  • The encoder uses convolutional and residual modules to downsample the image, while residual decoder modules upsample to the desired output resolution.The experiments use Adam with an initial learning rate of 0.001.Source: §3.6 · p. 6, Figure 5 · p. 6

4. Evaluation

Across ShapeNet-core evaluations, the paper compares single-image reconstruction, architecture and loss choices, and applications beyond image input. Its networks outperform voxel-decoder baselines at low resolution and octree-based approaches at high resolution, while also supporting shape generation and other input modes.

4.1. Reconstruction from a single view

  • The dense voxel-tube network outperforms voxel-decoder approaches and is on par with PSGN, while the sparse Matryoshka version performs slightly worse than the dense version but beats voxel-decoder baselines.The comparison uses ShapeNet-core single-image reconstruction.Source: §4.1 · p. 6
  • At all tested resolutions, Matryoshka networks outperform the octree-based approach; OGN saturates at 128^3 for cars, while Matryoshka saturates only for the more complex table category.The comparison evaluates predictions at 32^3, 64^3, 128^3, and 256^3 against ground truth at 256^3; airplane shapes show the biggest relative improvements at higher resolutions.Source: §4.1 · p. 7, Table 3 · p. 7

4.2. Ablation studies

  • Across categories, the ResNet-inspired architecture outperforms the other tested architectures by a significant margin.Architectures were constrained to fit within 3 GB of GPU memory for 32^3 predictions with mini-batches of 128.Source: §4.2 · p. 7, Table 4 · p. 7
  • The two proposed losses perform best across categories, while binary cross entropy is a strong baseline and L1 performs worst.Because the evaluated architecture constrains final-layer activations between 0 and 1, the authors report that a robust loss is less important.Source: §4.2 · p. 7, Table 5 · p. 7

4.3. Other applications

  • On Blendswap shape-from-ID reconstruction at 512^3, Matryoshka reaches 97.8% IoU, compared with OGN’s 96.9%.The Matryoshka result uses a 2D representation alone.Source: §4.3 · p. 8
  • From shape-similarity descriptors, the network reaches 81.1% mean IoU on held-out car shapes and can interpolate shapes or synthesize diverse samples from random noise.The descriptors are derived from pairwise 32^3 shape IoUs and reduced to 2424 dimensions; the network generates shapes at 128^3 resolution.Source: §4.3 · p. 8, Figure 7 · p. 8, Figure 6 · p. 8
  • The paper also presents a qualitative real-image reconstruction and reports a supplementary study of reconstruction from a single silhouette.The real-image example is shown in Figure 8.Source: §4.3 · p. 8, Figure 8 · p. 8

5. Conclusion

The conclusion frames 3D reconstruction as 2D prediction, using nested shape layers to extend the approach to high-resolution shapes. It also reports applications to shape IDs, shape similarity, and sampling, while noting a fixed component-count limitation.

5. Conclusion

  • The 2D prediction formulation leverages established 2D pixel-prediction architectures, and nested shape layers extend the approach to high-resolution reconstruction.The paper reports outperforming dense voxel-based approaches at low resolution and octree-based decoders at high resolution.Source: §5 · p. 8
  • The demonstrated applications include reconstruction from shape IDs and shape similarity, as well as shape sampling.The conclusion presents these as evidence of the approach’s broad applicability.Source: §5 · p. 8
  • Experiments fix the maximum number of shape components; handling arbitrarily complex objects without retraining would require predicting component counts per object recursively, which is left for future work.Learning shape fusion is also left for future work.Source: §5 · p. 8

A. Network Architecture

The architecture uses residual modules with consistent kernel size and stride, adapting feature channels and spatial resolution through down- and up-sampling. A final 1×1 convolution produces the output channels required by each network type.

A. Network Architecture

  • Downsampling drops every other pixel and zero-pads added feature channels, while upsampling uses nearest-neighbor interpolation followed by a 1×1 convolution.The authors report no significant benefits from more sophisticated up- and down-sampling alternatives.Source: A. Network Architecture · p. 10
  • For image inputs, a feature-generation module produces the initial feature representation, and each upsampling or downsampling module is paired with a same-resolution module to form a residual block.The feature-generation module removes the first rectified linear unit and identity pathway.Source: A. Network Architecture · p. 10
  • The number of encoder downsampling blocks matches the decoder upsampling blocks, with three blocks at 32-pixel output resolution and one additional block per doubling of output resolution.When input and output resolutions differ, additional downsampling or upsampling blocks account for the resolution difference.Source: A. Network Architecture · p. 10
  • A final 1×1 convolution produces output channels equal to the voxel-tube resolution or six times the number of Matryoshka shape layers.Models were trained on a single GPU, with learning rate scaled according to batch size.Source: A. Network Architecture · p. 10

B. More Results

Additional experiments show that Matryoshka networks produce detailed high-resolution reconstructions across synthetic and real-world images. Silhouette reconstruction works particularly well for cars, while residual modules use standard 2D operations.

Network architecture

  • Residual modules apply batch normalization and ReLU before 2D convolution; the up-sampling module substitutes a transposed convolution for its first convolution.The figure defines f as feature-channel count, k as filter size, and s as stride.Source: Figure 9 · p. 11

Shape from silhouette

  • On ShapeNet-core silhouettes, reconstruction performs much better for cars than for tables or chairs, which the authors attribute to cars’ approximately convex shape.The authors report strong performance relative to the harder color-image setting, whose network was trained category-agnostically.Source: B. More Results · p. 10, Table 8 · p. 11

Real-world images

  • Trained only on synthetic images, category-specific Matryoshka networks generalized to real-world chairs and cars, reconstructing thin structures and a wide variety of shapes.The real-world experiments used 128^3 output resolution.Source: B. More Results · p. 11, Figure 10 · p. 12, Figure 11 · p. 12

Synthetic images

  • Higher-resolution predictions are much more accurate, and high-quality reconstructions persist for varied classes such as airplanes and chairs.The qualitative examples also compare low-resolution voxel-tube and Matryoshka networks trained on 13 ShapeNet-core categories.Source: B. More Results · pp. 11–12
Loading 1804.10975v1…