PaperlayerView exact revision on arXiv

Source-linked AI summary

Multiview Compressive Coding for 3D Reconstruction

Chao-Yuan Wu, Justin Johnson, Jitendra Malik, Christoph Feichtenhofer, Georgia Gkioxari

arXiv:2301.08247v1cs.CV

TL;DR

Single-view 3D reconstruction must infer geometry obscured by occlusion, while prior approaches often require multiple inference views or costly, category-specific supervision. MCC learns from diverse RGB-D data using point-based encoding and a 3D-aware query decoder. The authors report generalization across objects and scenes, including novel types and in-the-wild or AI-generated examples.

  • Problem

    Single-view reconstruction must recover 3D structure hidden by image-plane occlusion, while existing approaches may need multiple inference views or costly CAD and category-specific priors.

    Source: §1 · p. 1

  • Method

    MCC encodes a single RGB-D image and its unprojected points, then uses a 3D-aware decoder to predict occupancy and color at queried points.

    Source: §1 · pp. 1–2

  • Results

    MCC is reported to outperform state-of-the-art methods for objects and scenes and to generalize to novel object and scene types, including in-the-wild captures and AI-generated objects.

    Source: §1 · p. 2, §4.4 · p. 6, §7 · p. 8

  • Takeaways & Limitations

    The authors conclude that point-based reconstruction coupled with category-agnostic large-scale training is effective across objects and scenes.

    Source: §1 · p. 1, §7 · p. 8

    MCC can fail on largely incorrect depth, far out-of-distribution targets, and fine texture details that are difficult to infer from one view.

    Source: §6 · p. 8

Abstract

from arXiv · show

A central goal of visual recognition is to understand objects and scenes from a single image. 2D recognition has witnessed tremendous progress thanks to large-scale learning and general-purpose representations. Comparatively, 3D poses new challenges stemming from occlusions not depicted in the image. Prior works try to overcome these by inferring from multiple views or rely on scarce CAD models and category-specific priors which hinder scaling to novel settings. In this work, we explore single-view 3D reconstruction by learning generalizable representations inspired by advances in self-supervised learning. We introduce a simple framework that operates on 3D points of single objects or whole scenes coupled with category-agnostic large-scale training from diverse RGB-D videos. Our model, Multiview Compressive Coding (MCC), learns to compress the input appearance and geometry to predict the 3D structure by querying a 3D-aware decoder. MCC's generality and efficiency allow it to learn from large-scale and diverse data sources with strong generalization to novel objects imagined by DALL$\cdot$E 2 or captured in-the-wild with an iPhone.

1. Introduction

Single-view 3D reconstruction must recover scene structure hidden by image-plane occlusion, while existing approaches often need multiple inference views or costly, category-specific supervision. MCC uses point-based encoding, a queryable 3D-aware decoder, and diverse RGB-D training to target generalization across objects and scenes.

1. Introduction

  • Single-view reconstruction addresses image-plane occlusion, while prior approaches either require multiple views at inference or rely on costly CAD supervision.The passage identifies the toy’s hidden left arm as an example of structure not visible in the image.Source: §1 · p. 1
  • The authors argue that versatile, efficient 3D points support category-agnostic large-scale training, which makes the model effective.They contrast this with category-specific priors that rely on onerous 3D annotations and may not hold generally.Source: §1 · p. 1
  • MCC represents inputs as 3D points and uses a queryable decoder to predict occupancy and color for seen and unseen geometry.The single RGB-D image is unprojected into visible 3D points, which are encoded with the image; sampled 3D queries are then decoded.Source: §1 · pp. 1–2
  • MCC trains from posed RGB-D views whose noisy point clouds serve as proxy ground truth and are sufficient when used at scale.The authors suggest these easier-to-collect video captures can replace expensive 3D annotations.Source: §1 · p. 2
  • MCC is evaluated across six diverse data sources and is reported to outperform specialized methods for both objects and scenes, with zero-shot generalization to novel types.The sources include CO3D, Hypersim, Taskonomy, ImageNet, in-the-wild iPhone captures, and DALL·E 2 generations.Source: §1 · p. 2

2. Related Work

Prior 3D reconstruction work spans multiview methods, supervised and weakly supervised single-view methods, shape completion, implicit representations, and self-supervised learning. MCC is positioned as a unified, category-agnostic approach for reconstructing objects and scenes from one RGB-D image.

2. Related Work

  • Multiview methods can produce high-quality outputs, but require multiple views at test time; this work assumes one RGB-D image during inference.Prior techniques include stereopsis, SfM, and SLAM.Source: §2 · p. 2
  • Single-view methods use CAD, mesh, voxel, or point-cloud supervision, but many target synthetic benchmarks or closed-world object categories.Some category-agnostic methods instead focus on synthetic datasets; MCC learns from diverse real-world object and scene sources.Source: §2 · p. 2
  • Shape-completion methods often use fixed-resolution object outputs or specialized scene techniques, whereas MCC targets both with one architecture and any-resolution geometry.The compared approaches include point-cloud and generative models for objects, and plane fitting, model fitting, retrieval, or symmetry-based methods for scenes.Source: §2 · p. 2
  • MCC uses occupancy with attention over encoded appearance and geometry to query any 3D region, and the authors report better performance than global-feature OccNets and single-location-feature NeRF methods.The passage contrasts MCC’s strategy with OccNets [37] and NeRF-based methods [21] [51].Source: §2 · p. 2
  • MCC extends self-supervised learning ideas from masked autoencoders and large-scale category-agnostic image training to 3D reconstruction.The cited image-learning advances use transformers and masked image patches to learn general representations.Source: §2 · p. 2

3. Multiview Compressive Coding (MCC)

MCC encodes a single RGB-D image and its unprojected 3D points, then uses a queryable decoder to predict occupancy and color. Posed RGB-D views provide point supervision, while the encoder-decoder design supports reconstruction at different output resolutions.

3. Multiview Compressive Coding (MCC)

  • MCC predicts occupancy and color for queried 3D points, whose occupied colored set forms the reconstruction.At inference, the model predicts over a grid at the desired resolution.Source: §3 · p. 3
  • Noisy point clouds from posed RGB-D views supervise MCC, with other views guiding predictions about unseen space.The authors report that these imperfect points are sufficient at scale, offering an alternative to clean CAD models or meshes.Source: §3 · p. 3

3.1. MCC Encoder

  • The encoder combines RGB-image and 3D-point transformer outputs into a single representation R.The point encoder processes 3D coordinates rather than RGB channels; a fusion function concatenates both outputs and projects them to C dimensions.Source: §3.1 · p. 3

3.2. MCC Decoder

  • The decoder appends projected 3D queries to R and uses a transformer to predict each query’s occupancy and color.Attention masking prevents queries from depending on one another, apart from themselves.Source: §3.2 · p. 3, Figure 3 · p. 4

3.3. Query Sampling

  • MCC samples 550 3D-space queries per training example, labeling points within radius τ = 0.1 of ground truth as occupied.The ground truth is the union of unprojected points from the scene’s RGB-D views.Source: §3.3 · p. 4
  • At inference, a uniformly sampled 3D grid is reconstructed from queries whose occupancy scores exceed 0.1 and their predicted colors.The output resolution can change without recomputing the input encoding R.Source: §3.3 · p. 4, §3.2 · p. 4

3.4. Implementation Details

  • The RGB and point encoders use 12-layer, 768-dimensional ViT-Base architectures, while the decoder is an 8-layer, 512-dimensional transformer.For points with unknown depth, the encoder learns a special embedding; valid 3D points are linearly transformed before patch summarization.Source: §3.4 · p. 4

4. Object Reconstruction Experiments

On CO3D-v2, MCC is evaluated on 10 held-out categories after training on the other 41. The experiments examine reconstruction quality, design choices, scaling, in-the-wild generalization, and comparisons with image-conditioned NeRF methods.

4. Object Reconstruction Experiments

  • CO3D-v2 provides about 37k videos across 51 categories; MCC trains on 41 categories and holds out 10 for novel-category evaluation.The CO3D point clouds are extracted with COLMAP and are noisy.Source: §4 · p. 4

4.1. Qualitative Results on Novel Categories

  • On novel CO3D categories, MCC completes geometry despite occlusion and noisy COLMAP depth, predicts color on unseen surfaces, and uses no symmetry or mean-shape priors.Examples include a barely visible mug handle and color propagated to the unseen left and back of a backpack.Source: §4.1 · pp. 4–5, Figure 4 · p. 5

4.2. Ablation Study

  • Decoupled encoders slightly outperform shared encoding; transformer and PointNet point encoders edge MLP, sampling performs similarly, detailed conditioning is validated, and the concat-attention decoder is critical.Uniform query sampling is selected for simplicity despite similar results from contrastive-style sampling.Source: §4.2 · p. 5
  • With the same number of output points, MCC outperforms PoinTr by a large margin and predicts shape details where PoinTr places points roughly around the object.The comparison uses a 12-layer PoinTr trained on CO3D and a texture-ignoring MCC variant.Source: §4.2 · p. 6, Figure 5 · p. 6

4.3. Scaling Behavior Analysis

  • Scaling training data steadily improves performance on held-out novel categories, with larger gains when adding categories than when adding examples within existing categories.The category expansion also increases training-data visual diversity.Source: §4.3 · p. 6, Figure 6 · p. 6

4.5. Comparison to Image-Conditioned NeRF

  • MCC outperforms NeRF-WCE and NerFormer for shape reconstruction; NerFormer captures texture but struggles with geometry, while MCC predicts more accurate shapes from one view.The evaluation uses novel-view depth metrics, which the authors say disadvantage MCC because it is not designed for rendering.Source: Table 2 · p. 6, §4.5 · p. 7, Figure 8 · p. 7

4.4. Zero-Shot Generalization In-the-Wild

  • Qualitative zero-shot examples span iPhone LiDAR captures, ImageNet web images, and DALL·E 2 imaginary objects, with shifts in object types, image styles, depth sources, and visual context.The ImageNet and generated-image examples use depth estimated by an off-the-shelf model.Source: §4.4 · p. 6, Figure 7 · p. 7

5. Scene Reconstruction Experiments

MCC reconstructs scene geometry from a single RGB-D image, including regions beyond the camera frustum. It outperforms DRDF on Hypersim and reconstructs room layouts in novel Taskonomy scenes.

5. Scene Reconstruction Experiments

  • The Hypersim evaluation uses 365 training scenes and 46 test scenes, with images and depth for training and available 3D meshes for evaluation.The task is to reconstruct the scene in front of the camera up to a specified range, including areas outside its frustum.Source: §5 · pp. 7–8

5.1. Hypersim Scene Reconstruction

  • From a single RGB-D image, MCC reconstructs room geometry beyond the view frustum, including heavily occluded or entirely unseen floors and walls, but misses fine shape and texture details.The authors suggest more data may improve performance, consistent with their scaling analysis.Source: §5.1 · p. 8, Figure 9 · p. 8
  • MCC outperforms DRDF across all metrics on Hypersim, including a DRDF variant using MCC’s architecture but retaining its original loss and ray-based inference.The architecture variant improves on original DRDF but remains worse than MCC.Source: §5.1 · p. 8, Table 3 · p. 8

5.2. Zero-Shot Generalization to Taskonomy

  • Trained on synthetic Hypersim data, MCC reconstructs room layouts—floors, walls, and ceilings—in novel real Taskonomy scenes.This evaluates both generalization to novel scenes and synthetic-to-real transfer.Source: §5.2 · p. 8, Figure 9 · p. 8

6. Failure Cases

MCC’s observed failure modes concern incorrect depth, distribution shifts, and detailed texture prediction.

  • Largely incorrect depth can make MCC fail to reconstruct accurate 3D geometry, although it can recover from noisy depth.Source: §6 · p. 8
  • Inputs far from the training distribution can produce errors in both texture and geometry, with Rubik’s cubes as an example.Source: §6 · p. 8
  • Detailed texture is difficult to predict from a single view, and MCC often omits details such as text on a volleyball.Source: §6 · p. 8

7. Conclusions

MCC is a general-purpose 3D reconstruction model for both objects and scenes. The authors report generalization to in-the-wild captures and AI-generated imagined objects, and caution that data-driven models may inherit dataset bias.

  • MCC is presented as a general-purpose 3D reconstruction model that works for both objects and scenes.Source: §7 · p. 8
  • The reported generalization settings include in-the-wild captures and AI-generated images of imagined objects.Source: §7 · p. 8
  • The authors train only on inanimate objects and scenes to minimize potential inherited data bias, and caution against future use without careful attention to training data.Source: §7 · p. 8

A.1. Animations and Interactive Visualizations

Qualitative results include 360-view animations and interactive 3D visualizations.

  • The paper provides 360-view animations for qualitative results in Figures 4, 7, and 9.Source: §A.1 · p. 9
  • Interactive 3D visualizations are available by clicking the 3D icon as directed on the project page.Source: §A.1 · p. 9
  • The project page provides additional animations beyond those shown in the main window.Source: §A.1 · p. 9

A.2. Architecture Specifications

MCC encodes RGB images and observed 3D points into tokens for a transformer decoder that predicts structure from 3D point queries. The appendix also specifies held-out CO3D categories and scene-reconstruction training and inference settings.

A.2. Architecture Specifications

  • The ERGB and EXYZ encoders use 12-layer, 768-dimensional ViT-Base transformers and each produce 196 tokens from image patches or observed-point patches.The RGB input is 224×224; observed points are linearly projected before patch embedding.Source: §A.2 · p. 9
  • The decoder combines 550 projected 3D point queries with concatenated encoder tokens in an 8-layer, 512-dimensional transformer.Its input has 746 tokens, and a global [cls] token summarizes the inputs; LayerNorm is used in all self-attention and MLP layers.Source: §A.2 · p. 9, Table 4 · p. 9

A.3. Held-Out CO3D Categories

  • The CO3D test set holds out 10 randomly selected categories containing 8,453 example videos in total.The categories include apple, ball, baseballglove, book, bowl, carrot, cup, handbag, suitcase, and toyplane.Source: §A.3 · p. 9

A.4. Additional Implementation Details for Scene Reconstruction Experiments

  • Scene reconstruction training uses Hypersim for 100k iterations, while inference restricts points to within 6.0 units of the camera origin.Scenes are normalized to zero mean and unit variance; images are randomly scaled by 0.8–1.2 without rotation augmentation, using camera-view coordinates.Source: §A.4 · pp. 9–10
Loading 2301.08247v1…