PaperlayerView exact revision on arXiv

Source-linked AI summary

Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction

Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, David Novotny

arXiv:2109.00512v1cs.CV

TL;DR

Real-world 3D category reconstruction has been constrained by the lack of large, annotated datasets. This paper introduces CO3D and NerFormer, and reports large-scale evaluation in which NerFormer outperforms 14 baselines.

  • Problem

    Real-world 3D category reconstruction lacks relevant datasets with 3D ground truth, so methods are predominantly trained and benchmarked on synthetic data.

  • Method

    The paper constructs CO3D, an in-the-wild dataset of object-centric videos with camera and point-cloud annotations, and introduces NerFormer for category reconstruction using neural implicit rendering and Transformers.

  • Results

    NerFormer outperforms 14 tested baselines and is either the best or second best across all metrics in the reported evaluations.

  • Takeaways & Limitations

    CO3D enables large-scale in-the-wild evaluation of 3D object-category learning, while NerFormer provides a strong baseline for reconstruction from real object views.

  • Takeaways & Limitations

    NeRF-WCE can fail when parts of the 3D domain receive noisy WCE labels because its pointwise MLP lacks spatial reasoning.

Abstract

from arXiv · show

Traditional approaches for learning 3D object categories have been predominantly trained and evaluated on synthetic datasets due to the unavailability of real 3D-annotated category-centric data. Our main goal is to facilitate advances in this field by collecting real-world data in a magnitude similar to the existing synthetic counterparts. The principal contribution of this work is thus a large-scale dataset, called Common Objects in 3D, with real multi-view images of object categories annotated with camera poses and ground truth 3D point clouds. The dataset contains a total of 1.5 million frames from nearly 19,000 videos capturing objects from 50 MS-COCO categories and, as such, it is significantly larger than alternatives both in terms of the number of categories and objects. We exploit this new dataset to conduct one of the first large-scale "in-the-wild" evaluations of several new-view-synthesis and category-centric 3D reconstruction methods. Finally, we contribute NerFormer - a novel neural rendering method that leverages the powerful Transformer to reconstruct an object given a small number of its views. The CO3D dataset is available at https://github.com/facebookresearch/co3d .

1. Introduction

The paper addresses the lack of large real-world datasets with 3D ground truth by introducing CO3D and evaluating category-centric reconstruction in the wild. It also proposes NerFormer, which combines Transformers with neural implicit rendering for reconstruction from few views.

  • Real category-centric 3D reconstruction methods were predominantly trained and evaluated on synthetic data or narrow real datasets because real 3D ground truth was scarce.
  • CO3D uses crowd-sourced object-centric smartphone videos and Structure-from-Motion to obtain camera and dense point-cloud annotations at scale.
  • Almost 19,000 videos cover 50 MS-COCO categories and 1.5 million frames, with 20% receiving semi-manually verified high-resolution 3D point clouds.
  • The dataset extends earlier work from fewer than 10 categories to 50 and adds human-in-the-loop camera checks and point-cloud annotations.
  • NerFormer combines Transformers and neural implicit rendering to reconstruct object categories from a small number of source views.
  • NerFormer outperforms 14 baselines, enabling one of the first large-scale evaluations of learning 3D object categories in the wild.

2. Related Work

Related work spans synthetic and real 3D datasets, multiple shape representations, differentiable rendering, and multi-view feature aggregation. CO3D expands real-world category coverage and full-view capture relative to earlier datasets, while NerFormer learns aggregation rather than averaging it.

  • 3D object datasets: ShapeNet enabled early 3D reconstruction, while real datasets varied in pose, 3D-model, camera, category, and full-360-degree coverage.
  • 3D object datasets: CO3D increases the number of categories by a factor of 5 over Objectron and covers the full 360-degree range.
  • 3D reconstruction: Existing supervised reconstruction methods use voxel grids, meshes, point clouds, signed distance fields, or continuous occupancy fields.
  • 3D reconstruction: Differentiable rendering replaces direct 3D-shape comparison with 2D-image comparison using rasterization, raymarching, sphere tracing, or differentiable point clouds.
  • Novel-view synthesis: Unlike methods that average view-conditioned embeddings, NerFormer learns multi-view aggregation and can hallucinate new colors from at most 9 source views.

3. Common Objects in 3D

CO3D scales object-centric video collection through crowd-sourcing and derives camera, segmentation, depth, and point-cloud annotations with automated reconstruction plus quality filtering. The process required 43,819 GPU-hours.

  • AMT video collection: Amazon Mechanical Turk workers selected objects from 50 MS-COCO categories and recorded full-circle videos while keeping each object visible.
  • Camera estimation: COLMAP Structure-from-Motion annotates video frames with camera projection matrices after frames are sampled uniformly in time.
  • Object segmentation: PointRend produces soft binary object masks, while camera tracking remains anchored to background regions rather than those masks.
  • Semantic dense reconstruction: COLMAP multi-view stereo generates dense depth maps, and point-cloud fusion back-projects masked depths while retaining points consistent across frames.
  • Labelling reconstruction quality with Human-in-the-loop: An active-learning SVM filters unreliable reconstructions by cycling through manual quality labels, classifier retraining, and automatic quality estimation.
  • The dataset: 43,819 GPU-hours were used for reconstruction, with each single-video reconstruction taking 1 hour 56 minutes on average.

4. Learning 3D categories in the wild

This section formulates category-centric reconstruction from a small set of source views and introduces NerFormer to address missing spatial reasoning and noisy multi-view aggregation. The model uses Transformers to reason along rendering rays and learn feature pooling across source views.

  • Problem setting: Category-centric reconstruction represents geometry and appearance with implicit functions conditioned on 3D points, viewing direction, and a latent code.The implicit surface function models geometry, while a color function models viewpoint-dependent appearance such as specularities.
  • Motivation: Simple MLPs process 3D points independently, limiting spatial reasoning when source views provide only partial information about the reconstructed shape.The paper also identifies simple averaging of source-view embeddings as a way important features can be suppressed.
  • NerFormer: NerFormer learns attention weights for aggregating source features and implicitly pools information from nearby source views.The model samples source-image features at projected ray points and predicts their relative importance before forming a source embedding per point.
  • Latent shape encoding z: Warp-conditioned embeddings associate sampled source-image features with world-coordinate points, enabling invariance to rigid scene misalignment.A source feature is sampled at the perspective projection of each 3D point and conditioned on that point’s world location.
  • NerFormer: NerFormer replaces the shared MLP with a Transformer that processes depth-ordered features along rays and unordered features across the pooling dimension.Its alternating Transformer operations jointly learn spatial reasoning and source-view feature aggregation.

5. Experiments

The experiments evaluate new RGB+D-view synthesis and category-centric reconstruction on CO3D across multiple representations, datasets, source-view counts, and viewpoint difficulties. NerFormer performs strongly across metrics and generalizes best among applicable methods on test-unseen categories, while several baselines show representation- or overfitting-related weaknesses.

  • Experimental setup: 18,619 CO3D videos are split by category into train/test videos, with train and test videos further divided into known and unseen frames.The evaluation uses a 9:1 video split and an 8:2 frame split within videos.
  • Experimental setup: Four metrics evaluate rendered RGB, masks, and depth: PSNR, LPIPS, Jaccard index, and foreground depth ℓ1 distance.Metrics are computed at 800x800 resolution for single-scene reconstruction and 400x400 for category-specific reconstruction.
  • Experimental setup: The benchmark includes 15 approaches spanning implicit surfaces, voxel grids, point clouds, and meshes.The implicit-surface baselines include NeRF, IDR, DVR, SRN, positional-embedding variants, and WCE-augmented variants.
  • 5.2. Single-scene reconstruction: NerFormer is either the best or second best across all metrics in single-scene reconstruction.Vanilla NeRF outperforming NeRF+WCE suggests noisy WCE embeddings can hurt without NerFormer’s spatial reasoning; IDR trades higher LPIPS performance for lower PSNR.
  • 5.2. Single-scene reconstruction: WCE- or γ-enhanced SRN shows a train/test PSNR discrepancy, indicating that increased expressivity can overfit training views in the single-scene setting.The observed discrepancy is specifically reported for γ/WCE-endowed SRN.
  • 5.3. Learning 3D Object Categories: On test-unseen categories, NerFormer is best for most color metrics, while SRN+AD leads train-unseen performance but is inapplicable to test-unseen evaluation.DVR+WCE and IDR+WCE fail to converge, and P3DMesh performs poorly on complicated real geometries and textures.

6. Conclusion

The paper introduces CO3D, an in-the-wild dataset with camera and point-cloud annotations, and NerFormer, which reconstructs categories more accurately than 14 tested baselines. The collection continues to expand.

  • CO3D contains in-the-wild object-centric videos from 50 categories with camera and point-cloud annotations.
  • NerFormer combines Transformers with neural implicit rendering to reconstruct 3D object categories from CO3D.
  • NerFormer achieves better accuracy than 14 other tested baselines.
  • CO3D collection continues at approximately 500 videos per week, with further data planned for release.

A.1. Results on all 50 categories

Across all 50 object classes, NerFormer performs best on test-unseen color metrics, while some alternatives perform better on train-unseen settings. Convergence analysis links faster training with stronger performance and exposes overfitting.

  • Results on all 50 categories: NerFormer is best in all color-based metrics on the test-unseen set across the evaluated 50-class setting.
  • Results on all 50 categories: SRN and NeRF+WCE appear prone to overfitting to training scenes, according to their weaker test-unseen color performance.
  • Convergence speed: SRN, IDR, NerFormer, and NeRF are the fastest-converging methods and also rank highest in the single-scene benchmark.The convergence analysis averages per-epoch mean PSNR over 40 test scenes.
  • Convergence speed: A large train–test PSNR gap for γ/WCE-equipped SRN indicates overfitting to training views in the single-scene setting.
  • Execution time: Rendering-time evaluation reports the average time to render an 800x800 pixel image on an NVIDIA Tesla V100 GPU.

A.4. Test-time autodecoder optimization

The supplementary material describes test-time latent optimization for autodecoder methods, evaluates camera-view difficulty, and documents additional training and rendering details. It reports that optimization often fails for NeRF and NV, while SRN performs slightly better.

  • Test-time autodecoder optimization: Autodecoder methods are trained on train-known data, then optimize input latent codes for test-sequence source frames while freezing trained weights.The latent codes are optimized with Adam until convergence.
  • Test-time autodecoder optimization: Latent-code optimization mostly fails for NeRF and NV, whereas SRN achieves slightly better performance.
  • Estimating new-view difficulty: Camera difficulty is computed from the average of the two lowest distances between a target camera and its source views.
  • Estimating new-view difficulty: The camera-distance measure uses projection-ray similarities over a voxel grid and accounts for both camera intrinsics and extrinsics.
  • Estimating new-view difficulty: Target cameras are assigned to easy, medium, or hard bins using thresholds 0, 1/6, and 1/3 on the difficulty measure.
  • Additional implementation details: NerFormer and NeRF training optimize RGB and mask-related losses, with fine and coarse renders included for NerFormer.

B.3. SDF methods - DVR, IDR

The SDF baselines represent surfaces through signed distance fields and render them with sphere tracing or learned ray marching. Their implementations include latent conditioning, source-view features, and mask supervision.

  • SDF representation: Signed distance fields represent the signed Euclidean distance from a 3D point to the nearest point on an implicit surface.
  • DVR and IDR: Sphere tracing renders a pixel by seeking the minimum signed distance along its projection ray and iteratively refining the surface intersection.
  • SRN baseline: SRN replaces SDF queries with an LSTM that predicts ray-marching steps using the current point, ray direction, and hidden state.
  • SRN baseline: SRN+WCE lets the learned ray marcher access source views at every iteration, providing a more direct triangulation signal.
  • SRN baseline: The WCE extension of SRN is reported as a strong baseline and achieves the best depth-prediction performance.
  • Neural Volumes: Neural Volumes uses voxel grids, while its autodecoder and WCE variants condition voxel-processing networks on scene codes or source-view features.

B.7. Implicit Point Cloud (IPC)

IPC represents shapes as implicit functions anchored on predicted point clouds, enabling efficient rendering while retaining point-cloud structure. Its predictor deforms a template using latent shape or scene codes, and training combines image reconstruction with projected-mask alignment.

  • Implicit representation: IPC converts a predicted point cloud into an implicit occupancy function for rendering with efficient alpha-compositing ray marching.Anchoring the implicit function to cloud points is described as faster and more memory-efficient than neural implicit occupancy.
  • Coloring and rendering: IPC colors arbitrary queried locations by applying the coloring MLP at each query’s nearest predicted point-cloud neighbor.This nearest-neighbor assignment defines the implicit point-cloud coloring function.
  • Coloring and rendering: Rendering projects predicted 3D points into the target camera, selects points near each pixel, orders them by depth, and alpha-composites their colors.The renderer uses PyTorch3D and approximates point-neighborhood intersections through camera-plane distances scaled by focal length.
  • Coloring and rendering: The transmission term suppresses a point’s contribution when a ray intersects its epsilon neighborhood and ignores its color outside that neighborhood.At an intersecting projection, transmission is zero and the point color is fully rendered; outside, transmission is one.
  • Point cloud predictor: The point-cloud predictor deforms template points with an MLP offset conditioned on a latent shape code.IPC+AD uses a video-specific scene code, whereas IPC+WCE uses a warp-conditioned embedding; single-scene IPC sets the scene code to zero.
  • Training: Training minimizes rendered-image MSE together with Chamfer distance between projected predicted points and ground-truth segmentation-mask points.The segmentation-derived geometric loss complements image-color supervision.

C. 3D annotations with Human-in-the-loop

The annotation pipeline combines manually labeled reconstructions, reconstruction-quality metrics, and an active SVM to label camera tracking and point-cloud quality. Iterative boundary-focused review produces 1.5k labels, with cross-validation accuracies of 90% for camera tracking and 78% for point-cloud quality.

  • Initialization: Several hundred reconstructions are initially labeled accurate or inaccurate by visually inspecting camera tracks and scene point clouds.These manual labels seed the subsequent quality-classification process.
  • Quality features: SfM-derived features include rendered RGB and depth errors, registered-camera counts, and final bundle-adjustment energy.The metrics are extracted per video and used as inputs to the quality classifier.
  • Active-SVM labeling: An RBF-kernel SVM classifies camera-tracking and point-cloud quality from reconstruction metrics.The classifier is trained on the manually labeled examples before assigning labels to the remaining videos.
  • Active-SVM labeling: 1.5k labels, or 8% of the dataset, are collected by repeatedly manually annotating samples nearest the SVM decision boundary and correcting major errors.The process alternates between SVM retraining and manual annotation, with additional inspection of highest- and lowest-scoring samples.
  • Validation: 90% camera-tracking accuracy and 78% point-cloud-quality accuracy are obtained in 5-fold cross-validation.These figures evaluate the SVM on the set of annotated videos.
Loading 2109.00512v1…