PaperlayerView exact revision on arXiv

Source-linked AI summary

NodeSLAM: Neural Object Descriptors for Multi-View Shape Reconstruction

Edgar Sucar, Kentaro Wada, Andrew Davison

arXiv:2004.04485v2cs.CV

TL;DR

Persistent 3D representations of objects are needed for shape inference and applications that interact with scenes. The paper combines compact class-level shape models with probabilistic rendering and uses them in object-level SLAM. Its system jointly optimises object shapes and poses with camera trajectory, achieving comparable surface reconstruction to Fusion++ [19] while reconstructing full objects from fewer observations.

  • Problem

    Computer vision needs persistent 3D scene representations that support useful shape inference, especially for objects available for interaction.

    Source: §1 · p. 1

  • Method

    The paper learns compact class-conditioned shape descriptors and uses probabilistic rendering to optimise object shapes from depth images and jointly map objects and track cameras.

    Source: §1 · p. 2, §6 · p. 5

  • Results

    Joint optimisation gives more robust tracking and reconstruction, with surface reconstruction comparable to Fusion++ [19] while reaching full object reconstruction from fewer observations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework is used in object-level SLAM, robotic manipulation, and augmented reality demonstrations.

    Source: §1 · p. 2, §8 · p. 8

    The single-code object VAE does not represent all object classes well; the authors propose investigating part-based coding and self-supervised training.

    Source: §8 · p. 8

Abstract

from arXiv · show

The choice of scene representation is crucial in both the shape inference algorithms it requires and the smart applications it enables. We present efficient and optimisable multi-class learned object descriptors together with a novel probabilistic and differential rendering engine, for principled full object shape inference from one or more RGB-D images. Our framework allows for accurate and robust 3D object reconstruction which enables multiple applications including robot grasping and placing, augmented reality, and the first object-level SLAM system capable of optimising object poses and shapes jointly with camera trajectory.

1. Introduction

The paper argues that persistent object representations should support efficient, robust, incremental inference, and presents class-level generative shape models with probabilistic rendering for that purpose. It applies these components to object-level SLAM and real-world demonstrations, including robotic manipulation and augmented reality.

1. Introduction

  • A class-conditioned generative model represents object shape with a compact code that is iteratively optimised against any number of depth measurements.The paper positions this approach between flexible classical reconstruction and discriminative methods that struggle to integrate multiple measurements; class-level priors also enable complete reconstruction from a single image, unlike partial TSDF fusion.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • A probabilistic volumetric renderer models pixel uncertainty and receptive field to support efficient, robust shape optimisation from depth images.The paper presents it as an improvement over the state of the art in volumetric rendering.Source: §1 · p. 2
  • Jointly optimising object shapes and poses with camera trajectory yields more robust tracking and reconstruction, with surface reconstruction comparable to Fusion++ [19] and full objects from fewer observations.The system uses the same measurement function for camera tracking, object poses, and shape optimisation; demonstrations include noisy table-top sequences and robotic packing and sorting.Source: §1 · p. 2
  • The framework is demonstrated on real-world image sequences and augmented reality, and integrated into a real-time robotic system for packing objects into a tight box and sorting by shape size.The paper attributes these tasks to the completeness and accuracy of its object reconstructions.Source: §1 · p. 2

2. Related Work

Prior reconstruction methods either recover only observed surfaces or struggle to integrate multiple observations, while earlier generative methods had not shown incremental, real-time mapping of multiple real-world objects. This work combines class-level generative models with probabilistic rendering and focuses map resources on objects.

2. Related Work

  • Discriminative single-image reconstruction approaches lack principled integration of multiple observations, while existing generative full-shape methods had not demonstrated incremental inference in real-world multi-object settings in real time.Source: §2 · p. 2
  • The proposed renderer gathers occupancy probabilities along back-projected rays, then uses a Gaussian pyramid to widen the receptive field and pixel uncertainty to weight residuals.Earlier differential volumetric renderers use a local receptive field, which the paper says can slow optimisation and trap it in local minima.Source: §2 · p. 2
  • Object-focused maps offer a route to efficient, semantically meaningful representations because dense whole-scene maps are expensive to store and difficult to optimise probabilistically from multiple views.Earlier visual SLAM representations progressed from point clouds through dense surfaces to semantically labelled dense maps [40].Source: §2 · p. 2
  • Class-level generative models can reconstruct whole objects across shape variation within a class, bridging arbitrary-class surface reconstruction and systems limited to precisely known CAD instances.The cited general systems reconstruct directly observed surfaces, while CAD-instance systems are efficient but require precisely known objects.Source: §2 · p. 2

3. Class-Level Object Shape Descriptors

The paper uses regularities among aligned objects of the same semantic class to learn a smooth, compact latent representation of shape. Its network conditions both encoding and decoding on the object class.

3. Class-Level Object Shape Descriptors

  • Objects of the same semantic class exhibit strong shape regularities under a common pose alignment, enabling a class-specific smooth latent space for compact instance representations.A single class-conditional variational autoencoder learns this representation.Source: §3 · pp. 2–3

3.1. Network Design

  • Shapes use 32 × 32 × 32 voxel grids with continuous occupancy probabilities, chosen to support arbitrary topology.The decoder reconstructs occupancy from the encoded shape descriptor and class one-hot vector.Source: §3.1 · p. 3, Figure 2 · p. 3
  • The 3D CNN variational autoencoder is trained on mug, bowl, bottle, and can models, using a 16-dimensional latent shape variable and class conditioning at both encoder and decoder.The models come from ShapeNet [1]; the network uses KL-divergence in latent space and binary cross-entropy for reconstruction.Source: §3.1 · p. 3

4. Probabilistic Rendering

The rendering function maps an occupancy grid and its camera pose to depth and uncertainty images for principled inference. It estimates depth probabilistically along back-projected rays, combines object renders to handle occlusion, and uses image pyramids for coarse-to-fine optimisation.

4. Probabilistic Rendering

  • The rendering function outputs both rendered depth and uncertainty, supporting differentiable, probabilistic inference with a wide receptive field for robust optimisation on noisy depth images.The design requirements are stated as differentiability, probabilistic output, and a wide receptive field.Source: §4 · p. 3
  • The renderer samples points along each back-projected ray, interpolates occupancy from the voxel grid, and uses those probabilities to define depth termination and escape probabilities.It aggregates the resulting discrete distribution to obtain expected rendered depth; the escape depth is set to 1.1dmax for practical reasons.Source: §4 · p. 3
  • For multiple objects, the renderer takes the minimum depth at each pixel to account for occlusion.Source: §4 · p. 3
  • Gaussian blur and down-sampling across four pyramid levels enable coarse-to-fine optimisation by associating rendered pixels with several back-projected rays at higher levels.Each pyramid level uses a 1-pixel standard deviation.Source: §4 · p. 4

5. Object Shape and Pose Inference

The section infers an object’s full shape and pose from a depth image by fitting a decoded class-level shape to the measurement. It initializes the pose and shape before iteratively optimizing them.

5. Object Shape and Pose Inference

  • A known-class object’s full shape and pose are inferred from a depth image using its segmentation and classification with a probabilistic rendering function.The method is presented for a single observation and extended to multiple observations in the SLAM system.Source: §5 · p. 4

5.1. Shape and Pose Optimisation

  • The decoded occupancy grid and object pose render a depth image whose Gaussian mean and variance support joint shape-and-pose fitting to the measured depth.The Gaussian shape prior and constant rendered variance yield a least-squares MAP objective optimized with Levenberg–Marquardt.Source: §5.1 · p. 4, Figure 3 · p. 4
  • A structural prior encourages the object’s bottom to contact the supporting plane, and marching cubes recovers a surface mesh from the occupancy grid.A virtual camera renders an image from beneath the object.Source: §5.1 · p. 4

5.2. Variable Initialisation

  • Translation and scale are initialized from the masked-depth point cloud, while orientation uses the horizontal surface normal and a learned vertical-axis angle for mugs.The mug-angle CNN takes a cropped RGB object image and is trained on simulated renders with randomized materials and lighting.Source: §5.2 · p. 4
  • The latent descriptor starts at d = 0, the Gaussian-prior mean class shape, which iterative optimization deforms to fit the observations.The initialization pipeline also estimates object pose from depth and masked RGB, and infers class from RGB.Source: §5.2 · pp. 4–5, Figure 4 · p. 5

6. Object-Level SLAM System

The system incrementally builds an object map while tracking the camera and jointly optimizing camera poses, object poses, and shapes. It associates detections with mapped objects and optimizes selected keyframes in a sliding window.

6. Object-Level SLAM System

  • The system extends single-image shape and pose inference into incremental object-level SLAM, jointly optimizing camera poses, object shapes, and object poses across image measurements.The system uses sliding keyframe-window optimization.Source: §6 · p. 5

6.1. Data association and Object Initialisation

  • Detections are associated first with previous-frame masks using an IoU threshold above 0.2, then unmatched masks are compared with rendered map-object masks; unmatched detections initialize new objects.Each incoming image is segmented and classified with Mask-RCNN.Source: §6.1 · p. 5

6.2. Camera Tracking

  • Camera tracking holds the object map fixed and estimates camera pose by minimizing the same render-to-depth loss used for object pose and shape inference.The loss is optimized iteratively with Levenberg–Marquardt.Source: §6.2 · p. 5

6.3. Sliding-Window Joint Optimisation

  • To limit accumulated motion drift and incorporate new viewpoints, the system jointly optimizes a selected bundle of keyframe camera poses and observed object shapes and poses.A frame becomes a keyframe when it initializes an object or its viewpoint differs by more than 13 degrees from an object’s initialization view; the window contains the new keyframe and its two closest keyframes.Source: §6.3 · p. 6
  • Rendering one object takes 7 ms, full object reconstruction approximately 1.5 seconds, camera tracking runs at 7 fps, and joint optimization takes 2 seconds.These timings are reported for the system’s rendering, reconstruction, tracking, and joint-optimization processes.Source: §6.3 · p. 6

7. Experimental Results

The experiments evaluate rendering, object-level SLAM, and robotic manipulation, reporting reconstruction results against DVR and TSDF and demonstrating packing and sorting.

7.1. Metrics

  • Shape reconstruction is evaluated with chamfer-L1 distance, accuracy, and completeness, using 20,000 sampled points on reconstruction and ground-truth meshes.Completeness uses a 1cm threshold.Source: §7.1 · p. 6

7.2. Rendering Evaluation

  • Across mug instances, the renderer improves precision and completion over DVR; additional views help, and uncertainty and Gaussian pyramids improve accuracy and completeness.Figure 8 also shows faster convergence and lower accuracy error for the proposed renderer.Source: §7.2 · pp. 6–7, Figure 8 · p. 7

7.3.1 Fusion++ comparison

  • NodeSLAM reaches near-full completion faster than gradual TSDF fusion, with comparable surface reconstruction quality close to 5mm.The comparison uses ground-truth poses and adds noise to depth images and camera poses.Source: §7.3.1 · p. 7

7.3.2 Ablation Study

  • Joint optimisation and uncertainty rendering matter: removing these features reduces shape accuracy by 2–7 mm, while camera-tracking mean errors are between 1 and 2 cm.Tracking without joint optimisation or uncertainty leads to significantly lower accuracy on most trajectories.Source: §7.3.2 · pp. 7–8, Table 2 · p. 8

7.4. Robot Manipulation Application

  • In packing and sorting demonstrations, reconstructed object pose and shape guide placement; robot kinematics are used for camera tracking in this application.Objects are packed into a tight box or sorted in ascending size.Source: §7.4 · p. 8, Figure 10 · p. 8

8. Conclusions

The conclusion presents generative multi-class object models as a basis for full-shape inference and practical SLAM, manipulation, and augmented-reality demonstrations, while noting a class-representation limitation.

8. Conclusions

  • Generative multi-class object models support full-shape inference, with practical demonstrations in jointly optimisable object SLAM, robotic manipulation, and augmented reality.The paper describes these models as enabling principled and robust inference.Source: §8 · p. 8
  • Figure 11 compares object surface completion against TSDF fusion and presents reconstruction-accuracy comparisons for an ablation study and Fusion++.Source: Figure 11 · p. 8
  • Some object classes may not be well represented by the paper’s single-code object VAE; alternative part-based or self-supervised coding schemes are future work.Source: §8 · p. 8

Related papers

Loading 2004.04485v2…