PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images

Shuran Song, Jianxiong Xiao

arXiv:1511.02300v2cs.CV

TL;DR

Amodal 3D object detection must estimate an object’s full metric extent in RGB-D images despite truncation or occlusion. Deep Sliding Shapes addresses this with a 3D ConvNet pipeline using multi-scale amodal proposals and joint 2D color and 3D geometry recognition. Experiments show significant improvements over state-of-the-art approaches and faster inference than original Sliding Shapes.

  • Problem

    Amodal 3D detection seeks full metric 3D boxes despite truncation or occlusion, while existing 2D- and 3D-centric approaches leave open how best to represent RGB-D geometry.

  • Method

    Deep Sliding Shapes uses a 3D ConvNet pipeline with a two-scale amodal 3D RPN and a joint network combining 3D depth geometry with 2D color features for direct box regression.

  • Results

    The algorithm significantly outperforms state-of-the-art approaches and is much faster than original Sliding Shapes.

  • Takeaways & Limitations

    The results support 3D deep learning as a powerful approach for 3D shape representation in RGB-D object detection.

  • Takeaways & Limitations

    The proposal orientation procedure assumes a Manhattan-world scene orientation, although square horizontal aspect ratios reduce orientation sensitivity for objects such as chairs.

Abstract

from arXiv · show

We focus on the task of amodal 3D object detection in RGB-D images, which aims to produce a 3D bounding box of an object in metric form at its full extent. We introduce Deep Sliding Shapes, a 3D ConvNet formulation that takes a 3D volumetric scene from a RGB-D image as input and outputs 3D object bounding boxes. In our approach, we propose the first 3D Region Proposal Network (RPN) to learn objectness from geometric shapes and the first joint Object Recognition Network (ORN) to extract geometric features in 3D and color features in 2D. In particular, we handle objects of various sizes by training an amodal RPN at two different scales and an ORN to regress 3D bounding boxes. Experiments show that our algorithm outperforms the state-of-the-art by 13.8 in mAP and is 200x faster than the original Sliding Shapes. All source code and pre-trained models will be available at GitHub.

1. Introduction

The paper frames amodal 3D detection as a more useful but harder alternative to visible-part 2D detection, then introduces Deep Sliding Shapes as an integrated deep 3D formulation combining learned proposals with joint geometric and color features.

  • Motivation: Amodal 3D detection predicts an object’s full metric 3D extent despite truncation or occlusion, supporting reasoning such as robotics perception-manipulation.Adding the 3D dimension enlarges the search space and makes detection more challenging.
  • Motivation: Depth-RCNN currently outperforms Sliding Shapes, motivating whether deep learning can make a 3D representation more powerful than a 2D-centric formulation.Depth-RCNN uses depth as additional 2D image channels, whereas Sliding Shapes searches directly in 3D with hand-crafted features.
  • Contributions: Deep Sliding Shapes introduces a 3D ConvNet pipeline with a 3D RPN for amodal proposals and a joint network combining 3D geometric and 2D color features.The recognition network also directly regresses 3D bounding boxes from 3D proposals.
  • Contributions: The amodal RPN operates at two scales to handle objects with different physical sizes, while the recognition network learns features for proposal classification and 3D box regression.The design addresses distinct challenges from volumetric computation and large variation in 3D physical bounding-box sizes.
  • Related work: Deep Sliding Shapes seeks to replace hand-crafted 3D features and CAD-model-based or 2D-centric alternatives with learned 3D geometric representations integrated with color.Prior work includes 2D depth-channel methods, 3D sliding windows, and learned features for retrieval or classification rather than RGB-D 3D detection.

2. Encoding 3D Representation

The paper encodes RGB-D geometry with a directional TSDF on a regular voxel grid, preserving spatial locality and directional surface information for 3D ConvNets.

  • Directional TSDF: Directional TSDF divides 3D space into an equally spaced voxel grid and stores a three-dimensional distance vector [dx, dy, dz] from each voxel to its closest depth surface.Values are clipped by 2δ, with δ equal to the grid size in each dimension.
  • Directional TSDF: The sign of each directional TSDF value indicates whether a voxel lies in front of or behind the surface, while GPU conversion enables efficient voxel-grid construction.The representation is designed to encode geometric shapes directly in 3D rather than as 2D depth channels or hand-crafted features.
  • Anchors: Figure 4’s anchor types encode width × depth × height in meters and the number of orientations for each anchor.The proposal system defines 19 anchors from object-size statistics, with rotated counterparts for non-square horizontal aspect ratios.
  • Encoding comparison: Projective TSDF is faster because it searches only along the camera line of sight, but it performs worse than accurate TSDF for recognition.The proposed directional TSDF outperforms the other evaluated encodings.

3. Multi-scale 3D Region Proposal Network

The multi-scale 3D RPN generates amodal object proposals directly from volumetric scenes while addressing 3D search complexity and wide physical size variation.

  • Proposal challenges: 3D proposal search is especially challenging because the additional dimension increases possible object locations by 30 times.The proposal stage must remain selective while inferring full boxes and handling category-dependent physical sizes.
  • 3D Region Proposal Network: The 3D RPN learns objectness from volumetric scenes and generates amodal 3D bounding-box proposals across whole-object locations.Unlike bottom-up segmentation, it seeks proposals covering objects beyond visible regions.
  • Anchors and orientation: The network uses 19 anchors with varied sizes and aspect ratios, adding rotated versions for non-square horizontal shapes.Anchor orientations follow major room directions estimated under the Manhattan-world assumption.
  • Multi-scale RPN: A multi-scale RPN predicts proposals for objects ranging from 0.3-meter trash bins to 2-meter beds.Separate scales reduce distractions from using identical receptive fields for substantially different physical object sizes.
  • Efficiency: 92.2% of 1,387,646 anchors are almost empty and are removed before training and testing, leaving 107,674 anchors on average.Empty-box removal uses a 3D integral image in constant time.
  • Efficiency: 3D NMS reduces 107,674 non-empty proposals to the top 2,000 boxes, only 0.14% of all sliding windows.NMS uses a 3D IoU threshold of 0.35 before proposals enter recognition.

4. Joint Amodal Object Recognition Network

The joint amodal recognition network combines geometric and color features for object classification and direct 3D box regression, while using amodal boxes to improve alignment and size-based filtering.

  • Network inputs: The ORN processes each proposal’s 3D region with TSDF geometry and projects its points to a 2D image window for color features.The 3D input uses a 30 × 30 × 30 voxel grid, while the 2D branch extracts VGG features with ROI pooling.
  • Amodal recognition: Amodal proposal boxes let the ORN examine full objects and align them in 3D despite occlusion or missing data.This alignment is presented as supporting more invariant recognition.
  • Error analysis: Reported false positives arise from inaccurate locations, incorrect box sizes, or wrong categories, while misses involve occlusion, field-of-view limits, atypical sizes, or missing depth.These error examples span localization, extent estimation, classification, and incomplete observations.
  • Color representation: Color encoded through a pre-trained VGGnet performs much better than encoding color directly on 3D voxels.The paper attributes the difference to higher image resolution and reuse of ImageNet pre-training.
  • Joint recognition: The joint network concatenates 4096-dimensional 2D and 3D features, then predicts object labels and 3D boxes.A fully connected layer reduces the concatenated representation to 1000 dimensions before prediction.

5. Experiments

Experiments evaluate proposal generation, feature encoding, design choices, detection accuracy, and runtime on NYUv2 and SUN RGB-D. Deep Sliding Shapes benefits from multi-scale proposals, box regression, learned geometric and color features, and substantially faster inference than prior methods.

  • Runtime: 5.62s RPN plus 13.93s ORN per image is much faster than Depth RCNN and Sliding Shapes.Sliding Shapes requires 25 minutes multiplied by the number of object categories.
  • Object proposals: Adding a second RPN scale significantly boosts recall for small objects such as lamps, pillows, and garbage bins.Adding RGB color to the 3D TSDF encoding slightly improves proposal performance.
  • Object detection: RPN proposals increase detection mAP from 27.4 to 32.3 compared with 3D Selective Search using the same ORN architecture.The comparison isolates proposal generation while keeping the recognition network fixed.
  • Feature encoding: Directional TSDF outperforms single-distance TSDF, accurate TSDF outperforms projective TSDF, and 2D image VGG features outperform direct voxel color encoding.These conclusions compare the feature encodings evaluated in Table 2.
  • Design justification: 3D box regression improves mAP by +4.4 with 3D Selective Search and +4.1 with RPN.The gains are reported for both proposal-generation methods.
  • Design justification: SVM confidence improves mAP by 0.5 over softmax, while size pruning improves AP by 0.1 to 7.8 for most categories.The paper attributes the SVM result possibly to class imbalance and reports size pruning as reducing false positives.
  • State-of-the-art comparison: The method outperforms Sliding Shapes and Depth-RCNN on amodal 3D detection, directly outputs 3D boxes, and avoids Depth-RCNN’s CAD-model fitting post-processing.The evaluation uses the shared test setting described for the NYUv2 comparison and also reports SUN RGB-D results.
  • Qualitative comparison: Side-by-side comparisons show better handling of atypical object sizes, color-based disambiguation, and extension to more object categories.Small child chairs and a table missed by Sliding Shapes are detected by Deep Sliding Shapes.

6. Conclusion

The paper presents a 3D ConvNet pipeline for amodal 3D object detection with joint proposal and recognition networks. Experiments report significantly better performance than state-of-the-art approaches and much faster execution than original Sliding Shapes.

  • 6. Conclusion: The pipeline combines a 3D Region Proposal Network with a joint 2D+3D Object Recognition Network.The conclusion identifies these as the core components of the proposed amodal 3D detection system.
  • 6. Conclusion: Experiments show significant state-of-the-art performance gains and much faster execution than original Sliding Shapes.The conclusion connects these results to the use of 3D deep learning for 3D shape representation.
Loading 1511.02300v2…