Source-linked AI summary
Subcategory-aware Convolutional Neural Networks for Object Proposals and Detection
Yu Xiang, Wongun Choi, Yuanqing Lin, Silvio Savarese
TL;DR
CNN-based detectors struggle with region proposals under scale variation, occlusion, and truncation and mainly estimate 2D boxes. The paper introduces subcategory-aware proposal and detection networks that jointly use subcategory information for detection and pose-related estimation, achieving state-of-the-art benchmark results.
Problem
CNN-based detection methods have difficulty generating proposals for scale-varying, occluded, or truncated objects and generally cannot estimate detailed properties beyond 2D boxes.
Method
The paper uses subcategory detections to guide a new region proposal network and adds subcategory classification to a detection network, with image pyramids for scale variation.
Results
The method achieves state-of-the-art results on KITTI, PASCAL3D+, and PASCAL VOC 2007 for detection and pose-related tasks.
Takeaways & Limitations
Subcategories related to object attributes such as pose can support joint object detection, pose estimation, boundary segmentation, and occlusion or truncation estimation.
Abstract
from arXiv · showhide
In CNN-based object detection methods, region proposal becomes a bottleneck when objects exhibit significant scale variation, occlusion or truncation. In addition, these methods mainly focus on 2D object detection and cannot estimate detailed properties of objects. In this paper, we propose subcategory-aware CNNs for object detection. We introduce a novel region proposal network that uses subcategory information to guide the proposal generating process, and a new detection network for joint detection and subcategory classification. By using subcategories related to object pose, we achieve state-of-the-art performance on both detection and pose estimation on commonly used benchmarks.
1. Introduction
The paper addresses weaknesses in CNN-based detection under scale variation, occlusion, and truncation, as well as limitations in estimating object properties. It proposes subcategory-aware networks for region proposals, detection, and subcategory classification, with state-of-the-art benchmark results.
- CNN-based region proposal methods perform poorly on KITTI objects with large scale variation, occlusion, and truncation.
- Existing CNN detectors mainly produce 2D bounding boxes and cannot estimate properties such as segmentation boundaries, 3D pose, or occlusion relationships.
- The framework introduces a subcategory-based CNN architecture for region proposal and a detection network for joint detection and subcategory classification.
- Subcategories can encode appearance, 3D pose, or 3D shape, enabling attribute estimation through subcategory classification.
- Experiments on KITTI, PASCAL3D+, and PASCAL VOC 2007 demonstrate advantages for subcategory-aware object recognition.
2. Related Work
Related work covers subcategory discovery for object detection and the one-stage and two-stage organization of CNN-based detectors.
- Subcategory in Object Detection: Prior methods discover subcategories using bounding-box aspect ratios, object viewpoints, visual appearance, or 3D voxel visibility.
- CNN-based Object Detection: CNN detectors are commonly organized as one-stage methods that directly output detections or two-stage methods that first generate region proposals.
3. Subcategory-aware RPN
The subcategory-aware RPN generates proposals from multi-scale subcategory detections. Image pyramids and feature extrapolation address scale variation while reducing feature-computation cost.
- 3. Subcategory-aware RPN: The RPN targets proposal coverage with as few proposals as possible, motivated by the difficulty of objects appearing at different locations and scales.
- 3.1. Region Proposal Network: An image pyramid supplies multi-scale inputs, producing convolutional feature maps at different image resolutions.
- 3.2. Feature Extrapolating Layer: The feature extrapolating layer interpolates intermediate-scale feature maps from nearby scales, accelerating computation with less memory.If N input scales and M intermediate scales are used between adjacent scales, the output contains N′ = (N−1)M + N feature maps.
- 3.3. Subcategory Conv Layer: The subcategory convolutional layer trains separate filters to detect specific subcategories and produces confidence heat maps across scales.
- 3.4. RoI Generating Layer: The RoI generating layer aggregates each heat map by taking the maximum across subcategory channels, then thresholds the result to produce proposals.
- 3.4. RoI Generating Layer: During testing, thresholded heat-map locations generate canonical boxes and boxes with varied aspect ratios, rescaled according to heat-map scale.
- 3.4. RoI Generating Layer: Training selects hard positives with low heat-map scores and hard negatives with high scores under a fixed RoI budget.The positive and negative counts are R × α and R × (1−α), respectively.
- 3.5. Network Training: The RPN applies RoI pooling, then jointly performs subcategory classification and bounding-box regression using sibling output layers.
4. Subcategory-aware Detection Network
The detection network extends Fast R-CNN to use image pyramids and jointly classify objects and subcategories while refining proposal locations. It is trained with a multi-task loss covering object classification, subcategory classification, and bounding-box regression.
- 4. Subcategory-aware Detection Network: Region proposals are processed with RoI pooling before joint subcategory classification and bounding-box regression.The pooled convolutional features support both prediction tasks.
- 4. Subcategory-aware Detection Network: The network uses image pyramids and feature extrapolation to handle object scale variation.Feature extrapolation increases the number of scales in the convolutional feature pyramid.
- 4. Subcategory-aware Detection Network: The detection network is trained with a multi-task loss for object classification, subcategory classification, and bounding-box regression.The loss balances the different task losses using predefined weights.
5. Experiments
Experiments evaluate subcategory-aware CNNs across KITTI, PASCAL3D+, and PASCAL VOC, testing proposal recall, detection, pose, segmentation, and localization. Results show strong gains on challenging KITTI and PASCAL3D+ settings, while performance on PASCAL VOC is comparable to existing methods.
- Experimental Settings: Experiments use KITTI, PASCAL3D+, and PASCAL VOC 2007, with 2D and 3D subcategories evaluated across detection and pose-related tasks.KITTI uses 3DVPs and pose subcategories; PASCAL VOC uses image-feature clusters.
- KITTI Validation: Under a 2k-proposal budget, Selective Search and Edge Boxes perform poorly on KITTI, while the proposed RPN performs on par with Faster R-CNN for cars and better for pedestrians and cyclists.The proposed architecture uses an image pyramid and hard-example mining to handle scale variation.
- KITTI Validation: Using the proposed RPN with the same detection network improves detection and orientation estimation across all three KITTI categories.The RPN and detection network develop separate task-suitable convolutional features; sharing them caused training not to converge.
- KITTI Validation: The full detection model outperforms Faster R-CNN with 2D subcategories, while pose-based subcategories additionally enable orientation estimation and feature extrapolation generally boosts performance.Feature extrapolation causes a minor drop in pedestrian orientation estimation.
- KITTI Validation: On KITTI car evaluation, the method significantly improves 2D segmentation and 3D localization at a 2-meter threshold, whereas 3DVP obtains better localization at 1 meter.The comparison includes DPM and 3DVP.
- KITTI Test Set: On KITTI, the method ranks among the top published methods for detection and orientation estimation on the test set.The evaluation covers cars, pedestrians, and cyclists.
- PASCAL3D+: On PASCAL3D+, the method improves detection over R-CNN across all 12 categories and significantly outperforms VDPM and DPM-VOC+VP for joint detection and pose estimation.Feature extrapolation helps both detection and pose estimation, but limited examples per subcategory constrain pose performance.
- PASCAL VOC 2007: On PASCAL VOC 2007, the method achieves performance comparable to state-of-the-art methods, with little RPN improvement because proposal generation is relatively easy.Subcategories are obtained by clustering image features.
6. Conclusion
The paper proposes subcategory-aware CNNs that use subcategory information in both region proposal generation and detection. Across KITTI, PASCAL3D+, and PASCAL VOC 2007, the method achieves state-of-the-art benchmark results while supporting detection, pose estimation, and related properties.
- Conclusion: The proposed RPN and detection network explicitly use subcategory information to improve proposal generation, object detection, and pose estimation.Image pyramids efficiently address object scale variation.