Source-linked AI summary
Illuminating Pedestrians via Simultaneous Detection & Segmentation
Garrick Brazil, Xi Yin, Xiaoming Liu
TL;DR
Pedestrian detection is important for applications such as autonomous driving, but combining detection with segmentation is limited by scarce pixel-wise annotations in classic pedestrian datasets. The paper proposes SDS-RCNN, which infuses segmentation supervision into shared detector features using weak pedestrian-box annotations. It reports improved Caltech performance, competitive KITTI performance, and roughly 2× faster runtime than competitive methods.
Problem
Pedestrian detection is challenging, while its pairing with semantic segmentation is rarely studied partly because classic pedestrian datasets lack pixel-wise annotations.
Method
SDS-RCNN jointly supervises pedestrian detection and semantic segmentation by infusing segmentation masks into shared layers in both detector stages.
Results
23% relative reduction of the error over the published state-of-the-art is reported on Caltech, with competitive KITTI performance and roughly 2× faster runtime than competitive methods.
Takeaways & Limitations
Weakly annotated pedestrian boxes are sufficient for considerable gains, while infused shared feature maps become more semantically meaningful and robust to pedestrian shape and occlusion.
Takeaways & Limitations
At the highly pooled conv5 representations used by the networks, pixel-wise annotations may not offer a significant advantage over box annotations.
Abstract
from arXiv · showhide
Pedestrian detection is a critical problem in computer vision with significant impact on safety in urban autonomous driving. In this work, we explore how semantic segmentation can be used to boost pedestrian detection accuracy while having little to no impact on network efficiency. We propose a segmentation infusion network to enable joint supervision on semantic segmentation and pedestrian detection. When placed properly, the additional supervision helps guide features in shared layers to become more sophisticated and helpful for the downstream pedestrian detector. Using this approach, we find weakly annotated boxes to be sufficient for considerable performance gains. We provide an in-depth analysis to demonstrate how shared layers are shaped by the segmentation supervision. In doing so, we show that the resulting feature maps become more semantically meaningful and robust to shape and occlusion. Overall, our simultaneous detection and segmentation framework achieves a considerable gain over the state-of-the-art on the Caltech pedestrian dataset, competitive performance on KITTI, and executes 2x faster than competitive methods.
1. Introduction
Pedestrian detection faces challenges that object detection and semantic segmentation address differently, while their combination remains underexplored because classic datasets lack pixel-wise annotations. SDS-RCNN infuses segmentation supervision into shared features, improving detection across benchmarks without reducing inference efficiency.
- Pedestrian detection supports applications including autonomous driving and robotics but remains challenging under low resolution, occlusion, and clothing variation.
- Object detection localizes distinct objects, whereas semantic segmentation distinguishes pixel-wise class boundaries but struggles to separate instances within one class.
- The detection–segmentation pairing is rarely studied for pedestrians partly because Caltech and KITTI lack the pixel-wise annotations available in COCO.
- The framework combines stricter second-stage supervision and score fusion with multi-task detection and segmentation supervision.These components are presented as contributions alongside the improved baseline.
- SDS-RCNN infuses semantic segmentation masks into shared layers in both network stages, shaping feature maps that illuminate pedestrians and suppress background regions.The infusion occurs during training only, leaving inference efficiency unaffected.
- 23% relative reduction of the error over the published state-of-the-art is reported on Caltech, alongside competitive KITTI performance and roughly 2× faster runtime than competitive methods.
2. Prior work
Prior work uses detection and segmentation either jointly for instance-aware segmentation or uses segmentation as a cue to improve detection. SDS-RCNN extends this direction with segmentation infusion in a two-stage pedestrian detector.
- Faster R-CNN-style systems use a region proposal network followed by a classification sub-network, with proposals generated from predefined anchors.
- Caltech's strongest approaches include multi-scale Fast or Faster R-CNN variants and RPN+BF, which uses the RPN as a standalone detector because downstream classification can degrade small-pedestrian performance.
- One simultaneous detection-and-segmentation research line targets instance-aware semantic segmentation through mask prediction and object-instance extraction.
- A second line uses segmentation masks as strong cues for object detection, including feature extraction, ensemble post-processing, and deformable part-based models.
- SDS-RCNN places semantic segmentation infusion layers in both the RPN and binary classification network to encode masks into shared feature maps for pedestrian classification.
3. Proposed method
The proposed detector uses a two-stage RPN–BCN architecture, with segmentation infusion in shared layers of both stages to provide semantic cues for pedestrian classification. Weak box-derived masks supervise the infusion layers, while stricter second-stage labeling and score fusion improve proposal refinement and complementarity.
- Two-stage detection: The architecture combines an RPN for candidate boxes and scores with a BCN that refines proposal scores.The RPN uses Faster R-CNN-style proposal generation, while the BCN performs pedestrian classification over RPN proposals.
- Region Proposal Network: The RPN jointly optimizes pedestrian classification, bounding-box regression, and segmentation losses.Classification uses pedestrian-versus-background labels at IoU 0.5, regression uses smooth L1 loss, and the third term is segmentation loss.
- Binary Classification Network: The BCN uses cost-sensitive classification and segmentation supervision, weighting larger pedestrians more heavily than smaller ones.The weight depends on proposal height relative to the pre-computed mean height, reflecting their stated detection priority and presumed feature usefulness.
- Binary Classification Network: The BCN requires IoU > 0.7 for positive labels and fuses its scores with RPN scores to suppress poorly localized proposals and diversify stage responses.The fused score becomes more confident when stages agree and otherwise leans toward the dominant score.
- Implementation choices: The BCN removes pool5, resizes inputs to 112 × 112, and pads proposals by 20% to trade modest efficiency gains for finer discrimination and background context.Inference uses 15 proposals per image, compared with 20 during training; the padding benefit is described as minor in practice.
- Simultaneous Detection & Segmentation: Segmentation infusion uses a single 1 × 1 layer attached to conv5 in both stages, directly encoding pedestrian and background mask likelihoods into shared feature maps.The placement is intended to maximize the effect on shared layers while preceding the RPN proposal layers and BCN fully connected layers.
- Simultaneous Detection & Segmentation: Weak masks label pedestrian bounding-box regions as foreground and become less different from pixel-wise masks after conv5 pooling.At conv5, Caltech pedestrians below 80 pixels correspond to 3 × 5 RPN feature pixels, while BCN proposals are pooled to 7 × 7.
4. Experiments
Experiments show that SDS-RCNN improves pedestrian detection across Caltech and KITTI while maintaining substantially higher efficiency. Ablations attribute the gains primarily to semantic segmentation infusion, with additional effects from supervision, stage fusion, and feature-sharing choices.
- Benchmark Comparison: 7.36% miss rate on Caltech, a 23% relative improvement over RPN+BF's 9.58%.The comparison uses the reasonable setting.
- Benchmark Comparison: 9.63% miss rate for SDS-RPN alone, at roughly 3× the speed of competitive methods.The standalone RPN is reported as comparable to top detectors and better than another standalone RPN at 14.9%.
- Benchmark Comparison: 63.05 mAP on KITTI's moderate pedestrian setting, demonstrating competitive cross-dataset performance.KITTI evaluation uses mean Average Precision sampled across recall [0, 1].
- Benchmark Comparison: Approximately 0.21 seconds per image, with the design generally running 2× faster than competitive methods.The efficiency comparison accompanies the cross-dataset evaluation.
- Ablation Study: Weak segmentation is the most critical component: disabling it worsens fused miss rate by a full 3.05%.Both individual stages also degrade when segmentation supervision is disabled.
- Ablation Study: About 48% of corrected pedestrians were at least partially occluded and about 28% had unusual poses, indicating robustness to atypical shapes.The analysis covers 43 cases missed without weak segmentation but corrected with it; together, these categories account for 75% of corrections.
5. Conclusion
SDS-RCNN jointly supervises pedestrian detection and semantic segmentation to improve shared features and downstream detection. It reports stronger robustness, state-of-the-art Caltech performance, competitive KITTI results, and faster runtime.
- The segmentation infusion layer produces more sophisticated shared feature maps that illuminate pedestrians and ease downstream detection.The framework infuses segmentation masks into shared layers in both network stages.
- Joint training makes the network more robust to pedestrian poses and occlusion than training without segmentation infusion.The paper analyzes how infused segmentation masks help correct pedestrian detection errors.
- 23% relative reduction in error establishes new state-of-the-art performance on the Caltech pedestrian dataset.
- The framework achieves competitive results on KITTI and runs approximately 2× faster than competitive methods.