Source-linked AI summary
Mask-Guided Attention Network for Occluded Pedestrian Detection
Yanwei Pang, Jin Xie, Muhammad Haris Khan, Rao Muhammad Anwer, Fahad Shahbaz Khan, Ling Shao
TL;DR
Severe occlusion remains a challenge because pedestrian detectors trained on full bodies must handle invisible regions and diverse occlusion patterns. MGAN uses visible-region-guided spatial attention to modulate full-body features, and it achieves improved heavily occluded detection on CityPersons and Caltech.
Problem
Severe occlusions caused by pedestrians and other objects remain difficult for pedestrian detectors, despite progress on standard benchmarks.
Method
MGAN generates spatial attention from visible body information, modulates full-body features, and uses coarse-level segmentation annotations instead of dense pixel labels.
Results
MGAN improves the CityPersons HO log-average miss rate from 44.2 to 39.4 under matched supervision, input scale, and backbone conditions.
Takeaways & Limitations
Experiments on CityPersons and Caltech show MGAN is effective, especially for heavily occluded pedestrians.
Takeaways & Limitations
Training a detector solely with visible-region annotations can reduce miss rate but produces high false-positive detections.
Abstract
from arXiv · showhide
Pedestrian detection relying on deep convolution neural networks has made significant progress. Though promising results have been achieved on standard pedestrians, the performance on heavily occluded pedestrians remains far from satisfactory. The main culprits are intra-class occlusions involving other pedestrians and inter-class occlusions caused by other objects, such as cars and bicycles. These result in a multitude of occlusion patterns. We propose an approach for occluded pedestrian detection with the following contributions. First, we introduce a novel mask-guided attention network that fits naturally into popular pedestrian detection pipelines. Our attention network emphasizes on visible pedestrian regions while suppressing the occluded ones by modulating full body features. Second, we empirically demonstrate that coarse-level segmentation annotations provide reasonable approximation to their dense pixel-wise counterparts. Experiments are performed on CityPersons and Caltech datasets. Our approach sets a new state-of-the-art on both datasets. Our approach obtains an absolute gain of 9.5% in log-average miss rate, compared to the best reported results on the heavily occluded (HO) pedestrian set of CityPersons test set. Further, on the HO pedestrian set of Caltech dataset, our method achieves an absolute gain of 5.0% in log-average miss rate, compared to the best reported results. Code and models are available at: https://github.com/Leotju/MGAN.
1. Introduction
Severe occlusion remains a major weakness for pedestrian detectors because invisible body regions and background within full-body windows reduce discriminative features. MGAN addresses this with visible-region-guided attention and improves heavily occluded detection on CityPersons.
- Severe occlusions remain difficult despite progress on standard pedestrian benchmarks.
- Occlusions from nearby pedestrians, cars, and bicycles create diverse patterns, while full-body training includes invisible body and background regions.
- Existing part-based methods can be computationally expensive or difficult to train, whereas MGAN avoids explicitly using part information.
- MGAN uses visible body information to generate spatial attention that modulates full-body features and suppresses occluded regions.
- From 44.2 to 39.4 in log-average miss rate, MGAN improves the prior state of the art on CityPersons HO pedestrians with 35–80% occlusion.
2. Related Work
Prior work handles occlusion through part detectors or visible-region information, but these strategies involve computational, training, or pattern-specific limitations. MGAN instead applies visible-region-guided spatial attention within a standard detector.
- Deep Pedestrian Detection: Deep pedestrian detectors use either two-stage proposal-and-classification pipelines or one-stage joint regression architectures.
- Occlusion Handling in Pedestrian Detection: Part-based occlusion methods learn detectors for specific patterns, but ensembles are computationally expensive and joint part learning requires score fusion.
- Occlusion Handling in Pedestrian Detection: Visible-region methods either regress visible parts for proposal generation or use visible information as external supervision for occlusion modes.
- Occlusion Handling in Pedestrian Detection: MGAN generates a pixel-wise attention map from visible bounding-box information to modulate multichannel full-body features.
- Occlusion Handling in Pedestrian Detection: 4.8% and 5.6% gains on the CityPersons HO set distinguish MGAN from and, respectively, under matched supervision, scale, backbone, and training data.
3. Proposed Approach
MGAN augments a standard full-body pedestrian detector with a lightweight mask-guided attention branch that emphasizes visible regions and suppresses occluded features. It uses coarse visible-region annotations to supervise spatial masks and jointly trains detection, masking, and occlusion-sensitive objectives.
- Mask-Guided Attention Network: MGAN adds a lightweight Mask-Guided Attention branch that produces pixel-wise attention maps for modulating full-body features.The branch is integrated into the standard pedestrian detection pipeline for end-to-end training.
- Motivation: Standard full-body detectors struggle with partial and heavy occlusions because occluding regions alter features contributing to proposal scores.Training only on visible-region annotations can reduce misses but produces high false-positive detections.
- MGA Architecture: The MGA branch takes RoI features, applies convolutional operations and ReLU nonlinearities, and outputs modulated multichannel features.A sigmoid layer produces the spatial attention probability map used for feature modulation.
- Feature Modulation: Element-wise multiplication between the spatial attention mask and each RoI feature channel emphasizes visible regions while concealing occluded parts.The resulting modulated features are fed to the classification network for proposal scoring.
- Coarse-level Segmentation Annotation: Visible-region bounding boxes are converted into coarse binary segmentation labels as an alternative to tedious dense pixel-wise annotations.Pixels inside the visible-region box receive label one, and pixels outside receive label zero; the paper reports accurate masks from this weak supervision.
- Loss Function: The overall MGAN objective jointly optimizes the Faster R-CNN loss, mask loss, and occlusion-sensitive loss.The mask loss uses per-pixel binary cross-entropy, while the occlusion-sensitive term weights proposals according to occlusion levels.
4. Experiments
Experiments on CityPersons and Caltech evaluate MGAN using log-average miss rate across occlusion subsets and compare it with baselines and state-of-the-art methods. MGAN consistently improves detection, particularly under heavy occlusion, while coarse-level segmentation closely approximates dense labeling.
- Datasets and metrics: Experiments use CityPersons and Caltech, reporting log-average miss rate over FPPI=[10^-2, 100] across reasonable and heavily occluded subsets.CityPersons evaluation includes R and H sets, while Caltech evaluation includes R, HO, and R+HO sets.
- Ablation study: 5.3% absolute reduction in log-average miss rate is achieved by MGAN over the SPD baseline on CityPersons’ HO set.Under the same training-example selection, MGAN reduces the baseline HO error from 57.0% to 51.7%.
- Ablation study: Coarse-level segmentation produces similar detection results to dense pixel-wise labeling, offering a trade-off between annotation cost and accuracy.Dense pixel-wise labels are expensive and unavailable for Caltech, whereas visible bounding-box information supplies coarse supervision.
- State-of-the-art comparison: MGAN outperforms ATT-vbb, OR-CNN, and Bi-Box on CityPersons validation subsets when training data, supervision, input scale, and backbone are controlled.Against Bi-Box at input scale 1.3×, MGAN achieves 10.5% versus 11.2% on R and 39.4% versus 44.2% on HO.
- Caltech dataset: 5.0% improvement over GDFL is reported on Caltech’s HO set, with MGAN also outperforming published methods on R and R+HO.Figure 8 compares log-average miss rates over FPPI=[10^-2, 100] for R, HO, and R+HO.
- State-of-the-art comparison: 5.02% and 1.80% absolute gains over GDFL are reported by MGAN on CityPersons HO and R+HO sets, respectively.MGAN also reports 6.83% on the R set, compared with SDS-RCNN’s 7.36%.
5. Conclusion
MGAN combines mask-guided attention, coarse visible-region segmentation, and an occlusion-sensitive loss for occluded pedestrian detection. Experiments on two datasets show effectiveness, especially for heavily occluded pedestrians.
- MGAN uses a mask-guided attention network whose spatial mask highlights visible pedestrian regions and suppresses background in full-body features.The mask is generated from visible body region information.
- The method uses coarse-level segmentation information instead of dense pixel labeling for visible regions.
- MGAN introduces an occlusion-sensitive loss term alongside the mask-guided attention module.
- Experiments on two datasets show the approach is effective, especially for heavily occluded pedestrians.