Source-linked AI summary
End-to-end people detection in crowded scenes
Russell Stewart, Mykhaylo Andriluka
TL;DR
Crowded-scene people detection is difficult for methods that independently generate or classify boxes and then merge overlapping detections. The paper instead encodes an image and uses an LSTM with a set-based loss to jointly decode distinct detections, achieving stronger reported results than OverFeat on its crowded-scenes evaluation.
Problem
Existing detectors often struggle with nearby or occluded people because they treat boxes independently and rely on box-based merging or non-maximum suppression.
Method
The model encodes an image into high-level features and uses an LSTM to decode a variable-length set of bounding boxes, trained with matching and confidence-ordering losses.
Results
The approach improves recall from 71% to 81%, AP from 0.67 for OverFeat-GoogLeNet to 0.78, and people counting error from 1.05 to 0.76.
Takeaways & Limitations
Joint sequence generation can produce coherent sets of detections for crowded, partially occluded people without requiring merging or non-maximum suppression.
Abstract
from arXiv · showhide
Current people detectors operate either by scanning an image in a sliding window fashion or by classifying a discrete set of proposals. We propose a model that is based on decoding an image into a set of people detections. Our system takes an image as input and directly outputs a set of distinct detection hypotheses. Because we generate predictions jointly, common post-processing steps such as non-maximum suppression are unnecessary. We use a recurrent LSTM layer for sequence generation and train our model end-to-end with a new loss function that operates on sets of detections. We demonstrate the effectiveness of our approach on the challenging task of detecting people in crowded scenes.
1 Introduction
The paper proposes an end-to-end detector that jointly decodes images into distinct object detections, avoiding proposal merging and non-maximum suppression. It targets crowded scenes where overlapping people challenge conventional detection pipelines.
- 1 Introduction: The architecture directly maps an image to a set of object bounding boxes, rather than separately classifying proposals and post-processing them.This design jointly predicts objects and can remember earlier outputs to avoid duplicate detections.
- 1 Introduction: The system uses high-level CNN features and an LSTM decoder to generate variable-length detection sequences with jointly predicted boxes.The LSTM serves as a controller that decodes an image representation into detections.
- 1 Introduction: Overlapping instances expose a weakness of box-only merging because deciding box placement and object count requires image information.The proposed architecture is generic rather than specialized for predefined object constellations such as pedestrian pairs.
- 1 Introduction: The paper frames crowded-scene people detection as a test of jointly predicting nearby individuals under occlusion.The authors also identify people tracking and multi-person articulated pose estimation as possible structured-output applications.
- 1.1 Related work: A set-based loss matches unordered predictions to ground-truth instances while encouraging detections to appear in descending confidence order.This combines permutation-aware matching with sequence ordering needed for confidence-threshold stopping.
2 Model
The model encodes images with deep convolutional features and decodes them into variable-length sequences of jointly predicted bounding boxes. A set-based matching loss trains these predictions to balance overlap, sequence rank, localization, and confidence.
- 2.1 Architecture: An image encoder produces high-level descriptors that an LSTM decoder converts into a set of bounding boxes.The model transforms each image into a grid of 1024-dimensional feature descriptors before recurrent decoding.
- 2.1 Architecture: Each recurrent step outputs a box location and confidence score, with low confidence serving as a test-time stopping signal.The box location has four relative coordinates, while confidence is a real value in [0,1].
- 2.2 Loss function: The matching algorithm assigns each ground-truth instance a unique candidate, distinguishing localization errors from false or duplicate detections.Candidates are matched through an injective function from ground-truth boxes to predicted candidates.
- 2.2 Loss function: Matching prioritizes sufficient overlap, then earlier prediction rank, and finally fine-grained localization distance.The comparison function encodes these criteria lexicographically, with overlap as the dominant term.
- 2.2 Loss function: The Hungarian algorithm finds the minimum-cost bipartite matching, while the resulting Hungarian loss is compared with fixed-order and first-k alternatives.The Hungarian loss combines set matching with localization and confidence training; experiments report that it performs best.
3 Implementation details
Training uses region-based prediction with shared recurrent decoding and end-to-end backpropagation, then stitches regional outputs into full-image detections. The implementation includes pretrained, fine-tuned image features, regularization, and a webcam-derived crowded-people dataset.
- 3 Implementation details: Training predicts distinct boxes on center 64x64 regions using a 15x20 grid of 1024-dimensional GoogLeNet features.Each feature cell has a 139x139 receptive field, while the prediction region is chosen to capture local occlusion interactions.
- Training: The decoder is trained on a fixed overcomplete number of predictions, assigning high confidence to ground-truth boxes and low confidence elsewhere.After generation, ground-truth instances are matched to predictions and the Hungarian loss is backpropagated through the full network.
- Training: Fine-tuning GoogLeNet features is critical, while decoder weight tying, dropout, and L2 regularization materially affect average precision.Tying output weights reduced AP from .85 to .82, and training without feature fine-tuning reduced AP by .29.
- Stitching: At test time, the model processes regions across full 480x640 images and recursively stitches new predictions into the accepted set.Decoding continues in each region until a stop symbol is produced before regional predictions are merged.
- Stitching: The stitching matcher uses non-intersection and L1 box distance to accept candidates that do not overlap their matched accepted boxes.A Hungarian minimum-cost matching is used for this regional merge process.
4 Experimental results
The evaluation compares the proposed LSTM-based decoding component with OverFeat baselines using the same image representation, showing stronger detection performance on crowded-scene images. The experiments also examine failure cases and demonstrate that the set-based loss design is important for calibrated detections.
- Dataset and evaluation: 11,917 images with 91,146 labeled people form the crowded-scene dataset, with 1,000 images reserved for testing and validation.Images were sampled from public-webcam video at fixed 100-second intervals, with no temporal overlaps between splits.
- Dataset and evaluation: The dataset includes small people, strong partial occlusions, and substantial clothing and appearance variation; detections count as correct above 0.5 intersection-over-union.Performance is summarized using average precision and equal error rate from recall-precision curves.
- Baselines and comparison: OverFeat-GoogLeNet performs significantly better than OverFeat-AlexNet, so it is used as the principal baseline for subsequent comparisons.The two OverFeat variants differ in their image representation, while the proposed model uses the GoogLeNet representation.
- Baselines and comparison: The proposed model and OverFeat use identical image representations, enabling a direct comparison of LSTM decoding against bounding-box regression followed by non-maximum suppression.The proposed decoder produces a variable-length output sequence.
- Performance: 81% recall versus 71%, AP 0.78 versus 0.67, and people counting error 0.76 versus 1.05 show improvements over OverFeat-GoogLeNet.The reported examples highlight detections under strong occlusion, while Figure 7 presents typical failure cases.
- Loss-function analysis: Lfix achieves only 0.60 AP, while Lfirstk improves over Lfix but produces poorly calibrated confidences, underscoring the importance of overlap terms in the matching function.The loss comparison attributes the difference to allowing easy-to-hard output ordering and permutation during training, while overlap terms help attach confidences to specific box locations.
5 Conclusion
The paper introduces an end-to-end object detector that decodes rich image representations into variable numbers of predictions for crowded scenes. Its set-based loss trains coherent outputs, and the authors suggest extending the approach to other structured prediction tasks.
- Conclusion: The method detects multiple partially occluded instances by decoding a variable number of outputs from rich intermediate image representations.It is demonstrated on a large dataset of crowded scenes.
- Conclusion: A set-based loss function trains the system end-to-end to produce coherent sets of predictions.The loss is designed for the structured output of multiple objects.
- Conclusion: The authors suggest that the approach may also be effective for structured outputs such as people tracking and articulated pose estimation.These applications are presented as possible extensions rather than evaluated results.