Source-linked AI summary
Context-aware CNNs for person head detection
Tuan-Hung Vu, Anton Osokin, Ivan Laptev
TL;DR
Person detection remains difficult under varied viewpoints, poses, lighting, and occlusions. The paper adds global-scene and pairwise context to a local CNN detector for head detection. The combined approach achieves state-of-the-art results on three datasets, and the Global model speeds detection.
Problem
Person detection remains challenging under variations in pose, background clutter, motion blur, low resolution, occlusions, and poor lighting.
Source: §1 · p. 1Method
The paper combines a Local CNN with a Global CNN that predicts locations and scales from the full image and a Pairwise CNN that models image-conditioned relations among object hypotheses.
Source: §1 · p. 2Results
The context-aware CNN model with the local detector achieves state-of-the-art results on HollywoodHeads, TVHI, and Casablanca.
Source: §6 · p. 8Takeaways & Limitations
The Global model provides a speed-up for object detection, and the introduced HollywoodHeads dataset contains 369,846 annotated heads in 224,740 frames.
Source: §1 · p. 2The Casablanca evaluation differs from because it uses improved annotations and the VOC evaluation procedure.
Source: §5.2 · p. 7
Abstract
from arXiv · showhide
Person detection is a key problem for many computer vision tasks. While face detection has reached maturity, detecting people under a full variation of camera view-points, human poses, lighting conditions and occlusions is still a difficult challenge. In this work we focus on detecting human heads in natural scenes. Starting from the recent local R-CNN object detector, we extend it with two types of contextual cues. First, we leverage person-scene relations and propose a Global CNN model trained to predict positions and scales of heads directly from the full image. Second, we explicitly model pairwise relations among objects and train a Pairwise CNN model using a structured-output surrogate loss. The Local, Global and Pairwise models are combined into a joint CNN framework. To train and test our full model, we introduce a large dataset composed of 369,846 human heads annotated in 224,740 movie frames. We evaluate our method and demonstrate improvements of person head detection against several recent baselines in three datasets. We also show improvements of the detection speed provided by our model.
1. Introduction
Person detection remains challenging under varied poses, clutter, blur, low resolution, occlusion, and poor lighting. The paper extends a local CNN detector with scene-level and pairwise context, and reports improved detection alongside a new large dataset.
1. Introduction
- People account for about 35% of pixels in movies and YouTube videos and about 25% in photographs, motivating reliable person analysis in visual data.Source: §1 · p. 1
- Person detection remains challenging despite mature face detection, with state-of-the-art detectors reaching only 65% Average Precision for the person class on Pascal VOC.Difficulties include pose variation, background clutter, motion blur, low resolution, occlusions, and poor lighting.Source: §1 · p. 1
- The paper targets head detection because other body parts are often occluded, while visible body parts, scene context, and interactions constrain head locations and appearance.Source: §1 · p. 2
- The Global CNN predicts coarse object locations and scales from the full low-resolution image, while the Pairwise CNN models image-conditioned relations among object hypotheses.The Pairwise model scores relative positions, scales, and head appearance using a structured-output loss; the final joint model combines both with the Local CNN.Source: §1 · p. 2
- The contextual CNN model improves on recent baselines, including R-CNN, across three datasets, while its Global model speeds up object detection.The paper also introduces a dataset of 369,846 heads annotated in 224,740 frames from 21 movies.Source: §1 · p. 2
2. Related works
Prior object-detection work includes CNN-based detectors and approaches that use global scene context or object relations. The paper positions its Global model among scene-level methods and its Pairwise model against approaches with image-independent pairwise terms or separately trained components.
2. Related works
- R-CNN combines a CNN and SVM on selective-search object proposals, and its pipeline is similar to the paper's Local model.CNN object detection began outperforming methods such as DPM after Krizhevsky et al..Source: §2 · p. 2
- Global context has been used for scene classification, object-location prediction, and detection speed-up, while this paper's Global CNN predicts likely locations and scales with a multi-scale heat map.Source: §2 · p. 2
- Earlier spatial-constellation models use unary and pairwise potentials or latent preferred configurations, but their binary potentials are not image-dependent and their unary terms are trained independently.The paper's Pairwise model conditions pairwise dependencies on image data and jointly trains the base detector and graphical model.Source: §2 · p. 2
- Prior structured-prediction CNN work includes dual message passing, structured SVM training, and pose models with separately trained parts or message-passing networks.The paper instead uses an explicit structured-output surrogate loss with external inference to jointly fine-tune model parameters.Source: §2 · p. 2
3. Context-aware CNN model
The section extends an R-CNN-style Local detector with full-image context and pairwise object relations, combining the models for head detection. It describes how the Global and Pairwise models score candidates and how the pairwise model is trained.
3.1. Local model
- The Local model follows R-CNN with selective-search proposals, context padding, and square-like head candidates, and scores candidates with CNN outputs rather than a second-pass SVM.The proposals are restricted to bounding boxes with aspect ratios from 2/3 to 3/2.Source: §3.1 · p. 3
3.2. Global model
- Global CNN scores 284 multi-scale image-grid cells for coarse object locations and scales, then rescores Local and Pairwise candidates because its localization is coarse.A cell is labeled as a head cell when its box has more than 0.3 IoU with any ground-truth box.Source: §3.2 · p. 3
3.3. Pairwise model
- The Pairwise model enriches prior pairwise scoring by making all potentials image-dependent and allowing joint training of the model parameters.It uses SGD to minimize a structured surrogate loss.Source: §3.3 · p. 3
3.3.1 Model formulation
- A joint score function assigns binary object or background labels to candidate boxes using image-dependent unary and pairwise potentials produced from candidate features.The feature extractor supplies candidate features to separate unary and pairwise networks.Source: §3.3.1 · pp. 3–4
3.3.2 Training the model
- The proposed surrogate loss penalizes incorrect individual candidate scores from the joint score, addressing the authors’ finding that SSVM optimization did not yield good precision-recall results.Training uses SGD and back-propagates through the unary, pairwise, and feature-extractor networks; the described gradient procedure assumes candidate scores can be computed exactly.Source: §3.3.2 · pp. 5–6
4. Datasets
This section introduces HollywoodHeads and describes TVHI and Casablanca as additional evaluation datasets. Their annotation formats and evaluation adjustments differ, so the datasets provide distinct settings for head detection.
4.1. HollywoodHeads dataset
- HollywoodHeads contains 369,846 annotated heads across 224,740 frames from 21 movies, with movie-disjoint splits; poorly visible heads marked difficult are excluded from evaluation.The training, validation, and test splits contain 216,719, 6,719, and 1,302 frames, respectively.Source: §4.1 · p. 6
4.2. TVHI dataset
- TVHI has 1,313 frames annotated with upper-body boxes; head-detector outputs are regressed using parameters tuned separately for each method on the training subset.The split is 599 training frames and 714 test frames.Source: §4.2 · p. 6
4.3. Casablanca dataset
- Casablanca contains 1,466 movie frames; its frontal-head annotations often use face-box scale and aspect ratios, so missing foreground heads were added and bounding-box regression applied.The regression compensates for differences in annotation policies.Source: §4.3 · p. 6
5. Experiments
The experiments compare context-aware model combinations with baselines across three datasets, assess local architectures and training-set size, and test Global-model filtering. The joint model performs best across the three datasets, while Global scores also support candidate filtering with high detection performance under aggressive filtering.
5. Experiments
- The experiments evaluate detection with Average Precision, counting detections at IoU > 0.5 as true positives and ignoring matches to difficult annotations.The section covers model combinations, state-of-the-art comparisons, local architectures, training-set size, and Global-model complexity reduction.Source: §5 · p. 6
5.1. Results of context-aware models
- The Global and Pairwise models each improve the Local baseline, while combining all three models gives the best performance on all three datasets.Table 1 reports performance for the model combinations on Casablanca, HollywoodHeads, and TVHI.Source: §5.1 · p. 6
5.2. Comparison with the state-of-the-art methods
- On Casablanca, comparisons with use improved annotations and the VOC evaluation procedure; results under Ren's original setup are reported in Appendix B.Source: §5.2 · p. 7
- Our joint model outperforms all compared methods on all three datasets, significantly beating R-CNN on two and performing slightly better on TVHI.The Casablanca evaluation differs from because the annotations were improved and the VOC evaluation procedure was used.Source: §5.2 · p. 7, Figure 5 · p. 8
5.3. Architectures of the Local model.
- Local architectures are compared by AP and speed, measured as image patches processed per second, after ImageNet pretraining and HollywoodHeads fine-tuning.The tested architectures include AlexNet, VGG-S, VGG-verydeep-16, and Oquab et al.'s model; training batch sizes maximize speed within GPU memory.Source: Table 2 · p. 8, §5.3 · p. 8
5.4. Size of the training set
- Training on the full 15-movie set, rather than the first 8 or 4 movies, improves performance; the authors identify training-data amount and diversity as helpful.Table 3 reports AP for each test set across the training-set sizes.Source: §5.4 · p. 8, Table 3 · p. 8
5.5. Complexity reduction with the Global model
- Global-model scores filter low-scoring candidates before Local CNN processing, and detection performance remains high despite aggressive filtering.Table 4 reports R-CNN and Local-model AP on the HollywoodHeads test set at different candidate-retention percentages.Source: §5.5 · p. 8, Table 4 · p. 8
6. Conclusion
The combined context-aware models and local detector achieve state-of-the-art results on HollywoodHeads, TVHI, and Casablanca. The authors identify extension to general object classes and use of motion for long-term tracking as future directions.
6. Conclusion
- The combined context-aware models and CNN-based local detector achieve state-of-the-art results on HollywoodHeads, TVHI, and Casablanca.The authors also collected the large-scale HollywoodHeads dataset of movie frames with human-head annotations.Source: §6 · p. 8
- Future directions include extending the context-aware models to general object classes and incorporating motion information for long-term tracking.The authors point to spatial constraints among small COCO object classes as one possible setting.Source: §6 · p. 8
A. Implementation details
The implementation trains local, global, and pairwise CNN components, then combines their scores. The pairwise stage uses a limited set of proposals and features describing candidate-box relationships.
A.1. Local model
- Candidates with IoU above 0.6 are positive and below 0.5 are negative; ambiguous proposals are excluded, and patches include 18 pixels of surrounding context.Each patch is resized to 188 × 188 before context padding to the 224 × 224 CNN input.Source: §A.1 · p. 9
A.2. Global model
- The Global model processes the whole resized, zero-padded image and outputs 284 scores, with each grid cell trained positive when its patch reaches at least 0.3 IoU with a ground-truth box.The model is trained with independent log-losses and stochastic gradient descent.Source: §A.2 · p. 9
A.3. Pairwise model
- Because inference limits candidate count, the Pairwise model uses 16 Local-model detections after non-maximum suppression; its edge features encode relative box size and horizontal and vertical displacement.The three features are normalized using training-set means and standard deviations, and increasing the number of clusters beyond 20 did not improve performance.Source: §A.3 · p. 9
- Pairwise training assigns binary targets using a 0.5 IoU threshold and optimizes a structured surrogate objective with batches of 64 candidates from four images.The feature extractor is initialized from the Local model, while the unary and pairwise networks are randomly initialized.Source: §A.3 · p. 9
A.4. Combining models
- The Local and Pairwise scores are affinely combined, using the Local score alone when no Pairwise score exists; the resulting score is then combined with the Global grid-cell score.Combination weights are selected by maximizing validation-set average precision.Source: §A.4 · p. 9
A.5. Implementation details
The implementation reports component-level training runtimes on an NVIDIA TITAN X, including the combined Pairwise model runtime.
A.5. Implementation details
- 0.08s forward and 0.18s backward for the Local model on a batch of 64 proposals.The measurements were taken on an NVIDIA TITAN X.Source: §A.5 · p. 10
- 0.06s forward and 0.12s backward for the Global model on a batch of 32 images.The measurements were taken on an NVIDIA TITAN X.Source: §A.5 · p. 10
- 0.36s for the Pairwise model's forward and backward passes on a batch of 64 candidates from four images.The feature extractor, unary network, pairwise network, and structured loss contribute to this combined runtime.Source: §A.5 · p. 10
B. Evaluation on the original Casablanca dataset
Under the original Casablanca annotations and evaluation procedure, the method outperforms VJ-CRF and other baselines. The authors improved the test annotations but report that some crowded scenes may still lack annotations for very small heads.
B. Evaluation on the original Casablanca dataset
- The method significantly outperforms VJ-CRF and other baselines on Casablanca under the original annotations and evaluation procedure.Figure 6 presents the corresponding precision-recall curves.Source: B. Evaluation on the original… · p. 10
- Some crowded scenes may still contain missing annotations of very small heads, despite adding missing labels and correcting existing test-frame annotations.The annotation improvements were made to the test frames defined in the original Casablanca dataset.Source: B. Evaluation on the original… · p. 10
C.1. Global model
The Global model produces multi-scale score grids with high responses at human-head positions and scales, while the appendix also presents corrected Casablanca annotations and qualitative Pairwise model results.
C.1. Global model
- The Global model outputs four score-grid scales, with high responses at positions and scales corresponding to human heads.The grids are 1×1, 3×3, 7×7 and 15×15, corresponding to cell grids of 28×28, 56×56, 112×112 and 224 × 224 pixels.Source: §C.1 · p. 10
- Figure 6 presents head-detection results on the original Casablanca dataset.Source: Figure 6 · p. 10
- The Casablanca examples compare original annotations with corrected annotations that add head bounding boxes marked in cyan.Source: Figure 7 · p. 10
C.2. Pairwise model
- Figure 9 shows qualitative Pairwise model results using the same bounding-box and link meanings and the same link and candidate thresholds as Figure 4.Source: §C.2 · p. 10