Source-linked AI summary
Pedestrian Detection aided by Deep Learning Semantic Tasks
Yonglong Tian, Ping Luo, Xiaogang Wang, Xiaoou Tang
TL;DR
Pedestrian detectors struggle to distinguish people from visually similar hard negatives because conventional deep models mainly learn middle-level features. The paper introduces TA-CNN, which jointly learns pedestrian and scene semantics from multiple datasets while coordinating task weights and dataset alignment. It reports lower miss rates than prior deep models on Caltech and ETH, with further gains from structural and unshared attributes.
Problem
Existing pedestrian detectors can confuse pedestrians with visually similar hard negatives, motivating high-level representations beyond middle-level pose features.
Method
TA-CNN jointly learns pedestrian detection, pedestrian attributes, and transferred scene attributes from multiple datasets using weighted multitask optimization and dataset-bridging features.
Results
17 and 5.5 percent reductions in miss rates are reported on Caltech and ETH, respectively, compared with previous deep models.
Takeaways & Limitations
Joint semantic tasks and multiple data sources enable TA-CNN to learn high-level features that improve pedestrian detection and support background-attribute prediction.
Takeaways & Limitations
Task importance coefficients assume equal task contributions initially, with the main pedestrian task fixed at coefficient 1 throughout learning.
Abstract
from arXiv · showhide
Deep learning methods have achieved great success in pedestrian detection, owing to its ability to learn features from raw pixels. However, they mainly capture middle-level representations, such as pose of pedestrian, but confuse positive with hard negative samples, which have large ambiguity, e.g. the shape and appearance of `tree trunk' or `wire pole' are similar to pedestrian in certain viewpoint. This ambiguity can be distinguished by high-level representation. To this end, this work jointly optimizes pedestrian detection with semantic tasks, including pedestrian attributes (e.g. `carrying backpack') and scene attributes (e.g. `road', `tree', and `horizontal'). Rather than expensively annotating scene attributes, we transfer attributes information from existing scene segmentation datasets to the pedestrian dataset, by proposing a novel deep model to learn high-level features from multiple tasks and multiple data sources. Since distinct tasks have distinct convergence rates and data from different datasets have different distributions, a multi-task objective function is carefully designed to coordinate tasks and reduce discrepancies among datasets. The importance coefficients of tasks and network parameters in this objective function can be iteratively estimated. Extensive evaluations show that the proposed approach outperforms the state-of-the-art on the challenging Caltech and ETH datasets, where it reduces the miss rates of previous deep models by 17 and 5.5 percent, respectively.
1. Introduction
Pedestrian detection is difficult because visually similar hard negatives confuse existing models, while deep detectors mainly learn middle-level representations. TA-CNN addresses this by jointly learning pedestrian and scene semantics from multiple datasets to obtain higher-level features.
- Motivation: Visual similarity between pedestrians and hard negatives makes pedestrian detection challenging.Examples include equivocal background patches with appearances resembling human shapes.
- Prior methods: Handcrafted detectors use global or part-based templates, while deep models learn middle-level representations from raw pixels.Existing approaches include HOG, channel features, deformable parts, and CNN-based models.
- Approach: TA-CNN jointly optimizes pedestrian detection with pedestrian and scene attributes to learn high-level representations.Pedestrian attributes include backpack, gender, and views; scene attributes include vehicle, tree, and vertical.
- Approach: Scene attributes are transferred from existing segmentation datasets rather than manually annotated in the pedestrian dataset.The method uses shared and unshared attributes to address differences among scene datasets.
- Approach: TA-CNN uses a weighted multivariate cross-entropy objective and iteratively updates network parameters and task weights.This design coordinates multiple tasks and bridges gaps between datasets.
- Results: 17 and 5.5 percent reductions in miss rates are reported on Caltech and ETH, respectively, relative to previous deep models.The reported evaluations cover both challenging pedestrian-detection benchmarks.
2. Our Approach
The proposed pipeline trains one TA-CNN on pedestrian detection, pedestrian attributes, and transferred scene attributes. It uses shared and unshared attributes across datasets, plus structural projection vectors, to align information from heterogeneous sources.
- Method Overview: The pipeline jointly learns pedestrian classification, pedestrian attributes, and scene attributes with one TA-CNN.Positive pedestrian patches receive nine manually labeled pedestrian attributes.
- Method Overview: Scene attributes are transferred from CamVid, Stanford Background, and LM+SUN, using hard negatives selected by a fast pedestrian detector.Shared attributes occur across all scene datasets, whereas unshared attributes occur in only one.
- Data and Labels: The training set combines patches from the pedestrian dataset and background scene datasets, with labels for pedestrian, shared-scene, and unshared-scene attributes.The model predicts a binary pedestrian label alongside attribute labels.
- Network: TA-CNN combines four convolutional layers, four max-pooling layers, and two fully connected layers to predict the label and attribute targets.The architecture is inspired by AlexNet but removes one convolutional layer and reduces parameters.
- Dataset Bridging: Shared attributes bridge scene datasets by giving their samples common targets, enabling shared representation learning.Without shared labels, each dataset primarily optimizes its own unshared attributes.
- Dataset Bridging: Structural projection vectors align samples from different datasets by projecting them into the pedestrian dataset's feature space.SPVs are computed from distances to leaf-node means in positive and negative HOG-based tree structures.
3. Learning Task-Assistant CNN
TA-CNN replaces heuristic multi-task weighting with a single weighted multivariate cross-entropy objective and alternates between learning network parameters and task importance coefficients. The formulation handles partially observed labels while reducing top-layer parameters.
- Objective: The main pedestrian task is combined with auxiliary pedestrian- and scene-attribute tasks through importance coefficients α, β, and γ.The task probabilities are modeled with softmax functions.
- Motivation: Different task convergence rates can cause over-fitting, while high-dimensional features make the top-layer parameter count grow substantially.The prior formulation optimized eighteen loss functions jointly and relied on heuristic coefficient adjustment.
- Objective: TA-CNN casts multi-task learning as a single weighted multivariate cross-entropy loss with iteratively estimated importance coefficients.The label vector concatenates the pedestrian label with pedestrian and scene attribute labels.
- Objective: 19H parameters replace 38H in the top-layer weight matrix when seventeen binary variables and one four-state viewpoint variable are encoded compactly.The paper states that this reduces the parameter count by half.
- Optimization: Optimization alternates between updating network parameters with coefficients fixed and updating coefficients with network parameters fixed.Network parameters are learned with stochastic gradient descent and back-propagation, while coefficients are updated by solving the coefficient objective.
- Optimization: Missing-label errors are marginalized so they are not propagated, regardless of whether the corresponding predictions are correct.This allows datasets that cover only subsets of attributes to participate in the joint model.
4. Experiments
Experiments evaluate TA-CNN on Caltech and ETH under standard benchmark settings, testing attribute contributions, component combinations, and comparisons with existing detectors. Semantic attributes and multi-source training improve detection, with strongest reported results on both datasets.
- Evaluation protocol: TA-CNN is evaluated on Caltech-Test and ETH using log-average miss rate over nine false-positive operating points and the reasonable pedestrian subsets.The reasonable subset contains pedestrians taller than 49 pixels with at least 65 percent visible body parts.
- Attribute effectiveness: 25.64 percent miss rate is achieved when all pedestrian attributes are combined, improving the main-task TA-CNN by 6 percent.Viewpoint improves miss rate by 3.25 percent, riding by 2.62 percent, and hat by 2.56 percent individually.
- Attribute effectiveness: Traffic-light attributes improve the main task by 2.53 percent, while vertical background patches are more effective than horizontal patches.The comparison indicates that selected scene attributes target hard negatives resembling pedestrians.
- Overall Caltech performance: 20.86 miss rate is obtained with the structure projection vector on Caltech, while adding shared and unshared scene attributes further decreases miss rate by 2.2 and 1.5 percent.The component study reports progressively better performance as pedestrian attributes, shared scene attributes, unshared attributes, and structure projection are added.
- Cross-dataset comparisons: TA-CNN reduces miss rate versus handcrafted-feature methods by 16 and 9 percent compared with DPM+Context and Spatial Pooling, respectively.On ETH, it achieves the lowest miss rate in the reported INRIA-Train setting, exceeding the second-best method by 2.5 percent and the best deep model by 5.5 percent.
- Cross-dataset comparisons: 34.99 percent log-average miss rate is achieved on ETH with the structure projection vector after pedestrian, shared-scene, and unshared-attribute improvements.The reported component gains are 5.5 percent, 1.8 percent, and another 1.2 percent, respectively.
- Cross-dataset comparisons: On ETH, TA-CNN outperforms SDN and DBN-Mul by 5.5 and 6 percent, respectively, and reduces miss rate versus MultiSDP by 12.7.The experiments also report averaged attribute-prediction accuracy exceeding 75 percent.
5. Conclusions
The conclusion presents TA-CNN as a multi-task, multi-dataset approach for learning high-level pedestrian-detection features. It reports superiority over handcrafted and other deep features, while identifying additional attributes and configurations as future improvement directions.
- Conclusions: TA-CNN learns features from pedestrian and scene attributes across multiple datasets, outperforming handcrafted features and features learned by other deep models.The paper attributes this to learning high-level representations through semantic tasks and multiple data sources.
- Future directions: The model could be further improved by incorporating more attributes and exploring additional attribute configurations.The conclusion also identifies potential application to scene parsing because TA-CNN predicts background attributes.
(a)HOG (b)ACF (c)JointDeep (d)TA-CNN
The figures show detection examples for continuous crossroad scenes and reasonable-subset images from Caltech-Test and ETH. Green boxes indicate true positives, while red boxes indicate false positives.
- Caltech crossroad scenes: Caltech-Test crossroad examples show true-positive and false-positive detections across a series of continuous scenes.The reasonable subset includes pedestrians taller than 49 pixels with 65 percent visible body parts.
- Caltech-Test examples: Caltech-Test detection examples use green boxes for true positives and red boxes for false positives.The examples are restricted to the reasonable subset under the stated size and visibility criteria.
- ETH examples: ETH detection examples use the same green-versus-red box encoding for true positives and false positives on the reasonable subset.The subset again requires pedestrians larger than 49 pixels with 65 percent visible body parts.