PaperlayerView exact revision on arXiv

Source-linked AI summary

Pedestrian Attribute Recognition: A Survey

Xiao Wang, Shaofei Zheng, Rui Yang, Aihua Zheng, Zhe Chen, Jin Tang, Bin Luo

arXiv:1901.07474v2cs.CVcs.AIcs.LG

TL;DR

Pedestrian attribute recognition is important for video surveillance and related vision tasks, but its performance is challenged by appearance variation and difficult imaging conditions. This paper surveys traditional and deep-learning-based PAR methods, benchmarks, learning formulations, architectures, applications, and future directions. It provides the first review of PAR and organizes existing algorithms and applications while identifying efficiency and fine-grained-recognition limitations in reviewed approaches.

  • Problem

    PAR must infer multiple semantic attributes from pedestrian images despite appearance variation, occlusion, viewpoint changes, and other challenging conditions.

  • Method

    The paper reviews PAR from traditional methods through deep learning, covering benchmarks, evaluation criteria, multi-task and multi-label learning, neural architectures, algorithm categories, applications, and research directions.

  • Results

    The survey provides the first review of PAR and reports conclusions and research directions by evaluating existing algorithms and top-ranked baseline methods.

  • Takeaways & Limitations

    The synthesis shows that PAR research spans multiple learning formulations and algorithm families, with attributes also used in person re-identification, detection, and tracking.

  • Takeaways & Limitations

    Sequential attribute estimation is time-inefficient, and whole-image models remain limited by insufficient fine-grained recognition.

Abstract

from arXiv · show

Recognizing pedestrian attributes is an important task in the computer vision community due to it plays an important role in video surveillance. Many algorithms have been proposed to handle this task. The goal of this paper is to review existing works using traditional methods or based on deep learning networks. Firstly, we introduce the background of pedestrian attribute recognition (PAR, for short), including the fundamental concepts of pedestrian attributes and corresponding challenges. Secondly, we introduce existing benchmarks, including popular datasets and evaluation criteria. Thirdly, we analyze the concept of multi-task learning and multi-label learning and also explain the relations between these two learning algorithms and pedestrian attribute recognition. We also review some popular network architectures which have been widely applied in the deep learning community. Fourthly, we analyze popular solutions for this task, such as attributes group, part-based, etc. Fifthly, we show some applications that take pedestrian attributes into consideration and achieve better performance. Finally, we summarize this paper and give several possible research directions for pedestrian attribute recognition. We continuously update the following GitHub to keep tracking the most cutting-edge related works on pedestrian attribute recognition~\url{https://github.com/wangxiao5791509/Pedestrian-Attribute-Recognition-Paper-List}

1 INTRODUCTION

Pedestrian attribute recognition describes people using semantic attributes that support video surveillance and related vision tasks. This survey addresses the lack of a comprehensive synthesis by reviewing methods, applications, evaluations, and research directions.

  • Motivation: Pedestrian attributes provide high-level semantic descriptions that are more robust to viewpoint and viewing-condition changes than low-level features.They can support person re-identification, face verification, human identification, person detection, and related tasks.
  • Prior methods: Traditional approaches use hand-crafted features, classifiers, or attribute relations, but large-scale evaluations indicate insufficient performance for realistic applications.Representative milestones include HOG, SIFT, SVM, and CRF models.
  • Prior methods: Deep learning methods exploit automatic feature extraction through multilayer nonlinear transformations for pedestrian attribute recognition.The survey situates these methods within broader advances in computer vision, speech recognition, and natural language processing.
  • Survey scope: The survey fills a stated gap by providing a detailed review, comprehensive evaluation, and analysis of traditional and deep-learning-based attribute recognition algorithms.It compares classification perspectives such as part-based, group-based, and end-to-end learning.
  • Survey scope: It also examines how attributes support person re-identification, object detection, and person tracking, and evaluates algorithms to derive conclusions and research directions.The paper additionally organizes benchmarks, methods, and the survey structure across later sections.

2 PROBLEM FORMULATION AND CHALLENGES

Pedestrian attribute recognition predicts multiple attributes from a person image using a predefined attribute list. The task remains difficult because appearance variation and surveillance conditions create ambiguity and unreliable visual evidence.

  • Problem formulation: Given a person image, PAR predicts a group of attributes from a predefined list, using approaches such as multilabel or binary classification.The attributes describe the characteristics of the depicted person.
  • Challenges: Large intra-class variation arises from appearance diversity and appearance ambiguity across attribute categories.These variations make attribute categories difficult to distinguish consistently.
  • Challenges: Multiple camera views introduce viewpoint changes, while the nonrigid human body complicates recognition across poses.These factors can alter the observed appearance of the same person.
  • Challenges: Occlusion can hide body regions and introduce misleading pixels, increasing the likelihood of incorrect attribute predictions.Occluders may be other people or objects.
  • Challenges: Unbalanced attribute distributions, low image resolution, and illumination variation further constrain recognition in practical surveillance settings.The passage notes that each person has a different set of attributes and that high-quality cameras may be expensive.

3 BENCHMARKS

PAR benchmarks span diverse datasets with varied annotations, poses, resolutions, and capture conditions. Evaluation includes both label-based metrics and example-based metrics, with the latter addressing prediction consistency across attributes.

  • Datasets: PAR datasets may include attribute labels at different semantic levels and annotations for perspectives, body parts, and occlusion.Examples include hairstyle, color, hats, and glasses, which correspond to different image regions.
  • Datasets: The survey lists benchmarks including PETA, RAP, RAP-2.0, PA-100K, WIDER, Market-1501, DukeMTMC, PARSE27K, APiS, HAT, Berkeley-Attributes, and CRP.The listed datasets support research across multiple pedestrian-attribute settings.
  • Datasets: PETA contains 19,000 images of 8,705 people with 61 binary and 4 multiclass attributes, partitioned into training, verification, and testing sets.Its image resolutions range from 17×39 to 169×365 pixels.
  • Datasets: Dataset conditions vary substantially in pose, age, clothing, accessories, capture source, identity coverage, and video tracklets.The passages describe datasets assembled from surveillance, driving, person re-identification, and other sources.
  • Evaluation criteria: ROC and AUC evaluate attribute classification using recall and false-positive rate, while mA averages positive and negative classification accuracies across attributes.mA is computed per attribute and then averaged over all attributes.
  • Evaluation criteria: Label-based criteria treat attributes independently, whereas example-based accuracy, precision, recall, and F1 measure consistency of predictions for each pedestrian image.Example-based evaluation was proposed to capture correlations across labels within an example.

4 REGULAR PIPELINE FOR PAR

The regular PAR pipeline maps pedestrian images to multiple attribute predictions using feature extraction and shared or label-aware learning strategies. Multi-task and multi-label formulations address efficiency and attribute correlations through different modeling choices.

  • Pipeline: Because pedestrians may have dozens of attributes, estimating all attributes in one model is more efficient than learning each independently.The survey frames each attribute estimation as a task within a multi-task setting.
  • Multi-task learning: Multi-task learning exploits correlations among related tasks, which independent processing can overlook.The survey gives gender and clothing style as examples of correlated pedestrian attributes.
  • Multi-task learning: Hard parameter sharing uses shared shallow layers and task-specific high-level layers, whereas soft sharing trains task networks independently while coupling their parameters.These are the two principal deep-learning parameter-sharing approaches described in the survey.
  • Multi-label learning: Multi-label strategies range from first-order independent classification to second-order pairwise modeling and high-order modeling of all label relations.Higher-order strategies are more generic but can have greater complexity.
  • Multi-label learning: Problem transformation uses binary relevance, classifier chains, or calibrated label ranking, while algorithm adaptation modifies existing algorithms for multilabel data.These approaches differ in how they represent or exploit label dependencies.
  • Pipeline: A conventional pipeline preprocesses the image, extracts HOG, SIFT, or deep features, and predicts attributes from the resulting representation.Normalization, random cropping, and whitening can improve input quality or augment features.

5 DEEP NEURAL NETWORKS

This section reviews neural network architectures relevant to pedestrian attribute recognition, from early CNNs to recurrent, capsule, and graph-based models. It highlights design choices for feature extraction, deep training, spatial dependencies, and structured outputs.

  • Convolutional Neural Networks: LeNet introduced convolutional and max-pooling layers for feature extraction before fully connected classification, originally targeting handwritten and machine-printed character recognition.It accepts 32×32 single-channel images and uses two convolutional-plus-pooling groups.
  • Convolutional Neural Networks: AlexNet marked a major deep-learning milestone with five convolutional layers, max pooling, ReLUs, fully connected layers, and dropout.It won ILSVRC-2012 with a TOP-5 test accuracy of 84.6% and contained 60 million parameters.
  • Deep CNN Architectures: VGG increased depth using stacks of small-receptive-field convolutional layers, while GoogleNet introduced parallel inception-module operations.VGG uses 16- or 19-layer networks; GoogleNet has 22 layers and combines operations including 1 × 1, large-sized, and small-sized convolutions.
  • Deep CNN Architectures: Residual and Dense Networks improve the training of deep models through identity skip connections or direct layer-to-layer information and gradient access.Residual blocks help address vanishing gradients, while DenseNets provide implicit deep supervision and improved parameter efficiency.
  • Alternative Architectures: Capsule Networks replace scalar neuron outputs and max pooling with vector-valued capsules and dynamic routing to preserve richer feature relationships.The reviewed approach uses routing-by-agreement together with margin and reconstruction losses.
  • Structured and Recurrent Models: GCNs, ReNet, and RNNs extend neural modeling beyond standard image grids by representing graph relations, sweeping spatial dimensions, or processing sequential dependencies.ReNet replaces convolution-plus-pooling with four RNNs sweeping vertically and horizontally, while GCNs produce node-level outputs from graph features and adjacency information.

6 THE REVIEW OF PAR ALGORITHMS

The review organizes deep PAR methods into eight categories, spanning holistic, part-based, attention, sequential, loss-based, curriculum, graphical, and other approaches. Across these methods, richer feature sharing, local information, pose, context, and fine-grained modeling address limitations of global-image systems, while added localization and data requirements create practical costs.

  • Deep PAR methods are reviewed across eight categories, including global-based, part-based, visual-attention, sequential-prediction, loss-function, curriculum-learning, graphical-model, and other approaches.The review uses these categories to structure existing deep neural network-based algorithms.
  • Global-based methods: Global-image methods use shared or multi-task representations for efficient PAR but remain limited by insufficient fine-grained recognition.Examples include multi-branch attribute classification, DeepSAR, DeepMAR, and MTCNN; DeepMAR additionally models all attributes jointly and can reweight imbalanced labels.
  • Part-based methods: Pose-guided modeling uses estimated body key points to extract part regions, while other approaches fuse independently computed whole-image and part-region scores.PGDM explicitly explores pedestrian pose structure for attribute learning; its final prediction combines whole-image and part-region evidence.
  • Context-based methods: Context-based recognition augments person and part features with human-centric and scene-level regions, then sums four branch scores into the final attribute score.The approach uses an image pyramid and extracts whole-person, detected-part, nearest-neighbor-part, and global-scene features.
  • Part-based methods: Part-based methods combine global and local features to capture detailed attribute information through patches, learned windows, pose, or adaptive part localization.Representative designs include flexible overlapping windows, mid-level patch mining, joint localization and classification, and pose-guided region extraction.
  • Limitations: Local and contextual modeling can improve recognition, but performance and efficiency depend on localization quality and may require additional training data, inference time, or manual annotations.The review identifies inaccurate part detection, extra computation, annotation cost, and external context data as practical limitations.

6.3 Attention-based Models

Attention-based PAR models focus network resources on attribute-relevant visual regions, views, or feature levels. The reviewed methods include multi-directional, view-specific, multi-scale, weakly supervised, and refined attention mechanisms.

  • HydraPlus-Net: HydraPlus-Net uses a regular Main Net and an Attentive Feature Net with multi-directional attention across semantic feature levels.The two networks share basic convolutional architectures and produce complementary outputs.
  • HydraPlus-Net: A shared attention map attends to adjacent feature maps, allowing fused features to encode multi-level information within one spatial distribution.The map generated from feature map 2 is also applied to feature maps 1 and 3.
  • VeSPA: VeSPA models view cues because attribute evidence can be strongly localized and attributes such as hair, backpacks, and shorts depend on pedestrian view.Its view classifier and attribute predictors use separate losses within an end-to-end framework.
  • DIAA: DIAA combines multi-scale visual attention with weighted focal loss and learns attention maps weakly supervised by attribute labels alone.Its attention sub-network produces attribute-specific masks from feature maps without bounding-box annotations.
  • CAM: CAM refines attention maps by encouraging concentrated regions, using a weighted-average map, downsampling, Softmax normalization, and global average pooling.The model is pretrained with classification loss and then fine-tuned with joint losses.
  • Summary: The survey concludes that visual attention has been introduced into PAR, but designing new attention models and transferring attention methods from other domains remains open.

6.4 Sequential Prediction-based Models

Sequential prediction models treat pedestrian attributes as related outputs rather than independent labels, using recurrent or sequence-alignment mechanisms to model dependencies. Their main limitation is reduced time efficiency from successive attribute estimation.

  • CNN-RNN: CNN-RNN combines CNN image features with an RNN to learn joint image-label embeddings that capture semantic label dependencies and image-label relevance.Inference searches for a high-probability attribute sequence, while beam search selects the top-ranked path.
  • JRL: JRL uses sequence-to-sequence learning to model intra-person attribute context from horizontal image strips and inter-person context from visually similar exemplars.Its decoder uses both contexts and attention to generate variable-length attributes.
  • GRL: GRL divides attributes into groups and predicts them sequentially, exploiting relationships within groups and between groups in an end-to-end model.Attributes in a group share fully connected features, while global and local features are modeled with LSTM.
  • JCM: JCM combines CTC and attention to predict arbitrary-length attribute sequences while reducing sensitivity to manual attribute division and ordering.It jointly performs attribute recognition and person re-identification using a shared ResNet-50 backbone.
  • RCRA: RCRA introduces Recurrent Convolutional and Recurrent Attention models to capture correlations between attribute groups and spatial or attention relationships.CNN features are processed group by group with ConvLSTM, followed by classification for each current group.
  • Summary: Sequential methods are described as elegant and effective but suffer from time inefficiency because attributes are estimated successively.The survey identifies more efficient sequential estimation as a future direction.

6.5 Loss Function based Models

Loss-function-based PAR models address weak supervision, class imbalance, and unequal task difficulty through weighted or adaptive objectives. WPAL weights categories by label prevalence, while AWMT adapts attribute-task weights using validation behavior.

  • WPAL-network: WPAL jointly recognizes and locates attributes using only attribute labels, combining multi-layer features with Flexible Spatial Pyramid Pooling.FSPP adds spatial constraints and allows detector-class correlations to be learned during training.
  • WPAL-network: WPAL introduces weighted cross entropy to handle the extremely imbalanced positive and negative distributions of attribute categories.The weighting vector represents the proportion of positive labels across attribute categories.
  • AWMT: AWMT targets negative transfer by adaptively coordinating learning weights when different attributes have unequal difficulty or dissimilarity.Its motivation is that indiscriminate information sharing can hurt performance for dissimilar tasks.
  • AWMT: AWMT updates attribute weights from validation-loss behavior and uses them to weight task losses during network training.The mechanism is updated periodically, and experiments on several attribute datasets validated its effectiveness.
  • Summary: The survey notes that relatively few PAR studies design new loss functions, although WPAL-style imbalance weighting has been widely used.

6.6 Curriculum Learning-based Algorithms

Curriculum-learning methods organize PAR training from easier to harder information, using multi-task transfer or task-correlation structure to address limited labels and differing attribute relationships.

  • MTCT: MTCT combines a multi-task network with curriculum transfer learning to address limited manually labeled training data.Its network uses parallel attribute-specific branches over stacked Network-In-Network units.
  • MTCT: MTCT trains clothing attributes progressively, beginning with cleaner source images before transferring knowledge toward harder target conditions.The strategy follows an easy-to-hard learning schedule motivated by cognitive studies.
  • CILICIA: CILICIA measures attribute-task correlations with Pearson coefficients and separates strongly correlated tasks from weakly correlated tasks.Tasks in the weakly correlated group receive guidance from knowledge learned by strongly correlated tasks.
  • CILICIA: CILICIA uses supervision transfer to assist attribute learning in the weakly correlated task group.
  • CILICIA-v2: CILICIA-v2 replaces the two-group division with hierarchical agglomerative clustering based on the Pearson correlation matrix.Ward variance minimization permits a variable number of task groups.
  • Summary: The survey positions curriculum learning as an emerging way to model relationships among attributes, alongside related self-paced learning approaches.

6.7 Graphic Model based Algorithms

Graphic-model approaches represent relations among body parts, poses, and attributes to address the structured dependencies and visual challenges in pedestrian attribute recognition. Reviewed methods include CRF, attribute And-Or grammar, and visual-semantic graph reasoning.

  • DCSA: DCSA uses pose estimation, upper-body localization, and hand-crafted appearance features before modeling attribute correlations with a CRF.Its features include SIFT, texture descriptors, LAB color, and skin probabilities; lower-body regions are omitted because of occlusion.
  • A-AOG: A-AOG explicitly represents body-part decomposition and articulation while modeling correlations between poses and attributes.And-nodes encode decomposition or dependency, whereas or-nodes encode alternative decompositions or part types.
  • A-AOG: A-AOG inference seeks the most probable parse graph under a Bayesian model combining likelihood, prior probability, part relations, and attribute relations.The resulting parse graph represents human-body detection, pose, and attributes.
  • VSGR: VSGR jointly models region-region, attribute-attribute, and region-attribute relations through visual-to-semantic and semantic-to-visual graph reasoning.Its semantic graph uses previously known outputs for current-attribute prediction, and both sub-networks are fused for end-to-end training.
  • Overview: Graphic models are integrated into PAR to model relationships among attributes, body parts, poses, and regions.The survey covers Markov and conditional random fields, And-Or graphs, and graph neural networks.

6.8 Other Algorithms

Other surveyed approaches address PAR through self-supervised pre-training, automatically structured multi-task networks, and generative reconstruction for difficult images. These methods target domain-specific initialization, compact task sharing, occlusion, and low resolution.

  • PatchIt: PatchTask pre-trains PAR networks using automatically generated patch-origin labels from unlabeled same-domain images.Images are divided into non-overlapping patches, and the network predicts each patch’s origin before initializing VGG16 for PAR.
  • FaFS: FaFS automatically grows a thin multi-task network into a tree-like architecture, grouping similar tasks into shared branches.Layer-wise widening proceeds from the output layer downward, while task affinity separates similar and dissimilar tasks.
  • FaFS: Multi-task learning shares relevant information across tasks to improve final generalization performance.FaFS operationalizes this objective by discovering task-specific branching rather than relying only on a hand-designed shared architecture.
  • GAM: The GAM approach uses attribute classification, reconstruction, and super-resolution sub-networks to address occlusion and low-resolution pedestrian images.Its classifier combines global and local features, while a generative adversarial network reconstructs and enhances images.

7 APPLICATIONS

Pedestrian attributes serve as mid-level representations for related vision tasks, including detection and person re-identification. Surveyed systems integrate attribute learning with task-specific objectives or transfer attribute knowledge across datasets.

  • Overview: Pedestrian attributes provide mid-level information for person re-identification, detection, tracking, retrieval, action recognition, and scene understanding.The survey focuses on representative applications rather than covering every related task.
  • Pedestrian Detection: Jointly optimizing person detection with semantic tasks uses attribute information to address confusion between positive and hard-negative samples.The approach transfers information from scene-segmentation datasets to learn high-level features across tasks and data sources.
  • Person Re-identification: A multi-task re-identification network estimates person attributes and person identity simultaneously to learn more discriminative representations.The surveyed work links PAR’s local-information emphasis with re-identification’s global-person representation.
  • Person Re-identification: Semi-supervised attribute learning transfers an attribute model from an annotated dataset to an identity-only dataset and then predicts attributes for the target data.The resulting deep attributes support person re-identification using simple Cosine distance on multiple datasets.

8 FUTURE RESEARCH DIRECTIONS

The survey identifies future directions that make PAR more localized, robust, multimodal, and temporally informed. It highlights generative modeling, attention, architecture design, prior knowledge, sensor fusion, video, and integration with other person-related tasks.

  • Local Parts: Part-based methods should move toward weakly supervised frameworks that jointly localize body regions and recognize attributes.Existing approaches often rely on manually annotated or detected body or pose information for localization.
  • Generative Models: Deep generative models are proposed for handling low-quality pedestrian images and unbalanced data distributions through image generation and augmentation.The survey connects attribute- or pose-guided generation with applications such as detection, re-identification, and tracking.
  • Visual Attention: Attention-based PAR still requires mechanisms that locate relevant regions accurately and efficiently.The survey identifies attention design and borrowing ideas from NLP as future research directions.
  • Network Architecture: Future PAR networks should account for task-specific characteristics instead of relying mainly on off-the-shelf backbones such as ImageNet-pretrained models.The survey suggests exploring novel architectures and AutoML for person attribute recognition.
  • Multimodal PAR: Multimodal fusion could improve robustness because thermal images can highlight human contours and carried or worn objects when RGB imagery is degraded.The survey motivates combining RGB with thermal or depth sensors for difficult illumination and weather conditions.
  • Video-based PAR: Video-based PAR can combine spatial and temporal information, support dynamic attributes, and use motion to reason about attributes difficult to recognize in one image.Video attributes may also provide information for other video-based tasks.
  • Cross-task Applications: Integrating fine-grained attribute learning into detection, tracking, re-identification, and social activity analysis remains an important research direction.The survey lists attribute-based systems across these person-related tasks and calls for better cross-task exploration.

9 CONCLUSION

The paper presents a review of pedestrian attribute recognition spanning traditional approaches and deep learning-based algorithms. It organizes PAR research across background, benchmarks, learning paradigms, network architectures, algorithmic strategies, related tasks, and future directions.

  • The survey covers pedestrian attribute recognition from traditional approaches through recent deep learning-based algorithms.
  • It reviews PAR background, benchmarks, multi-task and multi-label learning, neural networks, and diverse algorithmic strategies.The strategies include global-, part-, attention-, sequential-prediction-, loss-function-, curriculum-, and graph-model-based approaches.
  • The paper also examines combining person-attribute learning with human-related tasks and identifies possible research directions.
Loading 1901.07474v2…