Source-linked AI summary
Pose-driven Deep Convolutional Model for Person Re-identification
Chi Su, Jianing Li, Shiliang Zhang, Junliang Xing, Wen Gao, Qi Tian
TL;DR
Person ReID is challenged by large pose deformations and complex view variations, while existing methods often omit pose cues or adaptive part weighting. PDC jointly learns normalized global and local representations through pose-driven embedding and feature weighting. Experiments on three benchmark datasets report superiority over current state-of-the-art methods.
Problem
Large pose deformations and complex view variations make learning and matching person features difficult, while existing ReID methods often do not consider pose cues or part-specific representation weights.
Method
PDC uses FEN and PTN to crop and normalize pose-derived body parts, jointly learning global and local representations with FWN-based adaptive feature fusion.
Results
PDC outperforms many state-of-the-art ReID methods on three benchmark datasets.
Takeaways & Limitations
Normalized global and local body representations plus learned feature weighting provide the proposed model’s approach to pose variation and similarity measurement.
Abstract
from arXiv · showhide
Feature extraction and matching are two crucial components in person Re-Identification (ReID). The large pose deformations and the complex view variations exhibited by the captured person images significantly increase the difficulty of learning and matching of the features from person images. To overcome these difficulties, in this work we propose a Pose-driven Deep Convolutional (PDC) model to learn improved feature extraction and matching models from end to end. Our deep architecture explicitly leverages the human part cues to alleviate the pose variations and learn robust feature representations from both the global image and different local parts. To match the features from global human body and local body parts, a pose driven feature weighting sub-network is further designed to learn adaptive feature fusions. Extensive experimental analyses and results on three popular datasets demonstrate significant performance improvements of our model over all published state-of-the-art methods.
1. Introduction
Person ReID must handle complex viewpoint changes and large pose deformations, while existing methods often omit pose cues or adaptive weighting across body parts. The PDC model jointly learns normalized global and local representations with pose-driven feature weighting.
- Person ReID identifies a probe person across cameras using images, locations, and timestamps.
- Complex view variations and large pose deformations make person ReID difficult.
- PDC jointly learns global whole-body and local body-part representations, transforming parts into normalized, homologous regions.FEN automatically crops body parts and uses PTN to reduce pose variation before local representation learning.
- FWN learns adaptive weights for global and part representations to support feature similarity measurement.
- The framework uses pose joints to infer body parts and trains representation and weighting jointly from end to end.The pose estimation component is trained independently before incorporation into the framework.
- Existing deep ReID methods often overlook human pose cues and differing discriminative value across body parts.
2. Related Work
Related work generally learns person representations, distance metrics, or local features from rigid body partitions. PDC differs by normalizing pose-derived parts and learning their feature weights despite pose-estimation inaccuracies.
- Traditional ReID methods either encode robust local appearance features or learn discriminative distance metrics.
- Deep ReID methods commonly learn representations or distance metrics using convolutional, fully connected, Triplet, or Siamese objectives.
- Many deep methods use predefined rigid body parts to learn local pedestrian features.
- Pose-based methods estimate human pose more accurately than simple predefined segmentation, but some use manually cropped parts and are not end-to-end trained.
- PDC uses FEN to normalize part regions and FWN to weight each part feature, emphasizing parts with higher discriminative power.This addresses pose-estimation inaccuracies and distinguishes PDC from earlier pose-based methods that omit part weighting.
3. Pose-driven Deep ReID Model
The PDC framework jointly processes global images and pose-derived local body parts, using FEN to normalize and transform parts and FWN to adaptively fuse their features. PTN addresses complex pose variation through part-specific affine transformations, while FWN reduces the impact of unreliable part detections.
- Overall framework: The framework simultaneously considers global and modified part images during training, combining whole-body and local representations.Pose joints define body parts; FEN transforms and combines them before CNN processing.
- Feature Embedding sub-Net: FEN locates 14 body joints and uses them to define six regions: head, upper body, two arms, and two legs.The regions are normalized through cropping, rotation, and resizing to fixed sizes and orientations.
- Pose Transformation Network: Simple resizing and rotation are insufficient for complex pose variation, especially when pose estimates are inaccurate.PTN is introduced to learn the angles required for rotating the five body parts and to address this limitation.
- Pose Transformation Network: PTN computes separate affine transformations for body parts and combines the transformed regions, unlike a standard STN that transforms the whole image.The affine parameters encode scale, rotation, and translation; the head part is excluded from the learned transformation.
- Feature Weighting sub-Net: Inaccurate detections caused by occlusion or pose changes can make part features unreliable, motivating feature weighting before fusion with the global feature.FWN applies learned weights and a nonlinear transformation to part features, then concatenates them with the global feature.
- Feature Weighting sub-Net: FWN supports end-to-end learning by providing a gradient through the weighted nonlinear part-feature fusion.The gradient is nonzero for the corresponding part-feature dimensions and zero elsewhere.
4. Experiment
Experiments evaluate PDC across CUHK 03, Market 1501, and VIPeR, including component ablations and comparisons with related methods. Results show benefits from combining global and part features, pose transformation, and adaptive feature weighting, with strong benchmark performance.
- Datasets and protocol: PDC is evaluated on CUHK 03, Market 1501, and VIPeR, with VIPeR training supplemented by the other datasets because its image count is insufficient.The evaluation uses rank-based CMC accuracy, with mAP additionally reported for Market 1501.
- Individual components: Fusing global and part features improves performance over using either feature type alone, while adding FEN and FWN further increases rank1 identification.Global+Part reaches rank1 accuracies of 85.07% on CUHK 03 labeled and 76.33% on CUHK 03 detected.
- Related-work comparisons: 78.29% and 88.70% rank1 accuracy are achieved on CUHK 03 detected and labeled datasets, respectively, exceeding PIE and Spindle by 11.19% and 0.20%.The comparison includes distance metric learning, feature extraction, and deep learning methods.
- Related-work comparisons: 84.14% rank1 accuracy and 63.41% mAP are obtained on Market 1501 in single-query mode, outperforming the compared methods, including PIE.The paper attributes this result to pose-invariant features from FEN and feature fusion through FWN.
- Related-work comparisons: 51.27% rank1 accuracy is achieved on VIPeR, outperforming most compared methods except Spindle, whose training set combines Market 1501, CUHK 03, and VIPeR.PDC performs better than Spindle on the other two datasets.
- Feature weighting sub-network: One nonlinear Weight Layer gives the best FWN performance across all three datasets, whereas increasing the number of layers reduces performance despite adding computation.A nonlinear one-layer variant also outperforms one layer without nonlinear transformation.
5. Conclusions
The paper concludes that PDC uses human body and part cues to learn normalized representations and adaptive feature fusion for person ReID. Experiments on three benchmark datasets demonstrate superiority over current state-of-the-art methods.
- Conclusion: PDC transforms global and local human representations into a normalized, homologous state for feature embedding and learns adaptive weights for feature fusion.The architecture explicitly leverages human part cues for representation and similarity measurement.
- Conclusion: Experimental results on three benchmark datasets demonstrate PDC’s superiority over current state-of-the-art methods.