Source-linked AI summary
Pose Invariant Embedding for Deep Person Re-identification
Liang Zheng, Yujia Huang, Huchuan Lu, Yi Yang
TL;DR
Pedestrian pose variation and detector errors create misalignment that compromises person re-identification. The paper constructs standardized PoseBoxes and fuses them with original images and pose confidence in PBF to learn PIE. PoseBox alone yields fair or decent accuracy, while PIE achieves competitive accuracy on the evaluated datasets.
Problem
Pedestrian misalignment from pose variations and detection errors compromises feature learning and matching in person re-identification.
Method
The method builds PoseBoxes through pose estimation and affine transformations, then learns PIE with a PBF network fusing the original image, PoseBox, and pose-estimation confidence.
Results
PoseBox alone yields fair accuracy, while PIE produces competitive accuracy on Market-1501, CUHK03, and VIPeR.
Takeaways & Limitations
Combining original images with PoseBoxes and pose confidence reduces the impact of pose-estimation errors and detail loss during PoseBox construction.
Takeaways & Limitations
PoseBox construction is manually designed because the current re-ID datasets do not provide ground-truth poses, and pose-estimation errors remain a limitation.
Abstract
from arXiv · showhide
Pedestrian misalignment, which mainly arises from detector errors and pose variations, is a critical problem for a robust person re-identification (re-ID) system. With bad alignment, the background noise will significantly compromise the feature learning and matching process. To address this problem, this paper introduces the pose invariant embedding (PIE) as a pedestrian descriptor. First, in order to align pedestrians to a standard pose, the PoseBox structure is introduced, which is generated through pose estimation followed by affine transformations. Second, to reduce the impact of pose estimation errors and information loss during PoseBox construction, we design a PoseBox fusion (PBF) CNN architecture that takes the original image, the PoseBox, and the pose estimation confidence as input. The proposed PIE descriptor is thus defined as the fully connected layer of the PBF network for the retrieval task. Experiments are conducted on the Market-1501, CUHK03, and VIPeR datasets. We show that PoseBox alone yields decent re-ID accuracy and that when integrated in the PBF network, the learned PIE descriptor produces competitive performance compared with the state-of-the-art approaches.
1. Introduction
The paper targets pedestrian misalignment in person re-identification, caused by pose variation and detection errors, by introducing PoseBox and the PoseBox Fusion network for pose-invariant embedding.
- Motivation: Misalignment arises from varied pedestrian poses and detection errors, making body-part positions unpredictable and causing severe vertical shifts.These shifts can cause a person’s head to align with background regions in another image.
- PoseBox limitations: PoseBox construction can lose important pedestrian details and inherit pose-estimation errors, compromising the resulting representation.The paper specifically notes that details such as bags and umbrellas may be missed during cropping.
- PoseBox: PoseBox transforms detected body parts toward a standardized upright stance to support well-aligned pedestrian matching.The paper reports that PoseBox alone yields very decent re-ID accuracy.
- PoseBox Fusion: PBF fuses the original image, PoseBox, and pose-estimation confidence to reduce information loss and provide a fallback when pose estimation fails.PIE is defined as the fully connected activations of the PBF network.
- Results: PIE produces competitive re-ID accuracy on Market-1501, CUHK03, and VIPeR, while PoseBox alone provides decent accuracy.The paper presents this as competitive performance relative to state-of-the-art approaches.
2. Related Work
Related work addresses pose variation and misalignment through pose-aware features, articulated models, pose classes, and pictorial structures, while this paper uses CNN-based pose estimation and PoseBox construction.
- Pose estimation: The paper adopts convolutional pose machines, which use a sequential convolutional architecture with intermediate supervision to prevent vanishing gradients.CPM detects a set of body joints used for subsequent body-part construction.
- Pose-aware re-ID: Earlier re-ID studies modeled pose variation using body-part symmetry, upper-body orientation, articulated 3D models, or coarse front, back, and side pose classes.These approaches connect pose handling to feature extraction or person categorization.
- PoseBox construction: PoseBox construction discovers ten body parts from detected joints and projects quadrilateral parts into rectangles using affine transformations.The resulting representation supports the paper’s standardized-pose strategy.
- Normalized-pose methods: Pictorial structure methods share PoseBox’s motivation of constructing normalized poses, but use traditional pose estimators and hand-crafted descriptors.The paper contrasts these techniques with its CNN-based approach.
3. Proposed Method
The method constructs standardized pedestrian representations from pose estimates, then fuses PoseBox, original-image, and confidence features to form PIE for re-ID retrieval.
- PoseBox Construction: PoseBox construction estimates 14 body joints, discovers 10 body parts, and projects quadrilateral parts into rectangles with affine transformations.The parts cover the head, torso, arms, and legs.
- PoseBox Construction: PoseBox 1 contains the torso and legs, PoseBox 2 adds both arms, and PoseBox 3 additionally places the head above the torso.PoseBox 2 outperforms PoseBox 1, while PoseBox 3 is slightly inferior to PoseBox 2, probably because of head/neck estimation errors.
- PoseBox Construction: PoseBox corrects pose variations and largely removes background noise, but pose estimation errors and manual construction can cause imprecise joints and information loss.The paper constructs PoseBoxes manually because the re-ID datasets lack ground-truth poses.
- PoseBox Fusion Network: The PBF network uses the original image, PoseBox, and a 14-dimensional confidence vector, with separate CNN streams joined at FC7 before FC8.The two image streams do not share convolutional weights, while the confidence vector passes through a small fully connected layer.
- PoseBox Fusion Network: PIE is extracted either from concatenated FC7 activations or the next fully connected layer, producing 8,206-dimensional or dataset-dependent descriptors.For AlexNet, the next fully connected layer is 751-dimensional on Market-1501 and 1,160-dimensional on CUHK03.
- PoseBox Fusion Network: PBF combines complementary aligned and original-image cues while using confidence to assess PoseBox reliability, reducing the influence of detection errors and pose variations.The original image provides fallback information when pose estimation fails or PoseBox construction loses discriminative details.
4. Experiment
Experiments across Market-1501, CUHK03, and VIPeR evaluate PoseBox, PIE, feature combinations, and PBF components. PIE consistently improves over the baselines, while fusion reduces sensitivity to pose-estimation errors and information loss.
- Experimental setup: Experiments use VIPeR, CUHK03, and Market-1501, with AlexNet or ResNet-50 backbones and standard baseline comparisons.VIPeR uses averaged results over 10 splits, while CUHK03 uses 20 random train/test splits.
- Baselines: 73.02% rank-1 accuracy is achieved by the original-image baseline with ResNet-50 on Market-1501, while the PoseBox baseline is 8.9% lower.The reported drop is attributed to pose-estimation errors and information loss.
- PIE effectiveness: PIE improves Market-1501 rank-1 accuracy over baselines by +5.0% and +6.8% with ResNet-50, reaching 78.65% rank-1 accuracy and 53.87% mAP.Consistent improvement over the baselines is also observed on CUHK03 and VIPeR.
- Feature combinations: PBF learns more effective embeddings than separately concatenating baseline features, and its image and PoseBox subvectors outperform their corresponding baselines.The comparison uses ResNet-50 features and Kissme or Euclidean distance as specified for Figure 7.
- PoseBox variants: PoseBox2 outperforms PoseBox1 and PoseBox3, while integrating the original image in PBF narrows the performance gap among PoseBox variants.Arms improve discrimination relative to PoseBox1, whereas the head can add noise because of unstable detection.
- State-of-the-art comparison: 77.97% rank-1 accuracy and 52.76% mAP are reported on Market-1501, where the paper states that PIE sets a new state of the art.On CUHK03, Kissme results are reported as 67.10% rank-1 accuracy and 71.32% mAP; on VIPeR, PIE exceeds competing unsupervised methods in rank-1, rank-5, and rank-10 accuracy.
- Qualitative results: Qualitative examples show PIE reducing foreground-background confusion when background colors resemble pedestrian clothing.The reported example concerns cyan clothing in the background causing incorrect rankings based on local green or blue colors.
5. Conclusion
The paper addresses pedestrian misalignment in person re-identification with PoseBox and PIE. PoseBox standardizes pose, while PBF fuses it with the original image and pose confidence to reduce errors and detail loss; PIE achieves competitive accuracy.
- Conclusion: PoseBox uses 16 detected joints to correct pose variations from camera views, person motions, and detector errors.The resulting representation supports more well-aligned pedestrian matching.
- Conclusion: PIE is learned through PBF, which fuses the original image, PoseBox, and pose-estimation confidence.This fusion reduces the impact of pose-estimation errors and detail loss during PoseBox construction.
- Conclusion: PoseBox alone yields fair accuracy, while PIE produces competitive accuracy compared with state-of-the-art methods.The conclusion summarizes the reported benchmark findings without specifying a single dataset or metric.