Source-linked AI summary
Perceive Where to Focus: Learning Visibility-aware Part-level Features for Partial Person Re-identification
Yifan Sun, Qin Xu, Yali Li, Chi Zhang, Yikang Li, Shengjin Wang, Jian Sun
TL;DR
Partial re-ID must compare incomplete pedestrian observations despite severe spatial misalignment and distracting unshared regions. The paper proposes VPM, which learns region visibility through self-supervision and compares shared region features. VPM improves partial re-ID accuracy over global and part-based baselines, achieving performance on par with the state of the art.
Problem
Partial re-ID compares partial pedestrian observations with holistic or differently proportioned images, creating severe spatial misalignment and distracting unshared regions.
Method
VPM uses self-supervision to locate predefined visible regions, extract region-level features, and compare images using their shared regions.
Results
VPM significantly improves partial re-ID accuracy over global-feature and strong part-based convolutional baselines, with performance on par with the state of the art.
Takeaways & Limitations
Visibility-aware region comparison provides a representation that suppresses spatial misalignment and noise from unshared regions in partial re-ID.
Takeaways & Limitations
Training assumes original images contain holistic pedestrians tightly bounded by bounding boxes, although some dataset images are treated as tolerable noise.
Abstract
from arXiv · showhide
This paper considers a realistic problem in person re-identification (re-ID) task, i.e., partial re-ID. Under partial re-ID scenario, the images may contain a partial observation of a pedestrian. If we directly compare a partial pedestrian image with a holistic one, the extreme spatial misalignment significantly compromises the discriminative ability of the learned representation. We propose a Visibility-aware Part Model (VPM), which learns to perceive the visibility of regions through self-supervision. The visibility awareness allows VPM to extract region-level features and compare two images with focus on their shared regions (which are visible on both images). VPM gains two-fold benefit toward higher accuracy for partial re-ID. On the one hand, compared with learning a global feature, VPM learns region-level features and benefits from fine-grained information. On the other hand, with visibility awareness, VPM is capable to estimate the shared regions between two images and thus suppresses the spatial misalignment. Experimental results confirm that our method significantly improves the learned representation and the achieved accuracy is on par with the state of the art.
1. Introduction
Partial re-ID must handle severe spatial misalignment and distracting unshared regions when comparing incomplete pedestrian observations. VPM addresses both by learning visibility-aware region features and comparing only shared regions.
- Partial re-ID involves incomplete pedestrian observations caused by occlusion or pedestrians leaving the camera field of view.
- Partial observations aggravate spatial misalignment between probe and gallery images, even with matching pose and viewpoint.
- Unshared body regions become distracting noise rather than discriminative clues when partial and holistic images, or differently proportioned images, are compared.
- VPM uses self-supervision to locate predefined visible regions, learn region-level features, and compare two images through their shared regions.During testing, it calculates local distances between shared regions before producing an overall distance.
- VPM improves partial re-ID accuracy over global-feature and strong part-based convolutional baselines, with performance on par with the state of the art.Experiments cover synthetic and realistic datasets, including scalable large-scale synthetic partial re-ID settings.
2. Related Works
Prior work uses part-level features and self-supervision, but partial re-ID exposes weaknesses in precise part localization. VPM instead learns visibility directly and uses it to select regions during matching.
- 2.1. Deeply-learned part features for re-ID: Holistic re-ID methods form descriptors from multiple part-level features extracted using pose estimation, human parsing, or uniform partitioning.
- 2.1. Deeply-learned part features for re-ID: PCB suffers a substantial performance decrease in partial re-ID and can fall below the global feature learning baseline, likely because precise part localization is fragile under severe misalignment.
- 2.2. Self-supervised learning: VPM differs from SPReID by learning part extraction through self-supervision rather than requiring an extra human parser and human parsing dataset.
- 2.2. Self-supervised learning: VPM predefines uniformly divided regions and directly predicts which regions are visible in a partial pedestrian image.
- 2.2. Self-supervised learning: Unlike prior self-supervision methods that transfer learned models to detection or classification, VPM uses visibility awareness explicitly to decide which regions to compare.
3. Proposed Method
VPM is a fully convolutional model that locates predefined regions, estimates their visibility, and extracts region-level features. It trains these components with self-supervision and compares images by emphasizing shared visible regions.
- 3.1. Structure of VPM: VPM transforms each pedestrian image into convolutional feature tensor T, then outputs a constant number of region-level features and visibility scores.The model uniformly divides holistic pedestrian images into p = m × n regions and processes inputs resized to H × W.
- 3.1. Structure of VPM: The region locator classifies every tensor pixel into predefined regions, producing one probability map per region for locating and extracting region features.A 1 × 1 convolution and Softmax estimate P(R_i|g), while each probability map indicates a region’s location on T.
- 3.1. Structure of VPM: Visibility scores accumulate region-assignment probabilities, becoming large for visible regions and small for possibly invisible regions.These scores determine how much each local distance contributes when two images are compared.
- 3.2. Matching: During matching, VPM computes region-to-region Euclidean distances and weights shared visible regions more heavily than regions invisible in either image.Unreliable local distances from regions invisible in one image contribute little to the overall distance.
- 3.3. Training VPM: Auxiliary self-supervision generates region labels and visible-region sets from random crops, training the locator and restricting feature learning to visible or shared regions.Cross-entropy uses visible regions, while the region-selective triplet loss focuses on shared regions; the overall loss combines region prediction, identity classification, and triplet losses.
- 3.3. Training VPM: The region-selective triplet loss trains features by pulling same-pedestrian representations together and pushing different-pedestrian representations apart using shared regions.Its training focus mirrors the matching strategy, with hard visibility labels during training and soft visibility scores during testing.
4. Experiment
Experiments evaluate VPM on synthetic and public partial re-ID datasets, comparing it with global and part-based baselines and testing crop strategies, part counts, and training losses. VPM consistently improves partial re-ID performance, with robustness and accuracy depending on visibility-aware alignment and matching the testing crop condition.
- Large-scale partial re-ID evaluation: VPM surpasses the global-feature baseline across all tested crop ratios on Market-1501.Rank-1 gains range from +4.3% to +6.4%, while mAP gains range from +4.4% to +13.1% as γ varies from 0.5 to 1.
- Large-scale partial re-ID evaluation: VPM is more robust than PCB when partial observations intensify, while achieving comparable accuracy with PCB in holistic re-ID.At γ = 0.5, PCB reaches 0.9% rank-1 accuracy, whereas VPM decreases much more slowly as γ decreases; at γ = 1.0, their retrieval accuracy is comparable.
- Large-scale partial re-ID evaluation: Increasing the number of predefined parts generally improves rank-1 accuracy by enabling finer region features and more accurate alignment, though computing efficiency must also be considered.The experiments use p values of 2, 3, 4, 6, and 8; most experiments set p = 6.
- Comparison with the state of the art: On public partial re-ID datasets, the crop strategy matters: VPM (Top) performs best on Partial-iLIDS, while VPM (Bilateral) performs best on Partial-REID.The bottom strategy performs worst because it deviates from the testing condition and retains regions with relatively weak discriminative clues.
- Comparison with the state of the art: With appropriate crop strategies, VPM surpasses SFR by +10.6% Rank-1 on Partial-REID and +3.3% Rank-1 on Partial-iLIDS.When the testing condition is unknown, VPM (Bilateral) is suggested because it considers both top and bottom occlusions and maintains stronger robustness.
- The importance of self-supervision: Ablations show that triplet loss improves feature discrimination, while visibility awareness and self-supervised visible-region learning are critical under partial re-ID.Removing visibility awareness causes a dramatic decrease on Partial-iLIDS, and learning from invisible regions introduces larger sample noise that compromises region features.
5. Conclusion
VPM addresses partial re-ID by learning visibility-aware region-level features and comparing images through their shared visible regions. It suppresses spatial misalignment and achieves performance on par with the state of the art.
- VPM learns which pre-defined pedestrian regions are visible in partial images through self-supervision.It locates each region on convolutional feature maps and extracts region-level features.
- Visibility awareness focuses comparisons on shared regions between pedestrian images, suppressing severe spatial misalignment in partial re-ID.
- VPM surpasses global feature learning and part-based convolutional baselines, with performance on par with the state of the art.