Source-linked AI summary
Pose-Normalized Image Generation for Person Re-identification
Xuelin Qian, Yanwei Fu, Tao Xiang, Wenxuan Wang, Jie Qiu, Yang Wu, Yu-Gang Jiang, Xiangyang Xue
TL;DR
Re-id must handle large pose variations while lacking paired cross-view training data and requiring scalable generalization. The paper proposes PN-GAN to synthesize identity-preserving images in canonical poses and train complementary pose-normalized features. Across benchmarks, the model improves re-id performance and transfers to new datasets without additional data collection or fine-tuning.
Problem
Existing re-id models require sufficient labeled identities and images across camera views while struggling to learn identity-sensitive, view-insensitive features under large pose variations.
Method
PN-GAN conditionally replaces a person image’s pose with a desired canonical pose, then combines re-id features learned from synthesized pose-normalized images with features from original images.
Results
The model improves over ResNet-50-A baselines across three transfer-learning datasets and achieves state-of-the-art performance across four benchmarks.
Takeaways & Limitations
The model can be applied to new re-id datasets from new camera networks without additional data collection or model fine-tuning.
Takeaways & Limitations
The problem formulation assumes a training dataset containing person images paired with person identities.
Abstract
from arXiv · showhide
Person Re-identification (re-id) faces two major challenges: the lack of cross-view paired training data and learning discriminative identity-sensitive and view-invariant features in the presence of large pose variations. In this work, we address both problems by proposing a novel deep person image generation model for synthesizing realistic person images conditional on the pose. The model is based on a generative adversarial network (GAN) designed specifically for pose normalization in re-id, thus termed pose-normalization GAN (PN-GAN). With the synthesized images, we can learn a new type of deep re-id feature free of the influence of pose variations. We show that this feature is strong on its own and complementary to features learned with the original images. Importantly, under the transfer learning setting, we show that our model generalizes well to any new re-id dataset without the need for collecting any training data for model fine-tuning. The model thus has the potential to make re-id model truly scalable.
1. Introduction
Pose variation is the main source of appearance changes across camera views and makes scalable, identity-sensitive re-id difficult. The proposed PN-GAN framework normalizes pose through identity-preserving image synthesis and learns complementary re-id features for broader generalization.
- Challenge: Pose combines viewpoint and body configuration and is identified as the most important factor changing a person’s appearance across views.It can also cause self-occlusion, such as backpacks appearing differently from front and back views.
- Challenge: Existing re-id models require many labeled images per identity and camera view, limiting scalability across large camera networks.Manual cross-view annotation is tedious and difficult, especially when networks contain hundreds of cameras.
- Challenge: Even with sufficient labeled data, models struggle to separate identity-sensitive properties from pose-related, view-sensitive appearance changes.Identity properties and covariates such as pose are interconnected in the learned representations.
- Approach: The framework argues that removing pose variation can reduce data demands, focus learning on identity-sensitive features, and improve generalization to new camera networks.Features learned without pose variation are also expected to complement features learned from original images.
- Approach: PN-GAN takes a person image and desired pose as input, synthesizes the same identity in that pose, and supports pose-normalized re-id feature learning.Eight canonical poses produce eight synthesized images per input, and the resulting features are fused with features from original images.
2. Related Work
Prior work learns view-invariant re-id features with diverse DNN architectures, objectives, and GAN-based image-generation approaches. PN-GAN differs by conditionally changing pose while preserving identity, producing strongly labeled pose-controlled images.
- Deep re-id models: Deep re-id models differ in DNN architecture and training objectives, including identity classification, pairwise verification, and triplet ranking losses.These approaches generally target discriminative features that are insensitive to viewpoint.
- Deep image generation: GANs jointly optimize a discriminator and generator through a min-max game to distinguish real training data from generated samples.The paper builds its pose-normalization GAN on deep convolutional generative adversarial networks.
- PN-GAN: PN-GAN uses conditional generation to change pose while preserving the input person’s identity.Its residual-block design is intended to retain identity-related information while modifying pose.
- PN-GAN: Unlike the prior re-id image generator described here, PN-GAN controls both identity and pose, enabling strongly labeled synthetic training data.The earlier model’s generated images could only be used as unlabeled or weakly labeled data.
3. Methodology
The methodology combines pose-conditioned person-image generation with two complementary re-identification feature streams. PN-GAN normalizes images to eight canonical poses while preserving identity-related information, and the resulting features are fused with features learned from original images.
- The framework defines re-identification as comparing image features using Euclidean distance to judge whether two images share an identity.
- Deep Image Generator: PN-GAN generates a target-pose image of the same person by conditioning on a source image and a desired pose image.The generator replaces the source pose while aiming to preserve the person identity.
- Pose Normalization: Pose estimation uses pretrained OpenPose outputs, and eight canonical poses are used as the normalization targets.The canonical poses are obtained by clustering pose images, with cluster means serving as canonical poses.
- Deep Image Generator: The generator concatenates the source person image with a three-channel target pose image and uses a ResNet encoder-decoder with nine residual blocks.Residual connections pass relatively invariant identity, clothing, and background information toward decoding while pose information is changed.
- Deep Image Generator: The generator objective combines adversarial and L1 reconstruction losses, while training alternately minimizes the generator and discriminator losses until convergence.The L1 term is used to produce sharper and cleaner reconstructed images, with λ1 balancing the loss terms.
- Person Re-identification with Pose Normalization: Two ResNet-50 re-identification models learn complementary features from original images and pose-normalized synthetic images, which are fused with element-wise maximum pooling.At testing, one original-image feature and eight pose-normalized features are obtained for each image; fusion produces the final representation used for Euclidean-distance ranking.
4. Experiments
Experiments evaluate PN-GAN on four re-id benchmarks under supervised and transfer-learning settings, finding gains from pose-normalized images and complementary features. Transfer results improve over the ResNet-50-A baseline, while eight canonical poses outperform using one pose.
- Datasets and Settings: Experiments use four benchmark datasets and evaluate Rank-1, Rank-5, Rank-10, and, for Market-1501 and DukeMTMC-reID, mAP.The datasets include Market-1501, CUHK03, CUHK01, and DukeMTMC-reID, with supervised and transfer-learning settings.
- Supervised Learning Results: In supervised learning, PN-GAN improves over ResNet-50-A baselines on all three large-scale datasets and outperforms existing pose-guided re-id models.The authors attribute the advantage to synthesizing multiple normalized poses for handling large pose variation.
- Supervised Learning Results: On CUHK01, pose-normalized images boost ResNet-50-A by more than 3% in the supervised setting.The passage also notes that handcrafted-feature and metric-learning methods remain competitive on this smaller dataset.
- Transfer Learning Results: 27.58% Rank-1 accuracy is achieved on CUHK01 under transfer learning, comparable to some supervised models, while outperforming the ResNet-50-A transfer baseline on all three datasets.Transfer-learning performance remains much lower than supervised performance because the setting is intrinsically difficult.
- Further Evaluations: Combining the two feature types improves final results on all four datasets, indicating that features from original and synthesized images are complementary.Each feature model is reported as strong independently, while their combination yields further improvement.
- Further Evaluations: 72.58 to 69.60 on Market-1501 mAP is the drop when features use eight poses instead of one, supporting the benefit of eight canonical poses.Using all eight poses reduces sensitivity to poor generated-image quality under a particular pose.
5. Conclusion
The paper proposes pose-normalized synthetic images and complementary features from original and generated images for person re-id. Across four benchmarks, it reports state-of-the-art performance and transfer to new camera networks without additional data collection or fine-tuning.
- The method synthesizes pose-normalized person images and learns complementary features from original and synthetic images.The generated images are used alongside features learned from original images.
- Extensive experiments on four benchmarks show that the proposed model achieves state-of-the-art performance.
- The model generalizes to new re-id datasets from new camera networks without additional data collection or model fine-tuning.