Source-linked AI summary
Semantics-Aligned Representation Learning for Person Re-identification
Xin Jin, Cuiling Lan, Wenjun Zeng, Guoqiang Wei, Zhibo Chen
TL;DR
Person reID must match images whose semantics are misaligned by pose, viewpoint, and occlusion. The paper introduces SAN, jointly training a reID encoder and texture-generating decoder with reconstruction and feature-level Triplet ReID supervision, then discarding the decoder at inference. SAN achieves state-of-the-art performance on CUHK03, Market1501, MSMT17, and Partial REID without increasing inference computational cost.
Problem
Pose, viewpoint, and occlusion variations produce semantic misalignment across 2D person images, while dense semantic alignment remains difficult because visible regions differ.
Method
SAN jointly trains a reID encoder and a decoder that predicts densely aligned full textures using pseudo-groundtruth supervision and Triplet ReID perceptual constraints.
Results
SAN achieves state-of-the-art performance on CUHK03, Market1501, MSMT17, and Partial REID without increasing computational cost during inference.
Takeaways & Limitations
Aligned texture-generation supervision trains semantics-aligned reID representations while the decoder is discarded during inference for efficient deployment.
Takeaways & Limitations
Pseudo-groundtruth generation is limited by a synthetic-to-real domain gap, only 929 SURREAL textures, and average replacement faces in SURREAL.
Abstract
from arXiv · showhide
Person re-identification (reID) aims to match person images to retrieve the ones with the same identity. This is a challenging task, as the images to be matched are generally semantically misaligned due to the diversity of human poses and capture viewpoints, incompleteness of the visible bodies (due to occlusion), etc. In this paper, we propose a framework that drives the reID network to learn semantics-aligned feature representation through delicate supervision designs. Specifically, we build a Semantics Aligning Network (SAN) which consists of a base network as encoder (SA-Enc) for re-ID, and a decoder (SA-Dec) for reconstructing/regressing the densely semantics aligned full texture image. We jointly train the SAN under the supervisions of person re-identification and aligned texture generation. Moreover, at the decoder, besides the reconstruction loss, we add Triplet ReID constraints over the feature maps as the perceptual losses. The decoder is discarded in the inference and thus our scheme is computationally efficient. Ablation studies demonstrate the effectiveness of our design. We achieve the state-of-the-art performances on the benchmark datasets CUHK03, Market1501, MSMT17, and the partial person reID dataset Partial REID. Code for our proposed method is available at: https://github.com/microsoft/Semantics-Aligned-Representation-Learning-for-Person-Re-identification.
1 Introduction
Person reID is challenged by spatial and visible-body semantic misalignment across images. SAN addresses these problems by supervising a reID encoder with densely aligned texture generation while discarding the decoder during inference.
- Pose, viewpoint, and occlusion variations create semantic misalignment across 2D person images, making reID challenging.
- Dense semantic alignment remains an open problem because warped images contain holes from invisible body regions and inconsistent visible semantics.
- SAN combines a reID encoder with a decoder that regresses densely semantically aligned full texture images using pseudo-groundtruth supervision.
- The decoder is discarded during inference, avoiding increased computational cost while retaining the training-time alignment supervision.
- The framework introduces a semantics alignment constraint by giving encoded feature maps aligned full-texture generation capability.
- Triplet ReID constraints over decoder feature maps supplement reconstruction loss as a high-level perceptual metric.
- The method generates pseudo-groundtruth textures from synthesized person-image and aligned-texture pairs because reID datasets lack aligned texture ground truth.
- The method achieves state-of-the-art performance on CUHK03, Market1501, MSMT17, and Partial REID.
2 Related Work
Prior reID work uses pose, part, and dense semantic cues to address misalignment, but coarse alignment and holes from invisible regions remain issues. This work instead uses densely aligned full texture images as direct supervision for learning aligned features.
- Pose and part cues help align body regions across person images, but their use typically requires external part-detection subnetworks during inference.
- DensePose-based warped-image methods address semantic alignment, yet invisible body regions create holes and inconsistent visible regions that can hurt learning efficiency.
- Efficient frameworks for dense semantic alignment remain under-explored, with limited direct constraints enforcing alignment.
- A texture image represents the full 3D human surface in canonical UV coordinates, where each position has a unique semantic identity.
- The proposed approach uses densely semantically aligned full texture images as direct supervision to drive the reID network toward aligned feature representations.
3 The Semantics Aligning Network (SAN)
SAN addresses person re-identification misalignment by training an encoder to produce reID features alongside a decoder that generates densely aligned full-body textures. It uses synthesized paired data and identity-aware decoder supervision, while discarding the decoder during inference.
- SAN framework: SAN combines an SA-Enc reID encoder with an SA-Dec decoder that predicts densely semantically aligned full texture images.The decoder supervises the encoder through aligned texture generation.
- Inference: During inference, SA-Dec is discarded, leaving the reID encoder as the deployed representation network.This separates training-time texture supervision from inference-time feature extraction.
- Texture supervision: Because reID datasets lack groundtruth aligned textures, the method synthesizes paired person-image and texture-image data for supervision.The PIT dataset is generated by rendering textured 3D meshes under varied poses, shapes, viewpoints, and backgrounds.
- Limitations: The pseudo-groundtruth generation process is limited by synthetic-to-real domain gaps, only 929 SURREAL texture maps, and averaged facial textures.The authors identify these constraints as future-work directions despite reporting performance improvements over the baseline.
- SA-Enc: The reID feature vector is obtained by spatially average-pooling the fourth encoder-block feature map and applying identification and triplet losses.The encoder can use a standard reID backbone such as ResNet-50.
- SA-Dec: The decoder uses L1 reconstruction loss and Triplet ReID constraints on decoder feature maps to encourage identity-preserving, aligned texture reconstruction.The Triplet ReID constraints serve as perceptual supervision across decoder layers or blocks.
4 Experiment
Experiments evaluate SAN through implementation choices, ablations, state-of-the-art comparisons, and partial person reID tests. Results support aligned texture supervision, decoder perceptual constraints, and strong performance without added inference cost.
- Experimental Setup: Experiments cover six benchmark person reID datasets and evaluate CMC at Rank-k and mAP.The reported ranks are 1, 5, and 10.
- Ablation Study: SAN combines pseudo-texture supervision, decoder Triplet ReID constraints, and synthesized PIT data in the ablation framework.SAN-basic uses reconstruction and reID losses; the final SAN adds both Triplet ReID constraints and PIT texture supervision.
- Ablation Study: 6.4% and 6.6% gains over the baseline in Rank-1 and mAP on CUHK03 occur with the same inference complexity.On Market1501, SAN improves Rank-1 by 2.0% and mAP by 4.8%.
- Ablation Study: Texture-image reconstruction performs best because it supplies comprehensive, densely semantics-aligned information, unlike input or fixed-pose reconstruction.Fixed-pose supervision may discard body information absent from the prescribed pose.
- Comparison with State-of-the-Arts: SAN achieves the best performance on CUHK03, Market1501, and MSMT17, consistently outperforming DSA-reID, while MGN performs better on DukeMTMC-reID.MGN ensembles local features at multiple granularities with global features.
- Ablation Study: Generated textures are semantically aligned across varied poses, viewpoints, and scales, but directly using them for reID performs worse because they are smoothed or blurred.More complex texture generators improve texture quality and further boost SAN-basic reID performance.
- Partial Person ReID: On Partial REID, SAN* improves over Baseline* by 5.8%, 4.7%, and 7.8% on Rank-1, Rank-5, and Rank-10.On Partial-iLIDS, the corresponding improvements are 7.6%, 7.8%, and 5.8%; SAN also significantly outperforms the baseline without fine-tuning.
5 Conclusion
The paper proposes SAN to learn semantics-aligned reID representations through joint person reID and aligned texture-generation supervision. It reports state-of-the-art performance across standard and partial reID datasets without increasing inference cost.
- SAN jointly supervises person reID and semantics-aligned texture generation, adding decoder Triplet ReID constraints as perceptual losses.The decoder is discarded during inference.
- SAN achieves state-of-the-art performance on CUHK03, Market1501, MSMT17, and Partial REID without increasing inference computational cost.The PIT dataset supports pseudo-groundtruth texture generation and SAN training.