Source-linked AI summary
Part-Aligned Bilinear Representations for Person Re-identification
Yumin Suh, Jingdong Wang, Siyu Tang, Tao Mei, Kyoung Mu Lee
TL;DR
Person re-identification is hindered by body-part misalignment across poses, viewpoints, and imperfect detections. The paper learns part-aligned representations by combining appearance and part maps with bilinear pooling, using pose-estimation initialization without part annotations, and reports superior or competitive benchmark performance.
Problem
Body-part misalignment makes conventional global and strip/grid representations unreliable for person re-identification.
Method
A two-stream network combines appearance and learned part maps through bilinear mapping and spatial pooling to produce a normalized part-aligned representation.
Results
The approach achieves superior or competitive performance on standard image and video person re-identification benchmarks.
Takeaways & Limitations
Learned part descriptors provide fine-grained, robust body-part differentiation tailored to person re-identification.
Abstract
from arXiv · showhide
We propose a novel network that learns a part-aligned representation for person re-identification. It handles the body part misalignment problem, that is, body parts are misaligned across human detections due to pose/viewpoint change and unreliable detection. Our model consists of a two-stream network (one stream for appearance map extraction and the other one for body part map extraction) and a bilinear-pooling layer that generates and spatially pools a part-aligned map. Each local feature of the part-aligned map is obtained by a bilinear mapping of the corresponding local appearance and body part descriptors. Our new representation leads to a robust image matching similarity, which is equivalent to an aggregation of the local similarities of the corresponding body parts combined with the weighted appearance similarity. This part-aligned representation reduces the part misalignment problem significantly. Our approach is also advantageous over other pose-guided representations (e.g., extracting representations over the bounding box of each body part) by learning part descriptors optimal for person re-identification. For training the network, our approach does not require any part annotation on the person re-identification dataset. Instead, we simply initialize the part sub-stream using a pre-trained sub-network of an existing pose estimation network, and train the whole network to minimize the re-identification loss. We validate the effectiveness of our approach by demonstrating its superiority over the state-of-the-art methods on the standard benchmark datasets, including Market-1501, CUHK03, CUHK01 and DukeMTMC, and standard video dataset MARS.
1 Introduction
Person re-identification must match identities across disjoint camera views despite changes in lighting, pose, viewpoint, and imperfect detections. The paper targets body-part misalignment, which makes global and grid-based representations unreliable, with a learned part-aligned representation.
- Motivation: Person re-identification identifies the same person across videos captured by different cameras, despite disjoint views and varying conditions.The stated challenges include variable transition times, lighting, and poses.
- Motivation: Body parts become spatially misaligned across person images because of pose, viewpoint, and imperfect detection changes.This misalignment is identified as a key challenge for person re-identification.
- Motivation: Global and strip/grid-based representations are unreliable because they implicitly assume similar poses inside tightly surrounded bounding boxes.The paper argues that a body part-aligned representation can ease comparison without complex matching techniques.
- Proposed solution: The proposed method learns body poses as part maps and combines them directly with appearance maps to compute part-aligned representations.It uses a two-stream network and an aggregation module based on bilinear mapping and spatial averaging.
- Proposed solution: The part-map stream is initialized from a pre-trained pose-estimation sub-network, while the whole network is trained without body-part annotations on the re-identification dataset.The approach uses the re-identification loss for training.
2 Related Work
Prior work uses spatial partitioning, pose-guided parts, attention, specialized matching, and bilinear pooling to address person re-identification. The proposed approach differs by bilinearly aggregating appearance and part maps to produce part-aligned representations rather than merely augmenting appearance features.
- Prior approaches: Earlier person re-identification methods used hand-crafted features, metric learning, probabilistic patch matching, attributes, transfer learning, re-ranking, partial matching, and human-in-the-loop learning.The paper reviews these methods alongside spatial-partition and part-aligned representations.
- Part representations: Attention-map methods can produce unreliable attentions because they estimate attention from appearance maps without body-part-location guidance.The paper links this limitation to restricted performance.
- Part-aligned representations: Tang et al. used part maps as an additional appearance feature, whereas this method uses them to generate part-aligned representations.This distinguishes the proposed use of part maps from multi-people-tracking work.
- Bilinear pooling: Unlike bilinear pooling of two appearance maps, the proposed bilinear pooling aggregates appearance and part maps to compute part-aligned representations.The paper contrasts its design with a prior bilinear method reported to perform poorly for person re-identification.
3 Our Approach
The approach uses separate appearance and part-map streams, bilinear pooling, spatial average pooling, normalization, and triplet-loss training. The resulting representation encodes appearance through part descriptors while avoiding reliance on manually annotated parts.
- 3.1 Two-Stream Network: The two-stream network separately extracts an appearance map and a body part map from the input image.The appearance extractor is initialized from GoogLeNet, while the part extractor is initialized from OpenPose.
- 3.2 Bilinear Pooling: Bilinear pooling combines corresponding local appearance and part descriptors to form a part-aligned representation.The operation consists of bilinear transformation followed by spatial global pooling.
- 3.2 Bilinear Pooling: Spatial average pooling aggregates local part-aligned descriptors into the feature f, which is then normalized into the final representation ˜f.S denotes the spatial size, and the pooling averages over spatial locations.
- 3.2 Bilinear Pooling: The outer-product representation can be decomposed into sub-vectors, with each sub-vector corresponding to a part channel and selecting appearance features according to part information.A channel active on a knee, for example, retains appearance only at the knee location.
- 3.3 Loss: Training uses triplets consisting of a query, same-person positive, and different-person negative image.The triplet loss uses a margin and similarity between the resulting representations; the margin is set to 0.2.
4 Analysis
The analysis shows that bilinear aggregation produces part-aware image similarity, while learned part descriptors organize body regions smoothly and independently of appearance.
- Part-aware image similarity: Image similarity aggregates local appearance similarities between corresponding body parts.The proposed part-aligned representation expresses image similarity as a sum of local similarities across locations.
- Part-aware image similarity: Bilinear pooling weights local appearance similarity by corresponding body part similarity.This weighting provides the interpretation of the proposed local similarity used in image matching.
- Relationship to baseline models: The appearance-only baseline cannot reflect part similarity, whereas box-based representations are special cases of the proposed model.The proposed model is trained with the re-identification loss rather than restricting part maps to predefined bounding-box indicators.
- Descriptor visualization: t-SNE shows appearance descriptors cluster by color independently of body part, while part descriptors cluster by body part independently of color.The learned part embedding also places physically adjacent regions near one another.
- Map visualization: Appearance maps differentiate people while remaining invariant within each person, whereas part maps encode body parts independently of appearance.The visualization uses separate appearance and part maps to illustrate this decomposition.
- Learned part descriptors: The learned part descriptor maps body regions smoothly, making descriptors less sensitive to pose-estimation error than rapidly changing joint-based descriptors.Adjacent body parts are mapped closely in the proposed representation, unlike the baseline in the described example.
5 Implementation Details
Implementation uses separate appearance and pose-derived part sub-networks, compact bilinear pooling, and triplet-based stochastic training with multiple pose-map variants.
- Network architecture: The appearance extractor uses GoogLeNet features and can employ dilation filters to produce 20 × 10 response maps.The extractor reduces the feature dimension to 512 after inception4e.
- Compact bilinear pooling: Compact bilinear pooling uses tensor sketch to reduce the computational and memory cost of bilinear aggregation.The original bilinear transformation combines 512-dimensional appearance vectors with 128-dimensional part vectors.
- Network training: The network is fine-tuned from ImageNet- and COCO-pretrained models using stochastic gradient descent for 75,000 iterations.The learning rate is reduced by a factor of 5 every 20,000 iterations.
- Network training: Each mini-batch contains 180 images and yields approximately 3 × 10^5 triplets for training.The sampling averages 18 identities with 10 images per identity and uses all possible within-batch triplets.
- Pose sub-network: On Market-1501, the proposed joint-limb-internal pose sub-network achieves similar accuracy to joint and limb variants, while internal maps provide complementary information.The final model concatenates joint, limb, and internal confidence maps, totaling 185 channels.
6 Experiments
Experiments evaluate the method across image- and video-based person re-identification benchmarks, using CMC and mAP where multiple gallery matches exist. The proposed approach consistently outperforms baselines and competing methods across datasets and settings.
- Evaluation: CMC measures correct-match identification at each rank, while mAP evaluates ranking quality when multiple ground-truth images exist.mAP is reported for Market-1501, DukeMTMC, and MARS.
- Baseline comparisons: Using part maps consistently improves baseline accuracy across Market-1501, MARS, and DukeMTMC, with further gains when the part extractor is fine-tuned.Bilinear pooling also outperforms concat+averagepool+linear aggregation for fixed and trainable part extractors.
- Baseline comparisons: 4% rank@1 accuracy and 9% mAP improvements are reported over the compared pose-guided methods on Market-1501.The comparison uses an Inception V1 appearance network and R-CPM part network.
- Image benchmarks: The proposed method achieves the best mAP and rank@K on Market-1501 for both single-query and multi-query evaluation.It also outperforms other methods on expanded Market-1501 gallery sets containing up to 519,732 images.
- Image benchmarks: The method achieves state-of-the-art accuracy on CUHK03 with detected boxes and the best accuracy with manually labeled boxes.On CUHK01, it performs best with 486 test identities and second best with 100 test identities.
- Video benchmarks: The proposed method achieves the best DukeMTMC result with and without re-ranking and the highest accuracy on MARS among image- and video-based approaches.For MARS, frame representations are aggregated by temporal average pooling, with accuracy similar to RNN and LSTM aggregation schemes.
7 Conclusions
The paper concludes that learned part maps and part-aligned representations address body-part misalignment while supporting strong performance across standard image and video benchmarks.
- Contributions: Part maps are learned for person re-identification using guidance from a pre-trained pose estimation model rather than predefined parts.Training minimizes the re-identification loss.
- Contributions: The part-map representation provides fine-grained and robust body-part differentiation based on each part’s usefulness for re-identification.
- Contributions: Part-aligned representations address body-part misalignment and achieve superior or competitive performance on standard image and video benchmarks.
A.1 Details of the Visualization
The visualization details describe how local appearance descriptors are mapped into a two-dimensional space using t-SNE.
- Visualization: Normalized local appearance descriptors are visualized in two dimensions using t-SNE.
Figure 4 and 5
Figures 4 and 5 visualize normalized local descriptors by projecting their principal components into RGB space.
- Visualization: The top three principal components of normalized local appearance or part descriptors are mapped to RGB values for feature-map visualization.
A.2 Additional Visualization Examples of Feature Maps
On MARS, the proposed network produces separate appearance and body-part maps, with appearance features distinguishing identities and part features encoding body parts independently of appearance. A non-negative part-descriptor variant retains similar rank@1 accuracy while slightly improving mAP.
- Feature-map visualization: Appearance maps distinguish different persons while remaining invariant for each person.The visualization examples are drawn from the MARS dataset.
- Feature-map visualization: Part maps encode body parts independently from their appearance.The figure presents input images alongside their appearance and part maps.
- Variant comparison: The proposed model and non-negative variant show similar rank@1 accuracy, while the non-negative variant slightly improves mAP.The variant enforces element-wise non-negative part descriptors with a ReLU layer after the part map extractor.
- Body-joint and limb representation: The part map extractor uses an OpenPose sub-network producing a 185-dimensional map of joint, limb, and internal features.The map comprises 19 joint-confidence, 38 limb-confidence, and 128 internal-feature dimensions.