Source-linked AI summary
Learning to Adapt Invariance in Memory for Person Re-identification
Zhun Zhong, Liang Zheng, Zhiming Luo, Shaozi Li, Yi Yang
TL;DR
Unsupervised domain adaptation for person re-ID must transfer knowledge from labeled source data to unlabeled target data despite domain shifts and overlooked target-sample relations. The paper learns three target-domain invariances through an exemplar memory and uses GPP to infer reliable neighbors. Experiments report state-of-the-art adaptation accuracy on three large-scale domains, while the authors note trade-offs in neighborhood-based positives.
Problem
Existing UDA re-ID methods often reduce inter-domain shifts while overlooking target-domain sample relations and intra-domain variations.
Method
The framework learns exemplar-, camera-, and neighborhood-invariance using exemplar memory, with GPP inferring reliable target neighbors from candidate-neighbor relationships.
Results
The approach produces new state-of-the-art UDA accuracy on three large-scale re-ID domains, and experiments find the three invariance properties, memory, and GPP beneficial.
Takeaways & Limitations
Target-domain intra-domain variations and relations can be incorporated into re-ID adaptation through global memory-based invariance learning.
Takeaways & Limitations
Neighborhood-invariance may pull images from different identities closer because reliable neighbors cannot be guaranteed to share identity.
Abstract
from arXiv · showhide
This work considers the problem of unsupervised domain adaptation in person re-identification (re-ID), which aims to transfer knowledge from the source domain to the target domain. Existing methods are primary to reduce the inter-domain shift between the domains, which however usually overlook the relations among target samples. This paper investigates into the intra-domain variations of the target domain and proposes a novel adaptation framework w.r.t. three types of underlying invariance, i.e., Exemplar-Invariance, Camera-Invariance, and Neighborhood-Invariance. Specifically, an exemplar memory is introduced to store features of samples, which can effectively and efficiently enforce the invariance constraints over the global dataset. We further present the Graph-based Positive Prediction (GPP) method to explore reliable neighbors for the target domain, which is built upon the memory and is trained on the source samples. Experiments demonstrate that 1) the three invariance properties are indispensable for effective domain adaptation, 2) the memory plays a key role in implementing invariance learning and improves the performance with limited extra computation cost, 3) GPP could facilitate the invariance learning and thus significantly improves the results, and 4) our approach produces new state-of-the-art adaptation accuracy on three re-ID large-scale benchmarks.
1 INTRODUCTION
Unsupervised domain adaptation for re-ID must address target-domain intra-domain variations, not only source-target shifts. The paper proposes learning exemplar-, camera-, and neighborhood-invariance with memory and GPP, reporting improved adaptation across three benchmarks.
- Problem: UDA re-ID transfers knowledge from labeled source data to unlabeled target data, but domain shifts can substantially degrade performance.Shifts arise from changes in scenario, season, illumination, and camera deployment.
- Problem: Direct feature-distribution alignment is unsuitable for open-set re-ID because disjoint source and target identities can be aligned incorrectly.The paper describes re-ID domain adaptation as an open-set problem with completely disjoint classes or identities.
- Motivation: Existing adaptation methods often overlook target-domain intra-domain variations, including changes in appearance, camera style, pose, view, and background.The paper frames these variations as important factors affecting re-ID performance.
- Approach: The framework learns Exemplar-Invariance, Camera-Invariance, and Neighborhood-Invariance to model target-domain relationships and improve adaptation.EI separates exemplars, CI brings an image close to its CamStyle counterparts, and NI brings an exemplar close to reliable neighbors.
- Approach: An exemplar memory stores up-to-date representations for all training samples, enabling invariance constraints over the global dataset rather than only a mini-batch.The memory improves performance with limited extra computation cost and GPU memory.
- Results: Experiments report that the three invariance properties are indispensable and that the method outperforms state-of-the-art UDA approaches on Market-1501, DukeMTMC-reID, and MSMT17.The paper describes the improvements as large-margin gains across all three large-scale benchmarks.
- Approach: GPP uses relationships among candidate neighbors to infer reliable positives, facilitating neighborhood-invariance and consistently improving results, especially mAP.The method is built upon the exemplar memory and addresses inaccuracies from directly selecting nearest neighbors.
2 RELATED WORK
Prior work in unsupervised re-ID addresses domain shift through distribution alignment or target pseudo-labeling, while this method combines memory and graph-based neighbor prediction to model target relations.
- Unsupervised domain adaptation: Unsupervised domain adaptation methods commonly reduce source-target discrepancy through distribution alignment, but re-ID is an open-set problem with disjoint identities.Directly aligning distributions can associate samples from different identities and harm adaptation accuracy.
- Unsupervised person re-identification: Unsupervised re-ID methods mainly discover target pseudo-labels or reduce source-target discrepancy in a common label space.Pseudo-labels may be obtained through clustering, source-label association, nearest neighbors, or camera-style correspondences.
- Neural networks with augmented-memory: Augmented-memory networks store intermediate knowledge in structural, addressable tables across tasks including question answering, few-shot learning, and video understanding.This work applies an exemplar-memory design to feature storage for domain adaptation.
- Graph convolutional network: GPP predicts reliable neighbors across graphs formed by different candidates and is designed to find neighbors that help generalize re-ID networks.It differs from related graph methods by focusing on reliable neighbor discovery for re-ID generalization.
3 THE PROPOSED METHOD
The paper formulates unsupervised re-ID adaptation with labeled source data and unlabeled target data drawn from different distributions and containing completely different identities.
- Problem formulation: The adaptation setting provides a fully labeled source domain and an unlabeled target domain whose identities are completely different.The source contains person images associated with identity labels, whereas target identity annotations are unavailable.
- Problem formulation: The goal is to learn a transferable deep re-ID model that generalizes well on the target testing set.Both labeled source and unlabeled target data are exploited during training.
3.1 Overview of Framework
The proposed framework uses source and target training branches, each with exemplar memory, to learn source representations and enforce target invariance.
- Overview of Framework: Inputs from labeled source and unlabeled target domains pass through a feature extractor before separate source and target branches optimize the model.The extractor uses convolutional layers, global average pooling, and batch normalization, with ResNet-50 residual blocks pretrained on ImageNet.
- Overview of Framework: Each branch stores up-to-date source or target features in an exemplar memory module.The memory acts as feature storage for every sample in the corresponding domain.
- Overview of Framework: The source branch learns basic representations and trains a graph-based positive prediction network, while the target branch enforces invariance learning.GPP is trained using source-memory candidate neighbors and applied to infer reliable positive neighbors for target samples.
3.2 Supervised Learning for Source Domain
The source branch treats supervised training as identity classification, using labeled source images to learn a basic discriminative re-ID model before target adaptation.
- 3.2 Supervised Learning for Source Domain: The source-domain training process is formulated as an identity classification problem.The source data are labeled, enabling supervised learning of a basic discriminative model.
- 3.2 Supervised Learning for Source Domain: The supervised source-domain objective is calculated with cross-entropy loss.The loss is based on the identity classifier’s predicted probability for each source image’s associated identity.
- 3.2 Supervised Learning for Source Domain: A model trained on labeled source data performs well on same-distribution testing but deteriorates significantly on unseen target data.The deterioration is attributed to domain shift and becomes more serious as domain shift increases.
3.3 Invariance Learning for Target Domain
The framework addresses target-domain intra-domain variation through exemplar-, camera-, and neighborhood-invariance, implemented globally with an exemplar memory. These constraints respectively separate individual exemplars, align camera-style variants, and bring reliable neighbors closer.
- Three Properties of Target Invariance: The method targets three properties of target-domain invariance: Exemplar-Invariance, Camera-Invariance, and Neighborhood-Invariance.It focuses on intra-domain variations such as appearance, camera style, pose, view, and background.
- Target Invariance Learning with Memory: The exemplar memory stores up-to-date target features so similarities and invariance constraints can be computed over the whole target dataset rather than only a mini-batch.Each slot stores an L2-normalized feature, updated during training with a controlled update rate.
- Exemplar-Invariance: Exemplar-invariance treats each target image as its own class, encouraging it to remain close to itself and far from other exemplars.This is intended to improve discrimination based on apparent representation when person images vary substantially within an identity.
- Camera-Invariance: Camera-invariance classifies each real target image and its camera-style-transferred counterparts as the same class.CamStyle generates counterparts in other target-camera styles while preserving the original identity to some extent.
- Neighborhood-Invariance: Neighborhood-invariance classifies a target image toward the classes of its reliable neighbors using a soft-label objective.Reliable neighbors are selected from the memory, and the target image is not assigned to its own class in this constraint.
- Overall Loss of Invariance Learning: The overall invariance objective jointly considers exemplar-, camera-, and neighborhood-invariance, with the selected class determining which constraint is applied.Self-classification supports exemplar- and camera-invariance, while classification toward other selected classes supports neighborhood-invariance.
3.4 Graph-based Positive Prediction
GPP improves neighbor selection by modeling relationships among memory features instead of independently ranking similarities. It constructs a candidate graph, refines node features with GCNs, predicts positive probabilities, and selects reliable target neighbors.
- Motivation: Vanilla neighbor selection chooses the top-k memory samples by cosine similarity, but independent similarity estimates can misidentify hard positives and negatives.The problem is especially pronounced when the target-domain model has poor discriminative ability.
- Graph-based Positive Prediction: GPP uses graph convolutional networks and a positive classifier to refine memory-based similarities by leveraging relationships among candidate features.The network is trained with labeled source samples and then applied to target samples.
- Graph Construction: For an input sample, GPP computes similarities with memory features, selects top-k candidates, and constructs a complete undirected graph from their node features.The candidate node features are normalized relative to the input embedding, and edge weights are represented by an adjacency matrix.
- Feature Updating with GCNs: GCNs take node features and an adjacency matrix as input and output refined node representations for positive prediction.The implementation uses four graph convolutional layers before classification.
- Positive Prediction: A positive classifier predicts each candidate node's probability of belonging to the input sample's positive class, and source-domain positive classification loss trains GPP.The target-domain inference process omits loss computation and uses predicted probabilities to select reliable neighbors above a threshold.
- Reliable Neighbors Selection on Target Domain: On the target domain, GPP obtains candidate-neighbor probabilities from the target memory and selects reliable neighbors according to the predicted probabilities and threshold μ.This selection supplies neighbors for subsequent neighborhood-invariance learning.
3.5 Final Loss for Network
The final network objective combines supervised source learning, target invariance learning, and graph-based positive prediction learning. Each term has a distinct role in representation maintenance, target adaptation, or reliable-neighbor prediction.
- Final Loss: The overall loss combines Lsrc, Ltgt, and Lgpp.Lsrc maintains a basic person representation, Ltgt incorporates target-domain invariance, and Lgpp learns reliable-neighbor prediction.
- Final Loss: Lgpp is computed only on labeled source data and updates only the GPP network.The target invariance term transfers source knowledge while incorporating target-domain invariance properties.
3.6 Discussion on the Three Invariance Properties
Exemplar-invariance and neighborhood-invariance impose opposing pressures on target samples, while camera-invariance has an effect similar to exemplar-invariance. Their interaction creates a trade-off between separating dissimilar identities and grouping likely same-identity samples.
- Exemplar-invariance separates exemplars, enlarging distances between different identities but also pushing same-identity exemplars apart.
- Neighborhood-invariance brings exemplars and neighbors closer, reducing distances between same-identity samples but potentially attracting different identities.
- Together, exemplar-invariance and neighborhood-invariance form a trade-off between inter-identity separation and same-identity grouping.
- Neighborhood-invariance can progressively form local clusters through neighbor relations and shared-neighbor connections.
- Camera-invariance has an effect similar to exemplar-invariance by encouraging separation of dissimilar groups.
4 EXPERIMENT
Experiments evaluate the framework on three large-scale re-ID benchmarks using standard target-domain CMC and mAP metrics. Ablations show complementary benefits from the three invariances, while memory and GPP improve adaptation with reported computation and memory costs.
- The method is evaluated on Market-1501, DukeMTMC-reID, and the more challenging MSMT17 benchmark.
- Target-domain performance is measured with CMC rank-1, rank-5, rank-10, rank-20, and mean Average Precision.
- Parameter Analysis: β = 0.05 produces the best results, whereas an extremely low β = 0.01 prevents convergence and yields poor performance.
- Ablation Study: Rank-1 accuracy improves from 43.1% to 48.7% on Market-1501 and from 28.9% to 34.2% on DukeMTMC-reID with exemplar-invariance.
- Ablation Study: 84.1% rank-1 accuracy is achieved with EI+CI+NI transferring from DukeMTMC-reID to Market-1501, exceeding EI+CI by 21% and EI+NI by 12.3%.
- Memory and GPP: Memory-based training outperforms mini-batch training with approximately +0.02 s/iter and +200 MB GPU memory.
- Memory and GPP: GPP consistently outperforms VNS, while threshold-based reliable-neighbor selection is better than fixed top-k selection.
- Comparison with State-of-the-art Methods: On MSMT17, the method reaches 42.5% rank-1 accuracy and 16.0% mAP, exceeding ECN by 12.3% rank-1 and 5.8% mAP.
5 CONCLUSION
The paper concludes that exemplar memory, three target-domain invariances, and GPP improve transferable person re-ID and achieve state-of-the-art UDA accuracy across three large-scale domains.
- Exemplar memory enables relationships among target samples to enforce invariance constraints during training.
- GPP uses memory to infer reliable neighbors from candidate neighbors for neighborhood-invariance learning.
- The approach achieves new state-of-the-art unsupervised domain adaptation accuracy on three large-scale re-ID domains.