PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-paced Contrastive Learning with Hybrid Memory for Domain Adaptive Object Re-ID

Yixiao Ge, Feng Zhu, Dapeng Chen, Rui Zhao, Hongsheng Li

arXiv:2006.02713v2cs.CV

TL;DR

UDA object re-ID must transfer knowledge from labeled source identities to unlabeled target identities, but prior methods underuse available data and suffer from unreliable clustering. This paper introduces self-paced contrastive learning with hybrid memory that jointly uses source classes, target clusters, and un-clustered instances. It outperforms state-of-the-art methods across multiple adaptation tasks and also improves source-domain performance without extra annotations.

  • Problem

    Prior pseudo-label-based UDA methods do not fully use source and target information because source data may be discarded and unreliable clustering produces discarded outliers or noisy pseudo labels.

  • Method

    A self-paced contrastive learning framework uses hybrid memory to jointly distinguish source classes, target clusters, and target un-clustered instances while progressively retaining reliable clusters.

  • Results

    The method significantly outperforms state-of-the-art methods on multiple domain adaptation tasks, with up to 5.0% mAP gains, and boosts source-domain performance by 6.6%.

  • Takeaways & Limitations

    Jointly exploiting all available data with hybrid supervision improves object re-ID adaptation and can promote supervised source-domain performance using unlabeled target data.

  • Takeaways & Limitations

    Performance remains below the oracle, indicating that pseudo-class labels may still be unsatisfactory even with self-paced learning.

Abstract

from arXiv · show

Domain adaptive object re-ID aims to transfer the learned knowledge from the labeled source domain to the unlabeled target domain to tackle the open-class re-identification problems. Although state-of-the-art pseudo-label-based methods have achieved great success, they did not make full use of all valuable information because of the domain gap and unsatisfying clustering performance. To solve these problems, we propose a novel self-paced contrastive learning framework with hybrid memory. The hybrid memory dynamically generates source-domain class-level, target-domain cluster-level and un-clustered instance-level supervisory signals for learning feature representations. Different from the conventional contrastive learning strategy, the proposed framework jointly distinguishes source-domain classes, and target-domain clusters and un-clustered instances. Most importantly, the proposed self-paced method gradually creates more reliable clusters to refine the hybrid memory and learning targets, and is shown to be the key to our outstanding performance. Our method outperforms state-of-the-arts on multiple domain adaptation tasks of object re-ID and even boosts the performance on the source domain without any extra annotations. Our generalized version on unsupervised object re-ID surpasses state-of-the-art algorithms by considerable 16.7% and 7.9% on Market-1501 and MSMT17 benchmarks.

1 Introduction

UDA object re-ID methods adapt from labeled source data to unlabeled target data, but often discard useful source or un-clustered target samples. The proposed framework uses hybrid memory and self-paced learning to exploit these data and progressively improve target supervision.

  • Problem: Existing UDA methods typically pre-train on labeled source data before adapting to the target domain with pseudo labels.The source and target domains contain different identities.
  • Limitations: Source-domain images and accurate source labels were often ignored during target fine-tuning, or even found harmful by prior methods.This leaves valuable source-domain supervision unused.
  • Limitations: Existing methods also discard target-domain outliers to improve pseudo-label reliability, despite their potential value and prevalence in early training.Many such outliers may be difficult samples rather than unusable data.
  • Method: The hybrid memory jointly uses source class centroids, target cluster centroids, and target un-clustered instance features as supervisory signals.A unified framework dynamically updates and distinguishes these memory entries.
  • Method: Self-paced contrastive learning starts from the most reliable target clusters and gradually incorporates additional un-clustered instances to form more reliable clusters.The strategy dynamically refines the hybrid memory and learning targets.
  • Results: The method outperforms state-of-the-art methods on multiple domain adaptation tasks, with up to 5.0% mAP gains, and boosts source-domain performance by 6.6%.Its unsupervised version also significantly outperforms state-of-the-art methods on object re-ID tasks.

2 Related Works

Related work spans pseudo-label-based and domain-translation-based UDA, contrastive representation learning, and self-paced learning. The paper builds on these lines by combining target clustering with hybrid supervision and reliability-aware training.

  • UDA for object re-ID: Object re-ID UDA methods mainly use pseudo labels or domain translation, with this paper following pseudo-label-based adaptation.Pseudo labels can come from clustering instance features or measuring exemplar similarities.
  • UDA for object re-ID: Clustering-based UDA methods face the challenges of improving pseudo-label precision and mitigating noisy pseudo-label effects.Prior methods use local features or alternating regularization to address these challenges.
  • Contrastive learning: Contrastive learning trains instance-discriminative representations by treating each unlabeled sample as a distinct class.Existing approaches frame this as dictionary lookup or consistency learning.
  • Self-paced learning: Self-paced learning uses an easy-to-hard training scheme and has been extended to unsupervised learning by beginning with confident pseudo labels.The paper applies this principle to reliability-aware target clustering.

3 Methodology

The framework jointly learns from source-domain classes, target-domain clusters, and un-clustered target instances through a dynamically updated hybrid memory. Self-paced clustering preserves reliable clusters and gradually incorporates additional instances while alternating clustering, contrastive optimization, and memory updates.

  • Hybrid memory: The encoder is trained with source-class, target-cluster, and target-instance supervisory signals stored in a dynamically updated hybrid memory.The memory caches source class centroids and target instance features, while target cluster centroids are formed dynamically.
  • Optimization and updates: Training alternates between self-paced target clustering and unified contrastive optimization with momentum-based memory updates.Source centroids and target instance entries are updated from encoded mini-batch features; cluster centroids are then recomputed from updated instances.
  • Unified contrastive learning: The unified contrastive loss treats source classes, target clusters, and un-clustered target instances as independent classes.Positive prototypes are respectively source class centroids, target cluster centroids, or corresponding outlier features.
  • Unified contrastive learning: Class-level, cluster-level, and instance-level prototype semantics must match; otherwise, performance drops significantly.Source class centroids are used instead of learnable class weights to align their semantics with target cluster and outlier centroids.
  • Hybrid memory: The hybrid memory caches source class centroids and all target instance features, using target instances to dynamically represent cluster centroids or remain as un-clustered instances.Target cluster assignments and un-clustered membership can change during training, so the target memory is maintained at the instance level.
  • Self-paced learning: Before each epoch, only reliable target clusters are preserved, while unreliable clusters are disassembled into un-clustered instances using independence and compactness.This avoids merging instances into wrong clusters, which the paper states can do more harm than good.

4 Experiments

Experiments evaluate the framework across real→real and synthetic→real domain adaptation, source-domain retention, fully unsupervised re-ID, and component ablations. The method improves target-domain and source-domain performance, while ablations support using hybrid supervision and self-paced clustering.

  • Datasets and Evaluation Protocol: The evaluation covers real→real and synthetic→real adaptation tasks for person and vehicle re-ID, using mAP and CMC without post-processing.The datasets include real-world and synthetic domains, with only source-domain ground-truth IDs available for training.
  • Comparison with State-of-the-arts: 2–4% mAP improvements on common real→real tasks and up to 5.0% on synthetic→real tasks distinguish the method from state-of-the-art UDA approaches.The gains are reported across multiple person and vehicle adaptation tasks with a plain ResNet-50 backbone.
  • Comparison with State-of-the-arts: 6.6% maximum source-domain mAP improvement shows that the unified framework can retain or enhance source-domain performance after adaptation.The method also outperforms supervised re-ID methods on the source domain without extra annotations or more complex networks.
  • Comparison with State-of-the-arts: 16.7% maximum mAP improvement is achieved when the method is generalized to unsupervised re-ID without labeled source-domain training data.The ablation discussion attributes the advantage over instance discrimination to modeling intra- and inter-class variations rather than treating every instance as a separate class.
  • Ablation Studies: Removing hybrid supervision components produces failures or performance drops, while excluding un-clustered instances removes difficult samples and prevents their later memory updates.Instance-only training fails, cluster-only training discards many early-epoch samples, and the full framework uses source classes, target clusters, and target instances jointly.
  • Ablation Studies: 4.9% mAP drops occur when R_indep and R_comp are removed, and self-paced learning yields clusters closer in quantity and quality to ground-truth IDs.The reliability criterion uses hierarchical cluster stability through independence and compactness measures.

5 Discussion and Conclusion

The method improves unsupervised and domain-adaptive object re-ID across tasks, and can also promote supervised performance by incorporating unlabeled data. However, performance remains below the oracle because pseudo-class labels may still be unsatisfactory.

  • The method achieves considerable improvements across unsupervised and domain adaptive object re-ID tasks.
  • It can promote supervised performance without extra annotation by incorporating unlabeled data for training.
  • Performance remains below the oracle, suggesting that pseudo-class labels remain imperfect even with self-paced learning.
  • Further studies are needed to improve pseudo-class labels and explore applications beyond object re-ID.

Broader Impact

The framework is presented as applicable to object tracking across cameras and potentially to broader representation-learning tasks. Its algorithm jointly trains source classes, target clusters, and target outliers while progressively refining memory and learning targets.

  • Broader impact: The method can identify and track vehicles, cyclists, and pedestrians across cameras, with potential applications in smart retail, transportation, and security.
  • Broader impact: The self-paced contrastive learning strategy is proposed as extendable to unsupervised and semi-supervised representation learning.
  • Method comparison: The hybrid memory provides source class-level, target cluster-level, and un-clustered instance-level supervisory signals for unified training.
  • Method comparison: Self-paced learning gradually refines learning targets for clusters and un-clustered instances, unlike noisy k-nearest-neighbor targets applied uniformly.

C More Implementation Details

Training uses a ResNet-50-based encoder with domain-specific batch normalization and Adam optimization. Domain-adaptive training takes about three hours on Market-1501 and PersonX, while larger datasets require roughly twice as long.

  • Domain adaptation training takes approximately 3 hours on Market-1501 and PersonX using 4 GTX-1080TI GPUs.
  • Unsupervised learning training takes approximately 2 hours on Market-1501 and PersonX.
  • Training time doubles on MSMT17, VehicleID, VeRi-776, and VehicleX because their training sets contain over twice as many images.
  • The encoder uses an ImageNet-pretrained ResNet-50, domain-specific batch normalization, L2 normalization, and Adam with weight decay 0.0005.
  • The initial learning rate is 0.00035 and is reduced by a factor of 10 every 20 epochs during 50 total epochs.

C.2 Training Data Organization

Mini-batches combine labeled source images with target images assigned to pseudo classes, including both clusters and un-clustered instances. Clustering uses DBSCAN with Jaccard distance, while compactness thresholds preserve the most compact cluster points.

  • Each mini-batch contains 64 source images from 16 ground-truth classes and 64 target images from at least 16 pseudo classes.
  • Target clusters and un-clustered instances are treated as independent pseudo classes, with four images per cluster or one image per un-clustered instance.
  • Images are resized to 256 × 128 for person re-ID and 224 × 224 for vehicle re-ID.
  • Clustering before each epoch uses DBSCAN with Jaccard distance and k-reciprocal nearest neighbors, with k = 30, maximum neighbor distance d = 0.6, and minimum dense-point neighbors 4.
  • The dynamic threshold β identifies compact clusters by preserving the most compact points according to each cluster’s maximum Rcomp.

D Additional Experimental Results

Additional experiments test alternative backbones, clustering algorithms, and cluster-reliability criteria for the framework. The self-paced strategy improves performance with Agglomerative Clustering, while the proposed reliability criterion compares favorably with HDBSCAN.

  • Backbone comparison: IBN-ResNet further improves performance except on vehicle datasets.The backbone comparison evaluates ResNet-50 and IBN-ResNet within the framework.
  • Alternative clustering algorithm: Table 7 evaluates the framework with Agglomerative Clustering on unsupervised person re-ID tasks.This experiment provides the setting for the reported self-paced-learning improvement.
  • Alternative clustering algorithm: 4.8% mAP improvements are observed when applying self-paced learning with Agglomerative Clustering.The final performance is also better than with DBSCAN.
  • Cluster-reliability comparison: Table 8 compares DBSCAN with the proposed cluster-reliability criterion against HDBSCAN on unsupervised person re-ID tasks.The comparison corresponds to the reported mAP differences between the two criteria.
  • Cluster-reliability comparison: 1.4%/3.4% mAP drops occur on unsupervised Market-1501/MSMT17 when replacing the proposed criterion with HDBSCAN.The comparison supports using DBSCAN with the proposed cluster-reliability criterion in this framework.

E Parameter Analysis

The hyper-parameters are tuned on MSMT17→Market-1501 and then applied unchanged to all other tasks.

  • Hyper-parameters are tuned on the MSMT17→Market-1501 task.
  • The selected hyper-parameters are directly applied to every other task.
  • The parameter-analysis results therefore use a single tuning task as their selection basis.

E.1 Temperature τ for Contrastive Loss

The framework’s contrastive-loss temperature is analyzed through Figure 4. A value of τ = 0.05 gives optimal performance and remains effective across the reported adaptation settings.

  • τ = 0.05 gives the framework’s optimal performance on MSMT17→Market-1501.The value is used in Eq. (1).
  • The framework’s performance varies with different temperature values.Temperature affects methods using a temperature contrastive function similarly.
  • τ = 0.05 achieves the best performance across 6 UDA tasks and 3 unsupervised tasks.The same fixed temperature is used in all these tasks.

E.2 Momentum Coefficients ms, mt for Hybrid Memory

The analysis examines momentum coefficients for source-class and target-instance updates, as well as the reliability-criterion hyper-parameter. Target momentum is more influential, but broad tested ranges remain robust.

  • m_s = m_t = 0.2 is adopted after tuning on MSMT17→Market-1501.The hybrid memory updates source-domain class centroids and target-domain instance features with these momenta.
  • m_t is critical to optimal performance, whereas the framework is not sensitive to m_s.The experiments therefore set m_s = m_t for convenience.
  • m_t changes within [0.2, 0.6] while the framework remains robust.This result is shown when m_s = m_t.
  • ∆d = 0.00 corresponds to removing the self-paced strategy from training.This setting matches “Ours w/o Rcomp&Rindep” in Table 5.
  • Performance remains similar for ∆d within [0.02, 0.05].The proposed reliability criterion is not sensitive to this hyper-parameter in the tested range.
Loading 2006.02713v2…