PaperlayerView exact revision on arXiv

Source-linked AI summary

Noisy-Correspondence Learning for Text-to-Image Person Re-identification

Yang Qin, Yingke Chen, Dezhong Peng, Xi Peng, Joey Tianyi Zhou, Peng Hu

arXiv:2308.09911v3cs.CVcs.MM

TL;DR

TIReID methods assume training image-text pairs are aligned, but real-world noisy correspondences can violate that assumption. RDE combines dual-grained consensus selection with a triplet loss over negative pairs. The authors report best performance across three datasets both with and without synthetic noisy correspondences.

  • Problem

    TIReID methods assume correctly aligned training pairs, although annotation noise can make unmatched image-text pairs appear positive and degrade performance.

    Source: §1 · p. 2, §3.1 · p. 3

  • Method

    RDE combines dual-grained embeddings and consensus division of clean training data with Triplet Alignment Loss over negative pairs.

    Source: §1 · p. 2

  • Results

    RDE achieves the best performance on three public benchmarks both with and without synthetic noisy correspondence.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The experiments support RDE's effectiveness and robustness on three datasets under both ordinary and synthetically noisy settings.

    Source: §5 · p. 8

    The loss analysis assumes one paired text for each image in a mini-batch.

    Source: §3.3.3 · p. 6

Abstract

from arXiv · show

Text-to-image person re-identification (TIReID) is a compelling topic in the cross-modal community, which aims to retrieve the target person based on a textual query. Although numerous TIReID methods have been proposed and achieved promising performance, they implicitly assume the training image-text pairs are correctly aligned, which is not always the case in real-world scenarios. In practice, the image-text pairs inevitably exist under-correlated or even false-correlated, a.k.a noisy correspondence (NC), due to the low quality of the images and annotation errors. To address this problem, we propose a novel Robust Dual Embedding method (RDE) that can learn robust visual-semantic associations even with NC. Specifically, RDE consists of two main components: 1) A Confident Consensus Division (CCD) module that leverages the dual-grained decisions of dual embedding modules to obtain a consensus set of clean training data, which enables the model to learn correct and reliable visual-semantic associations. 2) A Triplet Alignment Loss (TAL) relaxes the conventional Triplet Ranking loss with the hardest negative samples to a log-exponential upper bound over all negative ones, thus preventing the model collapse under NC and can also focus on hard-negative samples for promising performance. We conduct extensive experiments on three public benchmarks, namely CUHK-PEDES, ICFG-PEDES, and RSTPReID, to evaluate the performance and robustness of our RDE. Our method achieves state-of-the-art results both with and without synthetic noisy correspondences on all three datasets. Code is available at https://github.com/QinYang79/RDE.

1. Introduction

TIReID methods generally assume correctly aligned training pairs, although pose, camera angle, illumination, and annotation errors can produce noisy correspondences that mislead learning. RDE addresses this problem with dual-grained consensus selection and a loss over negative pairs, and is reported to perform best across three benchmarks with and without synthetic noise.

1. Introduction

  • Existing TIReID methods assume aligned training pairs, but image conditions and annotation errors can create false correspondences that misguide learning.The paper defines noisy correspondence as negative image-text pairs used as positives, which can lead to overfitting and suboptimal solutions.Source: §1 · p. 2
  • RDE uses Confident Consensus Division to divide training data into clean and noisy sets through consensus across dual-grained decisions.The method also combines global and token-selection embeddings for coarse- and fine-grained interactions.Source: §1 · p. 2
  • Triplet Alignment Loss uses an upper bound over all negative pairs rather than relying only on the hardest negatives in Triplet Ranking loss.The paper says this supports stable learning under noisy correspondence while retaining hard-negative benefits.Source: §1 · p. 2
  • The paper presents noisy correspondence as a TIReID problem distinct from class-level noisy labels and describes it as a first exploration in the task.It characterizes the issue as erroneous correspondence in person-description pairs that can teach incorrect visual-semantic associations.Source: §1 · p. 2
  • RDE achieves the best performance on three public benchmarks both with and without synthetic noisy correspondence.The reported experiments describe robustness and superiority across all three datasets.Source: §1 · p. 2

2. Related Work

Prior TIReID work develops global and local alignment approaches, while noisy-correspondence research has explored sample selection and robust losses in other multimodal tasks. The paper situates its work at their intersection: a TIReID method designed for noisy pairings.

2.1. Text-to-Image Person Re-identification

  • Global-matching methods learn shared cross-modal embeddings, while local-matching methods align body regions with textual entities at greater computational cost.The paper also notes that existing methods generally assume correctly aligned training pairs, an assumption difficult to meet in practice.Source: §2.1 · p. 2

2.2. Learning with Noisy Correspondence

  • RDE combines dual-grained embeddings with Confident Consensus Division and Triplet Alignment Loss to learn more robustly from noisy pairs.The overview describes CCD as selecting confident clean data and TAL as an upper bound over all negative pairs.Source: Figure 2 · p. 3
  • Noisy-correspondence methods elsewhere use sample selection or noise-tolerant losses, but are not specifically designed for TIReID.The related work cites NC research in video-text retrieval, visible-infrared person re-identification, and image-text matching.Source: §2.2 · pp. 2–3

3. Methodology

RDE addresses noisy image–text correspondences in TIReID with dual-grained embeddings, consensus-based data division, and a loss that balances hard-negative emphasis with stability. Its training recalibrates correspondence labels before optimizing the matching loss.

3.1. Problem Statement

  • TIReID retrieves a gallery image matching a text description; noisy correspondence occurs when an unmatched image–text pair is labeled as matched.The method uses RDE with CCD and TAL to mitigate the negative impact of this label noise.Source: §3.1 · p. 3

3.2. Cross-modal Embedding Model

  • RDE uses CLIP’s visual and textual encoders to obtain token representations and implement cross-modal interactions through two embedding modules.The text encoder produces global and word-level local representations; the image encoder produces a global feature and patch-level local features.Source: §3.2 · p. 3, §3.2.1 · pp. 3–4

3.2.2 Dual Embedding Modules

  • BGE and TSE provide coarse- and fine-grained cross-modal interactions, respectively, using global features and selected informative local tokens.TSE transforms selected token features through an embedding module rather than using complex fine-grained correspondence discovery.Source: §3.2.2 · p. 4

3.3.1 Confident Consensus Division

  • For each pair, CCD fits per-sample loss distributions with a two-component GMM and uses posterior probabilities to identify clean or noisy assignments.The method uses expectation-maximization and a 0.5 threshold for the clean-component posterior.Source: §3.3.1 · pp. 4–5
  • CCD uses loss distributions from BGE and TSE to divide training pairs into clean and noisy sets, then treats their shared clean division as the confident clean set.A two-component GMM fits per-sample losses; the lower-mean component is assigned as clean.Source: §3.3.1 · pp. 4–5

3.3.2 Triplet Alignment Loss

  • TAL relaxes hardest-negative Triplet Ranking Loss optimization to an upper bound over all negatives, reducing domination by hardest negatives while retaining attention to hard examples.Under noisy correspondence, TRL may cause collapse, while TRL-S is more stable but can underperform by treating negatives equally.Source: §3.3.2 · p. 5

3.3.3 Revisit Triplet Raking Loss

  • With NC, TRL can collapse similarities into a narrow neighborhood, whereas TRL-S avoids collapse but neglects challenging negatives; TAL adaptively weighs negatives and avoids collapse while achieving superior performance.The analysis considers the image-to-text direction and assumes one paired text per image in the mini-batch.Source: §3.3.3 · pp. 5–6

3.3.4 Training and Inference

  • RDE trains with corrected correspondence labels and combines the BGE and TSE similarities by averaging them for joint inference.The final matching loss uses the corrected label with TAL losses computed from both embedding modules.Source: §3.3.4 · p. 6

4. Experiments

Experiments evaluate RDE on three TIReID benchmarks, including synthetic noisy-correspondence settings and comparisons on the original datasets. The reported results cover benchmark comparisons, component ablations, hyperparameter sensitivity, and training-epoch behavior under noise.

4.1.1 Datasets

  • The experiments use CUHK-PEDES, ICFG-PEDES, and RSTPReid; ICFG-PEDES has training and validation sets only, and Rank-K is the primary retrieval metric.mAP and mINP are auxiliary metrics; the comparisons include best-validation and last-epoch checkpoints under 20% and 50% synthetic noise.Source: §4.1.1 · p. 6, §4.1.2 · p. 6, §4.2 · p. 7

4.2. Comparison with State-of-the-Art Methods

  • 71.33% Rank-1, 87.41% Rank-5, and 91.81% Rank-10 on CUHK-PEDES under 50% noise; RDE exceeds DECL by +6.11%, +3.69%, and +2.53%, respectively.RDE also achieves the best results across metrics on datasets with synthetic noisy correspondences and outperforms baselines on the original datasets.Source: Table 1 · p. 7, §4.2 · pp. 7–8

4.1.3 Implementation Details

  • The training configuration uses CLIP-ViTB/16, 60 epochs, and batch size 64, with a cosine learning-rate schedule and image and text augmentations.The initial learning rates are 1e −5 for CLIP parameters and 1e −3 for TSE network parameters.Source: §4.1.3 · p. 7

4.2. Comparison with State-of-the-Art Methods

  • RDE is compared with ordinary and robust methods, including SSAN, IVT, IRRA, DECL, and CLIP-C, on the benchmark datasets.The study also compares RDE with local- and global-matching methods on the original datasets.Source: §4.2 · p. 7

4.3. Ablation Study

  • RDE performs best with BGE and TSE jointly, while CCD alleviates noise-related overfitting and TAL outperforms TRL and SDM under 50% noise.The ablation study evaluates component combinations on CUHK-PEDES.Source: §4.3 · p. 8, Table 2 · p. 8

4.4. Parametric Analysis

  • Too large or too small m yields suboptimal performance, while too small τ causes training failure; the chosen settings are m = 0.1 and τ = 0.015.Increasing τ gradually reduces positive-negative separability, also leading to suboptimal performance.Source: §4.4 · p. 8, Figure 4 · p. 8

4.5. Robustness Study

  • Figure 5 plots Rank-1 test performance against epochs on CHUK-PEDES and ICFG-PEDES under 50% noise.The figure is presented as a visualization of training robustness and effectiveness.Source: Figure 5 · p. 8, §4.5 · p. 8

5. Conclusion

The conclusion identifies noisy correspondence as a challenge to the usual assumption of perfectly aligned image-text data in TIReID. It presents RDE as a response and reports experiments demonstrating robustness and superiority across three datasets, with and without synthetic noise.

5. Conclusion

  • Noisy correspondence violates the common assumption that TIReID training image-text data are perfectly aligned.The conclusion frames NC as a novel challenging problem for TIReID.Source: §5 · p. 8
  • RDE is proposed to handle noisy correspondence and achieve superior performance.Source: §5 · p. 8
  • Experiments on three datasets demonstrate RDE's superiority and robustness both with and without synthetic noisy correspondences.Source: §5 · p. 8

A.1. Proof for Lemma 1(Lemma 2)

The appendix establishes an upper-bound relation between TAL and TRL and derives gradients for the losses. Its gradient derivation simplifies the setting to one retrieval direction and one paired text per image in the mini-batch.

A.1. Proof for Lemma 1(Lemma 2)

  • As τ → 0, TAL approaches TRL and focuses more on hard negatives; S+ is the weighted average similarity of positive pairs for an image.Source: §A.1 · p. 9
  • TAL is shown to upper-bound TRL, expressed as Ltrl(Ii, Ti) ≤ Ltal(Ii, Ti).The proof combines the image-to-text and text-to-image directions.Source: §A.1 · p. 9

A.2. Derivation for Gradient

  • The gradient derivation considers one retrieval direction and one paired text per image, simplifying the losses for analysis.The appendix treats image-to-text and text-to-image retrieval as symmetrical.Source: §A.2 · p. 10

B.1. Datasets.

The appendix describes the three benchmark datasets and their data splits, then lists baseline approaches for noisy-correspondence evaluation. It also identifies Table 3 as a comparison under an 80% noise rate.

B.1. Datasets.

  • CUHK-PEDES has 40,206 images and 80,412 descriptions for 13,003 identities; ICFG-PEDES has 54,522 images for 4,102 people, with validation performance used as test performance because it has no test set.RSTPReid contains 20,505 images for 4,101 identities, with five images and two descriptions per person and official train/validation/test identity splits.Source: §B.1 · p. 10

B.2. Baselines.

  • Table 3 compares performance across three benchmarks under 80% noise, reporting best-validation and last-epoch checkpoints and Rank-1, Rank-5, and Rank-10 accuracy.Source: Table 3 · p. 11
  • The baseline set includes local-matching SSAN, global-matching IVT and IRRA, clean-pair CLIP-C, and robust framework DECL.DECL uses IRRA as its base model for robust TIReID comparisons.Source: §B.2 · pp. 10–11

C. The Results under Extreme Noise

RDE outperforms the best baselines under 80% synthetic noise and also leads in additional comparisons across three benchmarks. The reported Rank-1 gains are positive in both settings.

C. The Results under Extreme Noise

  • At 80% noise, RDE achieves the best performance and alleviates noise-related degradation across the three datasets.The authors report Rank-1 gains of +7.56%, +5.95%, and +3.5% over the ‘Best’ rows, respectively.Source: C. The Results under Extreme Noise · p. 11

D. More Comparisons

  • RDE exceeds the best baselines by +0.92%, +2.63%, and +0.15% in Rank-1 across the three benchmarks, respectively.Source: D. More Comparisons · p. 11

E. Study on the Selection Ratio

The selection-ratio study finds suboptimal performance at either extreme and uses R = 0.3 in the experiments. The TSE ablation favors embedding transformation with max pooling.

E. Study on the Selection Ratio

F.1. Ablation analysis for TSE

  • The standard TSE version performs best with embedding transformation and max pooling.The ablation compares direct aggregation of CLIP token features and different pooling strategies.Source: §F.1 · p. 13

F.2. Ablation study on High Noise

The high-noise ablation reports conclusions consistent with the main text, while the case study illustrates real noisy correspondences and compares retrieval examples from IRRA and RDE.

F.2. Ablation study on High Noise

  • The additional ablation observations are reported as consistent with the main text and as demonstrating the effectiveness of RDE.The appendix describes ablations on CUHK-PEDES; Figures 10–12 show Rank-1 test performance versus epochs at 20%, 50%, and 80% noise.Source: §F.2 · pp. 13–14, Figure 10 · p. 14, Figure 11 · p. 14

G. Case Study

  • The case study identifies occlusion, lighting, and inaccurate text descriptions as sources of real noisy correspondences in the datasets.The authors connect these examples to the failure of the assumption that every training image-text pair is correctly aligned.Source: G. Case Study · p. 14
  • Figure 13 compares the top-10 retrievals for each query, with IRRA in the first row and RDE in the second.Matched and mismatched person images are marked with red and blue rectangles, respectively.Source: Figure 13 · p. 15

H. Robustness Study

As synthetic noise rises, non-robust methods show overfitting patterns and severe degradation at 80%, whereas RDE remains robust across the reported curves.

  • At 20% noise, the baselines show no obvious performance degradation from overfitting noisy pairs, but at higher noise SSAN, IVT, and IRRA curves first rise and then fall.The authors attribute this tendency to the DNN memorization effect: simple patterns are learned before noisy samples.Source: H. Robustness Study · p. 16
  • At 80% noise, SSAN fails, while IVT and IRRA suffer serious performance drops.Source: H. Robustness Study · p. 16
  • RDE's CCD and TAL are reported to support learning accurate visual-semantic associations from confident clean pairs, preventing overfitting noisy pairs.The authors report strong robustness alongside re-identification performance across the figures.Source: H. Robustness Study · p. 16

I. Qualitative Results

On CUHK-PEDES with 20% noisy correspondences, retrieved examples show RDE producing more accurate and reasonable re-identification results. The examples contrast this with IRRA results where retrieved images sometimes match only part of the query, while RDE filters erroneous correspondences.

I. Qualitative Results

  • RDE produces more accurate and reasonable re-identification results in examples from CUHK-PEDES with 20% noisy correspondences.Source: §I · p. 16
  • Some inaccurate IRRA results retrieve images whose visual information matches only part of the text query.Source: §I · p. 16
  • The authors attribute RDE's more robust results to filtering erroneous correspondences and learning reliable cross-modal knowledge.Source: §I · p. 16

Related papers

Loading 2308.09911v3…