PaperlayerView exact revision on arXiv

Source-linked AI summary

CLIP-Driven Fine-grained Text-Image Person Re-identification

Shuanglin Yan, Neng Dong, Liyan Zhang, Jinhui Tang

arXiv:2210.10276v1cs.CV

TL;DR

TIReID must match text descriptions to pedestrian images despite limited multimodal correspondence and possible information distortion from feature embedding. CFine transfers CLIP knowledge through fine-grained information excavation, and experiments report superior performance across multiple benchmarks.

  • Problem

    Existing TIReID methods lack multimodal correspondence, while joint-space feature embedding may distort intra-modal information.

  • Method

    CFine uses multi-grained global feature learning, cross-grained feature refinement, and fine-grained correspondence discovery to mine intra-modal clues and inter-modal correspondences.

  • Results

    CFine achieves superior TIReID performance across multiple benchmarks, with fine-grained excavation improving retrieval when baseline CLIP retrieval fails.

  • Takeaways & Limitations

    Fine-grained information excavation transfers CLIP’s multimodal knowledge to support text-image alignment without further feature embedding.

Abstract

from arXiv · show

TIReID aims to retrieve the image corresponding to the given text query from a pool of candidate images. Existing methods employ prior knowledge from single-modality pre-training to facilitate learning, but lack multi-modal correspondences. Besides, due to the substantial gap between modalities, existing methods embed the original modal features into the same latent space for cross-modal alignment. However, feature embedding may lead to intra-modal information distortion. Recently, CLIP has attracted extensive attention from researchers due to its powerful semantic concept learning capacity and rich multi-modal knowledge, which can help us solve the above problems. Accordingly, in the paper, we propose a CLIP-driven Fine-grained information excavation framework (CFine) to fully utilize the powerful knowledge of CLIP for TIReID. To transfer the multi-modal knowledge effectively, we perform fine-grained information excavation to mine intra-modal discriminative clues and inter-modal correspondences. Specifically, we first design a multi-grained global feature learning module to fully mine intra-modal discriminative local information, which can emphasize identity-related discriminative clues by enhancing the interactions between global image (text) and informative local patches (words). Secondly, cross-grained feature refinement (CFR) and fine-grained correspondence discovery (FCD) modules are proposed to establish the cross-grained and fine-grained interactions between modalities, which can filter out non-modality-shared image patches/words and mine cross-modal correspondences from coarse to fine. CFR and FCD are removed during inference to save computational costs. Note that the above process is performed in the original modality space without further feature embedding. Extensive experiments on multiple benchmarks demonstrate the superior performance of our method on TIReID.

I. INTRODUCTION

TIReID retrieves pedestrian images from text descriptions, but existing approaches lack multimodal correspondence and may distort intra-modal information through feature embedding. CFine uses CLIP knowledge and fine-grained information excavation to learn discriminative clues and cross-modal correspondences without further embedding.

  • TIReID retrieves a target pedestrian image from a witness’s language description when images are unavailable in complex or occluded scenes.
  • Existing methods commonly use single-modality pre-training and joint-space feature embedding, which may lack multimodal correspondence and distort intra-modal information.
  • CLIP provides semantic visual representations and multimodal knowledge that can support alignment in the original modality space.
  • CFine uses multi-grained global feature learning, cross-grained feature refinement, and fine-grained correspondence discovery to mine intra-modal clues and inter-modal correspondences.
  • CFine achieves 69.57 Rank-1 on CUHK-PEDES, 60.83 Rank-1 on ICFG-PEDES, and 50.55 Rank-1 on RSTPReid, exceeding previous state of the art by +5.13%, +4.79%, and +3.85%, respectively.

A. Text-Image Person Re-identification

TIReID is a challenging fine-grained multimodal retrieval task requiring cross-modal alignment. Existing interaction-based methods improve alignment but incur high computational cost, while interaction-free methods are efficient yet may rely on distorted joint embeddings and insufficient multimodal knowledge.

  • TIReID is challenging because it requires fine-grained cross-modal alignment between pedestrian images and text descriptions.
  • Cross-modal interaction-based methods mine local correspondences and can reduce the modality gap, but their high computational cost limits practicability.
  • Cross-modal interaction-free methods learn aligned image and text embeddings in a joint latent space and are efficient, but early approaches show unsatisfactory performance.
  • Single-modality initialization ignores multimodal correspondence information and can over-mine modality-specific information, increasing cross-modal alignment difficulty.
  • CFine extracts global and local features, mines multi-grained discriminative information, refines tokens, and discovers patch-word correspondences for cross-modal alignment.

B. Dual Encoders

The dual-encoder pipeline extracts global and local image/text representations, then selects informative tokens and uses a global-local decoder to produce multi-grained global features. This excavation addresses missing fine-grained information in the original encoder outputs.

  • CLIP uses separate image and text encoders whose projectors align both modalities in a shared 512-dimensional latent space.
  • The image encoder represents an image with a global [CLS] feature and patch-level local features from a ViT token sequence.
  • The text encoder uses BERT to produce a sentence-level global feature and word-level local features.
  • Token Selection: Token selection ranks local tokens by their correlation with the class token and retains the top 2K tokens for each modality.Images select 2Kv tokens and texts select 2Kt tokens from their local token sequences.
  • Global-Local Decoder: The global-local decoder splits selected tokens into high- and middle-level sequences, uses attention to propagate and contextualize information, and generates multi-grained global features.The resulting image feature set contains low-, middle-, and high-level global features, with an analogous text feature set.

D. Cross-Grained Feature Refinement

CFR refines selected image patches and text words by measuring cross-grained similarities between image-word and sentence-patch pairs. Softmax-weighted aggregation emphasizes modality-shared information and establishes rough cross-modal correspondence.

  • CFR filters selected tokens using cross-modal similarities because token selection alone identifies informative but not necessarily modality-shared information.
  • Image-word and sentence-patch similarities are computed as inner products between corresponding global and local representations.The image-word scores compare the image representation with selected words, while sentence-patch scores compare the sentence representation with selected patches.
  • Softmax assigns higher weights to image patches or words related to the opposing modality during similarity aggregation.
  • The weighted scores are summed into an instance-level similarity, through which CFR roughly establishes image-text correspondence.

E. Fine-Grained Correspondence Discovery

FCD discovers fine-grained patch-word correspondences instead of relying on one-to-one nearest matches. It selects multiple related patches or words, pools them, and aggregates pairwise similarities into an instance-level score.

  • FCD targets fine-grained patch-word correspondence after CFR establishes rough cross-modal correspondence.
  • Because individual words and patches can be ambiguous or related to multiple counterparts, FCD avoids selecting only one positive counterpart.
  • For each word, FCD computes cosine similarities to informative patches and applies Topk to select the Kp most related patches.
  • The selected patches are average pooled to form a new patch feature, which pairs with the word to create a matched word-patch pair.
  • FCD constructs Kv patch-word pairs and Kt word-patch pairs, then sums their similarities to obtain the instance-level similarity.

F. Training and Inference

CFine trains multi-grained image and text representations with cross-modal alignment, ranking, and diversity constraints. During inference, it removes CFR and FCD and sums similarities across granularities.

  • Training: CMPM and CMPC losses supervise image-text alignment at each multi-grained feature level.Image and text feature sets contain low-, medium-, and high-granularity representations.
  • Training: A bidirectional dual-constrained triplet ranking loss maximizes matched image-text similarity and minimizes mismatched-pair similarity.Hard negative samples are used, with α specifying the margin.
  • Training: Diversity regularization encourages different-grained features to focus on inconsistent information and avoid redundancy.
  • Training: The total objective combines cross-modal matching loss with weighted contrastive and diversity terms.λc and λd balance the different loss contributions during training.
  • Inference: During inference, CFR and FCD are removed, and image-text similarity is computed as sl + sm + sh.The inference score sums cosine distances from different-grained features.

1) Datasets and Metrics:

The evaluation uses the CUHK-PEDES benchmark and reports Rank-1, Rank-5, and Rank-10 retrieval accuracy. Experiments use standardized image and text preprocessing and a fixed training schedule.

  • Datasets and Metrics: CUHK-PEDES contains 40,206 images, 80,412 descriptions, and 13,003 persons, with two descriptions per image.The descriptions average at least 23 words.
  • Datasets and Metrics: CUHK-PEDES is split into 34,054 training images, 3,078 validation images, and 3,074 test images.The corresponding person counts are 11,003, 1,000, and 1,000.
  • Implementation Details: Input images are resized to 224×224, sentences are capped at 100 tokens, and visual and textual sequences contain 196 and 100 tokens.Image and text embedding dimension d is 768.
  • Implementation Details: The network uses batch size 32 for 50 epochs, with the learning rate reduced at epochs 20, 25, and 35.
  • Datasets and Metrics: The reported retrieval metrics are Rank-1, Rank-5, and Rank-10 accuracy.Table I lists these metrics for CUHK-PEDES comparisons.

B. Comparisons with State-of-the-art Models

CFine achieves state-of-the-art performance across three TIReID benchmarks, with CLIP-based encoding providing the strongest results. Ablations attribute gains to multi-grained feature learning and cross-modal refinement and correspondence modules.

  • Comparisons with State-of-the-art Models: CFine consistently achieves state-of-the-art results on CUHK-PEDES, ICFG-PEDES, and RSTPReid under the evaluated settings.The comparisons use either ImageNet-pretrained ViT or CLIP ViT as image encoder.
  • CUHK-PEDES: 65.07%, 83.01%, and 89.00% Rank1, Rank-5, and Rank-10 accuracy are obtained on CUHK-PEDES with ImageNet-pretrained ViT.These results outperform the compared Transformer-based methods and AXM-Net.
  • CUHK-PEDES: 69.57% Rank-1, 85.93% Rank-5, and 91.15% Rank-10 are achieved on CUHK-PEDES with CLIP ViT, exceeding AXM-Net by +5.13%, +5.41%, and +4.38%.
  • Other Benchmarks: 60.83% Rank-1 on ICFG-PEDES and 50.55% Rank-1 on RSTPReid surpass IVT by +4.79% and +3.85%, respectively, using CLIP ViT.The method outperforms all existing methods by a large margin across metrics in this setting.
  • Contributions of Algorithmic Components: Combining MGF medium- and high-granularity features raises Rank-1 from 65.56% to 68.62% in the ablation study.Adding MGF(m) or MGF(h) individually improves Rank-1 by 2.1% or 2.66%.
  • Contributions of Algorithmic Components: MGF, CFR, and FCD together provide the best retrieval performance in the component ablations on CUHK-PEDES.CFR filters redundant information, while FCD discovers inter-modal fine-grained correspondences.

2) Ablation of Multi-grained Global Feature Learning:

The ablation examines token-selection ratios, GLD depth, and multi-grained learning choices, showing that staged token processing and encoder-initialized global tokens preserve discriminative information. It also reports the role of similarity aggregation in CFR.

  • Multi-grained global feature learning: A one-time 2K-token learning variant caused a 2.14% performance drop, supporting multi-grained learning to avoid overwriting informative tokens.MGF-2K feeds 2Kv/2Kt tokens into GLD simultaneously, whereas multi-grained learning processes informative tokens in stages.
  • Multi-grained global feature learning: Cross-attention between selected and all tokens outperformed self-attention between selected tokens because it also absorbs contextual information beyond informative tokens.The reported explanation is that broader context helps the model retain complementary information beyond selected tokens.
  • Multi-grained global feature learning: Encoder-output initialization of the high-level and middle-level [CLS] tokens performed best among mean, random, and encoder-based initialization schemes.The paper attributes this to stronger cooperation between GLD and the encoder, emphasizing fine-grained clues from encoder outputs.
  • Cross-grained feature refinement: CFR aggregates image-word and sentence-patch similarity scores into instance-level scores for subsequent text-image matching.The compared strategies include summing or averaging the 2Kt or 2Kv similarity scores.
  • Parameter analysis: Figure 5 evaluates the image-token ratio Rv, text-token ratio Rt, GLD block count M, and selected correspondence count Kp.These parameters govern token selection, the amount of global-feature learning, and the number of positive patches or words selected for each counterpart.

4) Ablation of Fine-grained Correspondence Discovery:

The FCD and qualitative analyses examine how selecting related words or patches affects correspondence discovery and retrieval. CFine’s fine-grained excavation is illustrated as improving difficult retrievals and reducing reliance on color cues.

  • Fine-grained correspondence discovery: FCD selects the most related words or patches for each counterpart to discover fine-grained correspondence.The amount Kp is tuned because too few selections can make meaning ambiguous, while too many can introduce irrelevant information.
  • Fine-grained correspondence discovery: When Kp is too small, local correspondence can dominate and cause over-fitting; when Kp is too large, irrelevant information can obstruct accurate correspondence.The passage frames Kp as a trade-off between ambiguous semantics and noisy correspondence evidence.
  • Qualitative results: CFine retrieved accurately in some top-10 cases where Baseline+CLIP failed, according to the CUHK-PEDES qualitative comparison.The comparison distinguishes the methods by whether fine-grained information excavation is performed.
  • Qualitative results: Mining clues such as bags, guitars, and shoes reduced the model’s dependence on color information in the qualitative analysis.The authors describe this as alleviating the color over-reliance problem to a certain extent.
  • Framework contribution: CFine combines MGF, CFR, and FCD to mine intra-modal discriminative clues and cross-modal correspondences without further feature embedding.CFR filters non-modality-shared information, while FCD discovers word-patch relationships from the retained information.
Loading 2210.10276v1…