PaperlayerView exact revision on arXiv

Source-linked AI summary

CLIP-ReID: Exploiting Vision-Language Model for Image Re-Identification without Concrete Text Labels

Siyuan Li, Li Sun, Qingli Li

arXiv:2211.13977v4cs.CV

TL;DR

Fine-grained ReID lacks concrete text labels, leaving it unclear how vision-language models such as CLIP can be used. The paper introduces CLIP-ReID, which learns ID-specific ambiguous text descriptions before using them to constrain image-encoder fine-tuning, and validates the strategy across person and vehicle ReID datasets.

  • Problem

    ReID labels are indexes rather than concrete descriptions, so CLIP’s text-based downstream mechanism cannot be directly applied.

  • Method

    CLIP-ReID learns ID-specific text tokens with both encoders fixed, then freezes the resulting descriptions while fine-tuning the image encoder with cross-modal constraints.

  • Results

    CLIP-ReID achieves state-of-the-art performance across multiple person and vehicle ReID datasets with both ResNet-50 and ViT-B/16 models initialized from CLIP.

  • Takeaways & Limitations

    The results support using CLIP’s cross-modal description ability to improve visual representation for ReID despite non-textual identity labels.

Abstract

from arXiv · show

Pre-trained vision-language models like CLIP have recently shown superior performances on various downstream tasks, including image classification and segmentation. However, in fine-grained image re-identification (ReID), the labels are indexes, lacking concrete text descriptions. Therefore, it remains to be determined how such models could be applied to these tasks. This paper first finds out that simply fine-tuning the visual model initialized by the image encoder in CLIP, has already obtained competitive performances in various ReID tasks. Then we propose a two-stage strategy to facilitate a better visual representation. The key idea is to fully exploit the cross-modal description ability in CLIP through a set of learnable text tokens for each ID and give them to the text encoder to form ambiguous descriptions. In the first training stage, image and text encoders from CLIP keep fixed, and only the text tokens are optimized from scratch by the contrastive loss computed within a batch. In the second stage, the ID-specific text tokens and their encoder become static, providing constraints for fine-tuning the image encoder. With the help of the designed loss in the downstream task, the image encoder is able to represent data as vectors in the feature embedding accurately. The effectiveness of the proposed strategy is validated on several datasets for the person or vehicle ReID tasks. Code is available at https://github.com/Syliz517/CLIP-ReID.

Introduction

ReID matches people or vehicles across non-overlapping camera views, but applying CLIP is difficult because ReID labels are indexes rather than concrete text descriptions. CLIP-ReID addresses this gap with ID-specific learnable tokens and two-stage training that uses text features to constrain visual representation.

  • Introduction: ReID matches the same person or vehicle across different, non-overlapping camera views despite background, illumination, pose, and occlusion changes.Images are mapped into an embedding space where same-object features tend to be close and different-object features far apart.
  • Introduction: CNN-based ReID models can overfocus on irrelevant regions, while ViT models require large datasets and may optimize erratically on relatively small ReID datasets.These limitations motivate using transferable representations from vision-language pre-training.
  • Introduction: CLIP connects visual representations with high-level language descriptions, enabling transferable features and zero-shot classification by comparing image and text embeddings.CoOp extends this setup with learnable prompts while keeping the pre-trained parameters fixed.
  • Introduction: Because ReID labels are indexes without specific descriptive words, CLIP and CoOp cannot directly form the text descriptions required for downstream matching.The paper therefore introduces learnable text tokens for each identity and a two-stage strategy.
  • Introduction: CLIP-ReID fixes both encoders while optimizing ID-specific tokens first, then keeps the descriptions static while fine-tuning the image encoder with cross-modal constraints.The strategy is designed to exploit CLIP’s text encoder for ReID.
  • Introduction: The method reports state-of-the-art performance across multiple person and vehicle ReID datasets.The paper presents CLIP-ReID as an early use of CLIP for ReID and also provides competitive CLIP-initialized visual baselines.

Related Works

Prior ReID work develops discriminative descriptors, metric-learning objectives, compact architectures, regional features, and attention mechanisms, while recent vision-language methods extend CLIP to other downstream tasks. CLIP-ReID applies CLIP to ReID through a two-stage token-and-image training process and reports state-of-the-art results with both CNN and ViT backbones.

  • Related Works: Earlier ReID methods learn discriminative descriptors or formulate matching as metric learning over inter- and intra-class distances.Examples include foreground histograms, local maximal occurrences, bag-of-visual words, hierarchical Gaussian descriptors, and learned distance measurements.
  • Related Works: Other approaches address CNN overfitting and limited data through lightweight architectures, neural architecture search, data selection, regional feature division, and attention.These methods seek compact models, more generalizable training data, local representations, or enlarged receptive fields.
  • Related Works: CLIP-ReID uses both CNN and ViT models initialized from CLIP, and both achieve state-of-the-art results on different datasets with two-stage training.This extends the method across the two backbone families evaluated in the paper.
  • Related Works: Vision-language pre-training matches image and language representations using paired encoders and directional InfoNCE losses, improving downstream-task performance over supervised ImageNet pre-training.CLIP and ALIGN exemplify this approach.
  • Related Works: Prompt and adapter methods such as CoOp, CoCoOp, and CLIP-Adapter adapt CLIP representations with learnable context, visual meta-tokens, or lightweight modules.These approaches motivate parameter-efficient adaptation of vision-language models.
  • Related Works: Before CLIP-ReID, CLIP had been adapted to segmentation, detection, and retrieval, but the cited literature did not address CLIP-based ReID.CLIP-ReID fills this application gap with a dedicated training process.

Method

CLIP-ReID adapts CLIP to ReID by learning ID-specific ambiguous text descriptions, then using them to constrain image-encoder fine-tuning. Its training combines contrastive alignment in the first stage with ReID and image-to-text cross-entropy losses in the second.

  • Method: CLIP represents images and text with separate encoders whose outputs are projected into a shared cross-modal embedding space for similarity comparison.The method uses either a transformer or CNN image encoder and the text encoder’s [EOS] representation.
  • Method: CLIP-ReID addresses index-only ReID labels by complementing missing textual information with ID-specific learnable tokens.The tokens form ambiguous descriptions that can be processed by CLIP’s text encoder.
  • The second training stage: During stage two, the learned descriptions and text encoder remain static while the image encoder is optimized with ID, triplet, and image-to-text cross-entropy losses.The stored text features provide cross-modal constraints for image-encoder fine-tuning.
  • Method: Camera information is added only to the [CLS] token, while overlapping patches can enhance the ViT-based model by changing token-embedding stride.These are optional design choices for incorporating viewpoint information and additional computation.

Datasets and Evaluation Protocols

The evaluation covers person and vehicle ReID datasets and uses Rank-1 cumulative matching characteristics and mean average precision as metrics.

  • Datasets: The study evaluates four person ReID datasets and two vehicle ReID datasets: MSMT17, Market-1501, DukeMTMC-reID, Occluded-Duke, VeRi-776, and VehicleID.The datasets span both person and vehicle identification settings.
  • Evaluation Protocols: Performance is measured with cumulative matching characteristics at Rank-1 and mean average precision.These metrics follow common ReID evaluation practice.

Implementations

The implementation uses CLIP visual and text encoders with CNN- and ViT-based visual backbones, and specifies staged optimization settings for both model types.

  • Models: The implementation uses CLIP’s ViT-B/16 transformer or CNN visual encoder together with its text encoder.For ViT-B/16, a linear layer reduces the 768-dimensional image feature to 512 dimensions to match the text feature output.
  • Training details: Stage one uses Adam with an initial learning rate of 3.5 × 10^-4 and cosine decay, while only learnable text tokens are optimized.The stage-one batch size is 64 and no augmentation is used.
  • Training details: Stage two trains the image encoder with identity and triplet losses using batches of B = P×K images, with P = 16 identities and K = 4 samples per identity.Random flipping, padding, cropping, and erasing are applied during this stage.

Comparison with State-of-the-Art Methods

CLIP-ReID is compared with state-of-the-art CNN- and ViT-based methods on person and vehicle ReID benchmarks, showing strong performance across both settings.

  • CLIP-ReID is compared with state-of-the-art CNN- and ViT-based methods on person and vehicle ReID benchmarks.
  • 63.0% mAP and 84.4% R1 are achieved on MSMT17 with the CNN-based backbone.
  • 73.4% mAP and 88.7% R1 are achieved on MSMT17 with the ViT-based backbone, exceeding Transreid+SIE+OLP by 6.0% and 3.4%.
  • 75.8% mAP and 89.7% R1 are obtained on MSMT17 when SIE and OLP are further used.
  • 85.3% mAP and 97.6% R1 are reached on VehicleID with the ViT-based backbone, while CLIP-ReID! reaches 84.5% mAP and 97.3% R1 on VeRi-776.

Ablation Studies and Analysis

Ablations examine the CLIP initialization, two-stage training, text-encoder losses, token count, auxiliary modules, and visual focus, identifying choices associated with stronger ReID performance.

  • Baseline comparison: CLIP pre-training gives the proposed baseline superior performance compared with other MSMT17 baselines.
  • Necessity of two-stage training: Two-stage training is preferred because jointly optimizing image encoders and text tokens early makes poorly descriptive tokens affect image-encoder optimization.
  • Text-encoder constraints: Comparing text embeddings with all training-set IDs performs better than comparing only the IDs present in the current batch, and Lt2i is unnecessary in stage two.
  • Token analysis: M = 4 learnable tokens gives the best result, whereas M = 1 is insufficient and larger values become redundant and unhelpful.
  • SIE and OLP: Using SIE and OLP together raises MSMT17 mAP and R1 by 2.4% and 1.0%, respectively.
  • Visualization: CLIP-ReID focuses on a more comprehensive image area than TransReID and the baseline models, which focus on local regions.

Conclusion

The paper applies vision-language pre-training to image ReID through CLIP-ReID, using ID-specific learnable text tokens and two-stage training to improve visual representations.

  • Fine-tuning CLIP-initialized ResNet-50 or ViT-B/16 visual models gives good performance compared with other baselines.
  • CLIP-ReID optimizes ID-specific learnable text tokens first, then uses the tokens and text encoder as constraints while optimizing the image encoder.
  • Experiments on person and vehicle datasets demonstrate the effectiveness of the text descriptions and the superiority of the proposed model.

Supplementary Material

Supplementary analyses describe a faster first-stage variant and retrieval visualizations, while noting a performance trade-off for the efficiency improvement.

  • A faster first-stage method computes average image features offline and buffers them in memory because the image encoder remains fixed.
  • The offline feature-buffering method shortens first-stage training time at the expense of lower final performance.
  • Retrieval results on MSMT17 mark incorrectly identified samples in orange.

The Ltri of the previous layer.

Adding Ltri after a preceding layer generally improves discrimination between IDs, with the insertion point chosen differently for ViT-B/16 and ResNet-50. Additional analyses examine token-count sensitivity, inference-feature combinations, and vehicle ReID performance.

  • The Ltri of the previous layer.: Adding Ltri after the previous layer usually improves the model’s ability to discriminate between different IDs.The method applies it after the 11th transformer layer in ViT-B/16 and the 3rd residual layer in ResNet-50.
  • The Ltri of the previous layer.: Ltri is evaluated in the CNN-based CLIP-ReID illustration shown in Figure 5.
  • Number of learnable tokens M.: M = 4 gives the best results, while performance is not sensitive to the number of learnable tokens.
  • Inference features.: The final inference representation concatenates the image feature and post-image feature.
  • Vehicle ReID evaluation.: CLIP-ReID achieves promising results across the small, medium, and large VehicleID test settings.The vehicle evaluation also covers VeRi-776 and VehicleID comparisons with state-of-the-art methods.
Loading 2211.13977v4…