PaperlayerView exact revision on arXiv

Source-linked AI summary

Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese

An Yang, Junshu Pan, Junyang Lin, Rui Men, Yichang Zhang, Jingren Zhou, Chang Zhou

arXiv:2211.01335v3cs.CVcs.CL

TL;DR

Chinese CLIP addresses the challenge of adapting CLIP to Chinese-native vision-language data. It constructs a large Chinese image-text dataset and uses two-stage pretraining with frozen-then-joint encoder optimization. The resulting models achieve state-of-the-art retrieval performance across three Chinese benchmarks and competitive zero-shot image classification.

  • Problem

    CLIP is difficult to transfer effectively to Chinese-native cross-modal retrieval because machine-translated data perform poorly and language-specific visual domains remain insufficiently modeled.

  • Method

    Chinese CLIP pretrains CLIP-compatible models on large-scale Chinese image-text pairs, first freezing the image encoder while training the text encoder, then jointly optimizing both encoders.

  • Results

    Chinese CLIP achieves state-of-the-art performance on MUGE, Flickr30K-CN, and COCO-CN for zero-shot learning and finetuning, plus competitive zero-shot classification across 10 datasets.

  • Takeaways & Limitations

    Language-specific multimodal pretraining with two-stage optimization provides a Chinese CLIP foundation model effective for retrieval and zero-shot image classification.

  • Takeaways & Limitations

    The pretraining dataset is relatively small compared with recent studies, and many image-text pairs remain improperly matched, potentially providing negative information.

Abstract

from arXiv · show

The tremendous success of CLIP (Radford et al., 2021) has promoted the research and application of contrastive learning for vision-language pretraining. In this work, we construct a large-scale dataset of image-text pairs in Chinese, where most data are retrieved from publicly available datasets, and we pretrain Chinese CLIP models on the new dataset. We develop 5 Chinese CLIP models of multiple sizes, spanning from 77 to 958 million parameters. Furthermore, we propose a two-stage pretraining method, where the model is first trained with the image encoder frozen and then trained with all parameters being optimized, to achieve enhanced model performance. Our comprehensive experiments demonstrate that Chinese CLIP can achieve the state-of-the-art performance on MUGE, Flickr30K-CN, and COCO-CN in the setups of zero-shot learning and finetuning, and it is able to achieve competitive performance in zero-shot image classification based on the evaluation on the ELEVATER benchmark (Li et al., 2022). We have released our codes, models, and demos in https://github.com/OFA-Sys/Chinese-CLIP

1 Introduction

Chinese CLIP addresses the difficulty of transferring CLIP to Chinese-native vision-language data by pretraining on Chinese image-text pairs with a two-stage method. It achieves strong retrieval and classification performance across Chinese benchmarks.

  • Motivation: CLIP transfers poorly to Chinese-native cross-modal retrieval, motivating language-specific vision-language pretraining.The paper identifies language-native data modeling as important and reports large performance gaps between original CLIP and Chinese CLIP.
  • Approach: Chinese CLIP uses publicly available Chinese image-text pairs and retains the OpenAI CLIP architecture.The proposed model is pretrained on collected Chinese data rather than transferred only through machine translation.
  • Approach: The two-stage method first freezes the image encoder while optimizing the text encoder, then jointly trains both encoders.The method initializes the encoders from pretrained CLIP and Chinese RoBERTa models before these stages.
  • Results: Chinese CLIP achieves state-of-the-art cross-modal retrieval on MUGE, Flickr30K-CN, and COCO-CN in zero-shot and finetuning setups.The evaluation covers three Chinese retrieval datasets and reports strong performance for large- and huge-size models.
  • Results: Chinese CLIP achieves competitive zero-shot image-classification performance on the ELEVATER benchmark and outperforms Chinese baselines.The paper also provides TensorRT and ONNX models with inference around 2 to 10 times faster than PyTorch models.

2 Method

The method adapts pretrained vision and language encoders to Chinese multimodal data through data filtering, initialization, and two-stage contrastive pretraining. It evaluates the resulting models using retrieval tables covering zero-shot and finetuning settings.

  • Method overview: Chinese CLIP is pretrained on large-scale Chinese image-text data using the same general CLIP architecture.The approach targets a Chinese-specific vision-language foundation model.
  • Data: The dataset combines publicly available sources, including large-scale web image-text collections, with additional internal image-text pairs.Around 20 million high-quality internal pairs are added for diversity.
  • Data: Preprocessing removes low-scoring, blacklisted, and overly short or long captions before image processing.LAION-5B samples with mCLIP scores below 0.26 are removed.
  • Initialization: The encoders are initialized from pretrained CLIP vision checkpoints and Chinese RoBERTa-family text checkpoints.This initialization leverages existing pretrained models for transfer to Chinese data.
  • Two-stage pretraining: Stage 1 applies Locked-image Tuning with the image encoder frozen, while Stage 2 unfreezes and jointly optimizes both encoders.The first stage aligns text representations with the pretrained vision model; the second adapts the image encoder to Chinese-domain images.

3 Evaluation

Chinese CLIP is evaluated on Chinese cross-modal retrieval and open-domain zero-shot image classification. It outperforms baselines across retrieval benchmarks, benefits from scaling and two-stage pretraining, and remains sensitive to domain-specific concepts, prompts, and negation.

  • Datasets and Metrics: Chinese CLIP is evaluated on MUGE-Retrieval, Flickr30K-CN, and COCO-CN using zero-shot and finetuning setups, with Recall@K and Mean Recall metrics.MUGE-Retrieval contains Chinese e-commerce data, while Flickr30K-CN and COCO-CN translate captions from English datasets.
  • Cross-Modal Retrieval Results: 17.0 MR: CN-CLIPViT-B/16 surpasses WukongViT-B/32 in zero-shot MUGE-Retrieval, while its finetuning advantage over R2D2ViT-B is 8.7 MR.The tiny CN-CLIPRN50 also exceeds the base-size WukongViT-B/32 by 8.9 MR in zero-shot learning and 8.0 MR in finetuning.
  • Cross-Modal Retrieval Results: 17.0 R@1: CN-CLIPViT-B/16 surpasses WukongViT-B/32 in zero-shot Flickr30K-CN text-to-image retrieval, while the tiny model exceeds it by 3.1 R@1 in image retrieval.CN-CLIP also outperforms baselines on COCO-CN, including a 2.6 R@1 gain for tiny-model finetuned text retrieval.
  • Cross-Modal Retrieval Results: 3.9 MR: CN-CLIPViT-H/14 surpasses the 2.5-billion-parameter T-Bletchley7 on finetuned COCO-CN despite using fewer model parameters and pretraining data.Scaling from CN-CLIPViT-L/14 to CN-CLIPViT-H/14 improves almost all Flickr30K-CN and COCO-CN metrics.
  • Ablation Study: Switching from Stage 1 to Stage 2 effectively boosts retrieval performance, while the benefit of LiT varies between Chinese-native and translated datasets.Pretraining from scratch performs much worse; two-stage pretraining is significantly better than omitting LiT on Flickr30K-CN and COCO-CN, but the gap is small on MUGE.
  • Open-Domain Image Classification: Chinese CLIP significantly surpasses Chinese baselines on all ELEVATER image-classification datasets, with relative gains exceeding 100% on some datasets.It also outperforms English CLIP or ALIGN on several datasets, including CIFAR, MNIST, and Pascal-VOC-2007.
  • Analysis: 16.0 accuracy: aircraft-specific prompts outperform translated OpenAI prompts at 13.8, showing sensitivity to handcrafted prompt design.Proper-noun classification remains difficult, and translation or transliteration can significantly affect Chinese CLIP performance.
  • Analysis: 48.1% in accuracy: PatchCamelyon performance changes from 49.9 to 25.9 under label variation, illustrating limited learning of negation.Pretraining texts usually describe present objects or features rather than their absence.

4 Related Work

Vision-language pretraining has shifted from cross-modal fusion models toward contrastive-learning-based two-tower models such as CLIP, which supports retrieval, zero-shot classification, and downstream multimodal applications.

  • CLIP is a contrastive-learning-based two-tower model that can serve as a vision foundation model.
  • Contrastive-learning-based multimodal models have achieved new state-of-the-art results in cross-modal retrieval and zero-shot classification.
  • CLIP can adapt to other models and is used in image-generation systems including DALL-E, DALL-E 2, and Stable Diffusion.

5 Conclusion

The paper introduces Chinese CLIP, a Chinese-specific vision-language foundation model trained on around 200 million samples with a two-stage pretraining method. Experiments report state-of-the-art retrieval performance and competitive zero-shot classification across multiple datasets.

  • Chinese CLIP constructs a pretraining dataset of around 200 million samples and trains a series of Chinese CLIP models.
  • Chinese CLIP reaches state-of-the-art performance on multiple cross-modal retrieval datasets in zero-shot learning and finetuning.
  • Chinese CLIP achieves competitive performance in zero-shot image classification across 10 datasets.

Limitations

The authors identify limitations involving pretraining data scale and quality, model scaling range, and the effectiveness of small models for deployment.

  • Data: The pretraining dataset contains around 200 million samples but remains relatively small compared with recent studies.
  • Data: The authors still find it difficult to determine what constitutes a high-quality CLIP dataset.
  • Data: Some image-text pairs are improperly matched and may provide negative information during pretraining.
  • Model: Chinese CLIP model scaling improves downstream retrieval and classification performance, but the largest model remains smaller than recent larger-scale models.
  • Model: The smallest CN-CLIPRN50 performs much worse than ViT variants, motivating distillation to transfer large-model capability to deployable small models.

Ethics Statement

The model produces image and text features that can support downstream applications, but those representations may contribute to harmful generated content. The authors plan to filter pretraining data to reduce potential risks.

  • The model’s image and text representations can support search engines, recommender systems, and image generation models.The authors note that generated content may contain hate, bias, or pornography.
  • The authors identify harmful information in generated content as a potential risk associated with CLIP representations.
  • Future work will study filtering pretraining data to avoid these potential risks.

A.1 Model Architecture Details

Chinese CLIP provides five models ranging from approximately 77 to 958 million parameters, using ResNet or ViT image encoders with Chinese text encoders. Their architectures remain aligned with OpenAI CLIP or LAION CLIP to enable image-encoder initialization from pretrained weights.

  • 5 Chinese CLIP models span approximately 77 to 958 million parameters.The family includes one ResNet-50 model and four ViT models.
  • The smallest CN-CLIPRN50 pairs a ResNet-50 image encoder with an RBT3 text encoder.
  • CN-CLIPViT-B/16 uses a ViT-B/16@224px image encoder and a RoBERTa-wwm-Base text encoder.
  • Chinese CLIP retains OpenAI CLIP backbone architectures for ResNet-50, ViT-B/16, and ViT-L/14, while ViT-H/14 follows LAION CLIP.This architecture compatibility enables initialization of Chinese CLIP image encoders with pretrained weights.
  • The model configurations are organized by size and architecture through shared and ViT-specific hyperparameter tables.

A.2 Pretraining Details

Chinese CLIP pretraining initializes image and text encoders from pretrained models, then uses a two-stage procedure that first freezes and later unfreezes the image encoder. Training spans multiple hardware and compute configurations across model sizes.

  • Image encoders are initialized from OpenAI CLIP or LAION CLIP weights, while text encoders use released Chinese RoBERTa weights.
  • Stage 1: Stage 1 pretrains with image augmentation, cross-GPU all-gather communication, and a frozen image encoder.Stage 1 uses random resize cropping and AutoAugment on input images.
  • Stage 2: Stage 2 unfreezes the image encoder and updates all model parameters with a reduced learning rate of 2e-5.Batch sizes are reduced for several models because of GPU memory limits.
  • CN-CLIPViT-L/14 is adapted to 336 × 336 resolution by interpolating image positional embeddings and continuing pretraining for 2 epochs.This continuation uses 128 NVIDIA A100 GPUs for 0.7 days.
  • The ResNet-based CN-CLIPRN50 architecture is included among the pretraining configurations.

A.3 Finetuning Details

Chinese CLIP is evaluated through finetuning and zero-shot experiments across retrieval and image-classification benchmarks, with additional deployment tests. The results report strong retrieval performance, scaling gains in classification, faster converted-model inference, and limited retrieval degradation after conversion.

  • Retrieval: Finetuning is conducted on MUGE, Flickr30K-CN, COCO-CN, and ICR using strategies and losses consistent with pretraining.
  • Retrieval: Chinese CLIP achieves state-of-the-art cross-modal retrieval performance on the longer-text ICR dataset.ICR captions average 45.3 words, so the maximum finetuning text length is set to 128.
  • Zero-shot classification: Scaling model size consistently improves zero-shot image-classification performance on the ELEVATER ICinW benchmark.The tiny CN-CLIPRN50 performs much worse than the substantially larger ViT variants.
  • Deployment: Chinese CLIP supports conversion to ONNX and TensorRT models for faster text and vision representation generation.
  • Deployment: ONNX-based and TensorRT-based models optimize inference speed over native PyTorch implementations for almost all model scales.The benchmark uses batch size 1 and FP16 precision on a single NVIDIA T4 GPU.
  • Deployment: Converted ONNX-based and TensorRT-based models retain representation quality with no more than 0.1 MR degradation on zero-shot MUGE retrieval.
Loading 2211.01335v3…