PaperlayerView exact revision on arXiv

Source-linked AI summary

Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmark

Jiaxi Gu, Xiaojun Meng, Guansong Lu, Lu Hou, Minzhe Niu, Xiaodan Liang, Lewei Yao, Runhui Huang, Wei Zhang, Xin Jiang, Chunjing Xu, Hang Xu

arXiv:2202.06767v4cs.CVcs.LG

TL;DR

Chinese VLP research lacks large-scale datasets, benchmarks, and reusable pretrained models. Wukong supplies a 100-million-pair web dataset, benchmark resources, and several pretrained model variants, with strong reported classification and retrieval results. Its scope remains bounded by the time, culture, language style, and text-length coverage of Chinese Internet data.

  • Problem

    Large-scale Chinese vision-language datasets, benchmarks, and publicly available pretrained models are limited, while training such models requires substantial resources.

  • Method

    The paper releases Wukong with 100 million web-collected Chinese image-text pairs, human-verified benchmarks, and dual-stream models using varied encoders and similarity methods.

  • Results

    Wukong models show prominent performance across downstream tasks, including 73.03% average top-1 accuracy on 10 zero-shot classification datasets and 71.6% mean recall on AIC-ICC.

  • Takeaways & Limitations

    Wukong serves as a promising Chinese pretraining dataset and benchmark for different cross-modal learning methods.

  • Takeaways & Limitations

    The dataset reflects Chinese Internet culture and language style, may contain more written than spoken language, excludes newly emerging concepts, and lacks very long texts.

Abstract

from arXiv · show

Vision-Language Pre-training (VLP) models have shown remarkable performance on various downstream tasks. Their success heavily relies on the scale of pre-trained cross-modal datasets. However, the lack of large-scale datasets and benchmarks in Chinese hinders the development of Chinese VLP models and broader multilingual applications. In this work, we release a large-scale Chinese cross-modal dataset named Wukong, which contains 100 million Chinese image-text pairs collected from the web. Wukong aims to benchmark different multi-modal pre-training methods to facilitate the VLP research and community development. Furthermore, we release a group of models pre-trained with various image encoders (ViT-B/ViT-L/SwinT) and also apply advanced pre-training techniques into VLP such as locked-image text tuning, token-wise similarity in contrastive learning, and reduced-token interaction. Extensive experiments and a benchmarking of different downstream tasks including a new largest human-verified image-text test dataset are also provided. Experiments show that Wukong can serve as a promising Chinese pre-training dataset and benchmark for different cross-modal learning methods. For the zero-shot image classification task on 10 datasets, $Wukong_{ViT-L}$ achieves an average accuracy of 73.03%. For the image-text retrieval task, it achieves a mean recall of 71.6% on AIC-ICC which is 12.9% higher than WenLan 2.0. Also, our Wukong models are benchmarked on downstream tasks with other variants on multiple datasets, e.g., Flickr8K-CN, Flickr-30K-CN, COCO-CN, et al. More information can be referred to: https://wukong-dataset.github.io/wukong-dataset/.

1 Introduction

Wukong addresses the limited availability of large-scale Chinese vision-language data and reusable models by releasing a 100-million-pair dataset, benchmark resources, and multiple pretrained architectures. Experiments report strong performance across Chinese classification and retrieval benchmarks.

  • Motivation: Large-scale Chinese VLP datasets and publicly available pretrained models remain limited, while training comparable models requires substantial hardware and financial resources.Examples cited include CLIP training for 18 days on 592 NVIDIA-V100 GPUs and M6-10T training on 512 GPUs for around 10 days.
  • Findings: Token-wise similarity preserves fine-grained word-patch alignment, while character-grained tokenization outperforms Chinese word-grained tokenization in the reported models.The alignment finding applies even when image encoders are frozen during contrastive learning.
  • Results: 73.03% average top-1 accuracy is achieved by WukongViT-L on 10 zero-shot image-classification datasets.This is reported as an average across the 10 datasets.
  • Results: 71.6% mean recall is achieved on AIC-ICC, 12.9% higher than WenLan 2.0 for image-text retrieval.The comparison is reported for WukongViT-L against WenLan 2.0.
  • Contributions: Wukong releases 100 million Chinese image-text pairs, human-verified benchmark data, and models trained with multiple architectures and pretraining methods.The released models use different image encoders and methods including ViT, SwinT, CLIP, FILIP, and LiT.

2 Related Work

Prior VLP work established dual-modality pretraining alongside broader single-stream and dual-stream architectures. Dataset scale and diversity range from small hand-annotated benchmarks to large online-collected image-text corpora.

  • Vision-Language Pre-training: VLP models extend large-scale pretraining and downstream transfer paradigms from language modeling to joint vision-language learning.CLIP, ALIGN, and FILIP are cited as examples showing strong downstream performance.
  • Vision-Language Pre-training: Single-stream models concatenate visual and textual embeddings, whereas dual-stream models encode the modalities separately for cross-modal representation learning.Single-stream models are described as suitable for image captioning and text-to-image generation, while the supplied passage contrasts the architectural categories.
  • Vision-Language Datasets: Small hand-annotated datasets have limited domain and diversity, while online-collected data has supported promising large-scale VLP results.Examples of small datasets include COCO-Captions, Flickr30k, Visual Genome, and VQA2; larger examples include CC3M and CC12M.

3 Construction of Wukong Dataset

Wukong is constructed from Chinese web image-text pairs retrieved with frequent Chinese keywords and refined through image-, text-, and privacy-oriented filtering. The resulting Wukong and Wukong-Test datasets support pretraining and evaluation, while released models use several similarity designs.

  • Data Collection: The collection begins with the 200,000 most frequent Chinese keywords and retrieves image-text pairs from Baidu.At most 1,000 pairs are retained per keyword, producing 166 million raw pairs before filtering.
  • Released Models: Released models use global similarity, token-wise similarity, or token-wise similarity with a token reduction layer across ViT-B, ViT-L, and Swin-L encoders.The Wukong-style models combine token-wise similarity with reduced-token interaction.
  • Image-based Filtering: Image filtering retains images larger than 200 pixels in both dimensions with a large-to-small dimension ratio of at most 3.The stated goal is to remove images that are too small, tall, or wide for later processing.
  • Text-based Filtering: Text filtering keeps Chinese descriptions with 1 to fewer than 32 Chinese characters and removes meaningless or overly repeated descriptions.Texts appearing more than 10 times in the corpus are discarded; sensitive words and person names receive additional privacy handling.
  • Dataset Outputs: The finalized outputs are Wukong for pretraining and Wukong-Test for model testing.Table 3 is identified as providing their dataset statistics.

4 Methodology

Wukong uses dual-stream Transformer encoders to align images and Chinese text through contrastive learning. Its methodology combines global and token-wise similarity, learnable visual-token reduction, and locked-image text tuning.

  • Model Architecture: The dual-stream architecture encodes images and text separately into a shared embedding space, using contrastive loss to align paired examples and separate unpaired ones.Visual and textual encoders produce embeddings of the same dimension before contrastive training.
  • Model Architecture: ViT and SwinT serve as alternative visual backbones, while the text encoder is a decoder-only Transformer with Chinese character-oriented WordPiece tokenization.The text encoder uses 12 layers, 8 attention heads, and hidden dimension 512.
  • Similarity Functions: Global similarity compares projected, L2-normalized representations of complete image and text sequences, using [CLS] or pooled patch features for images and [SEP] for text.SwinT representations use average pooling because SwinT has no [CLS] token.
  • Similarity Functions: Token-wise similarity matches each visual token with the most similar non-padded text token and averages these maxima to capture fine-grained word-patch alignment.The text-to-image direction excludes [CLS], [SEP], and padding tokens.
  • Efficiency: Reduced-token interaction lowers computation by learning a smaller set of visual tokens before token-wise similarity calculation.The method targets visual tokens because they typically outnumber textual tokens; the reduced representations replace the original visual tokens for similarity computation.
  • Cross-lingual Tuning: Locked-image text tuning freezes an English-pretrained image encoder while training a Chinese text encoder to align cross-lingual image-text representations.The method removes the existing projection layer and adds a new linear projection layer.

5 Wukong Chinese Benchmarks

Wukong models are benchmarked across zero-shot classification, image-text retrieval, and ablation settings. Results favor ViT-L and show benefits from locked-image tuning, token-wise similarity, and character-grained tokenization, with some architecture-specific trade-offs.

  • Benchmark Scope: The benchmark evaluates global similarity, token-wise similarity, and token-wise similarity with reduced-token interaction across ViT-B, ViT-L, and Swin-L encoders.
  • Zero-shot Image Classification: 73.03% average top-1 accuracy is achieved by WukongViT-L across 10 zero-shot image-classification datasets.
  • Image-Text Retrieval: Wukong-trained models generally outperform BriVL on zero-shot retrieval, while WukongViT-L remains competitive across multiple datasets.
  • Image-Text Retrieval: 12.9% higher performance than WenLan 2.0 is achieved by Wukong on AIC-ICC image-text retrieval.
  • Ablations and Findings: Locked-image text tuning decreases loss more slowly but maintains higher validation accuracy than unlocked-image training across almost every iteration.
  • Ablations and Findings: Word-patch visualizations show that FILIP-based models can identify target-object patches, suggesting potential support for image object localization.
  • Ablations and Findings: Character-grained tokenization performs better than word-grained tokenization, supporting finer-grained image-patch and text-token alignment.

6 Conclusion

The paper presents Wukong as a hundred-million-scale Chinese vision-language dataset and introduces three associated VLP models. WukongViT-L achieves state-of-the-art performance on Chinese zero-shot classification and image-text retrieval benchmarks, while future work targets multilingual models and additional applications.

  • Wukong is described as the first hundred-million-level Chinese vision-language dataset, intended to support future Chinese cross-modal pre-training research.
  • The authors introduce WukongViT-B, WukongViT-L, and WukongSwin-L using the Wukong dataset.
  • WukongViT-L achieves state-of-the-art performance on Chinese zero-shot image-classification and image-text-retrieval benchmarks.
  • Future work includes multilingual cross-modal training, additional downstream-task evaluation, image search engines, and visual question answering.

A Examples in Wukong Dataset

Wukong contains diverse image-text content spanning social news, sporting events, and product introductions. Its word and noun-frequency analyses indicate broad conceptual coverage with a long-tail distribution.

  • Examples in Wukong Dataset: Dataset examples include social news, sporting events, product introductions, and other content types.
  • Examples in Wukong Dataset: The dataset word cloud illustrates frequent Chinese terms, including words for month, day, doing, and one.
  • Examples in Wukong Dataset: Noun frequencies follow a long-tail distribution while covering a wide range of concepts.

B Experimental Setup

The experiments use standardized 224 × 224 images and 32-token texts, with model-specific GPU budgets and augmentation. Training uses LAMB and cosine learning-rate scheduling, with translated ImageNet labels for checkpoint selection.

  • Experimental Setup: WukongViT-B, WukongViT-L, and WukongSwin-L are trained for 3, 10, and 5 days using 32, 32, and 40 GPUs, respectively.
  • Experimental Setup: Autoaugment is applied to image data to improve generalization and data efficiency by building more image-text pairs.
  • Experimental Setup: Training uses the LAMB optimizer and cosine learning-rate scheduling, with detailed hyperparameters reported separately.
  • Experimental Setup: All models use 224 × 224 image resolution and text sequences of length 32.
  • Experimental Setup: Translated ImageNet class names are used for zero-shot validation when selecting the optimal checkpoint.

C Supplementary Experiments

The supplementary experiment compares character-grained and word-grained Chinese tokenization for zero-shot image classification. Character-grained tokenization performs better under otherwise identical settings.

  • Character-grained tokenization achieves better top-1 accuracy than word-grained tokenization in WukongViT-B.The comparison keeps all experimental settings the same except tokenization.
  • Chinese word segmentation uses jieba to split text into words for the word-grained model.
  • Character-grained tokens are more fine-grained because a Chinese word can consist of multiple characters, such as “蜂鸟” (“蜂” and “鸟”).

C.2 Visualization of Word-patch Alignment

The visualization evaluates fine-grained alignment between image patches and Chinese textual tokens in Wukong models. FILIP-based models identify target objects and can align individual Chinese characters or words with localized image regions.

  • Alignment method: FILIPViT-L and FILIPSwin-L capture image-text correspondence through token-wise similarity rather than global similarity.The models follow FILIP’s fine-grained interaction design.
  • Alignment method: Token-wise similarity assigns each image patch the textual token with which it has maximum similarity.The input text is tokenized as [CLS], class-label tokens, and [SEP].
  • Visualization findings: Both models predict patches belonging to target objects, while FILIPViT-L provides finer alignment than FILIPSwin-L because its image patches have higher spatial resolution.FILIPViT-L uses 16×16 patchification, whereas FILIPSwin-L has 7×7 output resolution.
  • Visualization findings: FILIPViT-L aligns the Chinese tokens “教” and “堂” with a small church region and outlines a hummingbird in the visualization.
  • Visualization findings: The visualization supports that Wukong models learn Chinese word-patch correspondence, suggesting potential use for image object localization.

D.2 Datasets for Image-text Retrieval

The retrieval benchmark combines datasets with different image-text scales and annotation sources. Caption supervision is human-annotated for several datasets, machine-translated for part of Flickr30K-CN, and paired at different ratios across benchmarks.

  • Table 10 reports the data scale of the image-text retrieval datasets.
  • Flickr8K-CN, COCO-CN, and AIC-ICC use human-annotated texts, while Flickr30K-CN train/validation texts are machine-translated.Flickr30K-CN test texts are human-translated from the original English captions.
  • Each image is paired with five texts in Flickr8K-CN, Flickr30K-CN, and AIC-ICC, whereas COCO-CN pairs each image with one to two texts.

E Limitations and Societal Impacts

The paper identifies temporal, linguistic, coverage, and societal limitations for Wukong. It also describes plans for long-term maintenance and provides datasets under non-commercial licensing terms.

  • Limitations: Wukong may miss concepts, words, expressions, visual objects, and designs that emerge after its collection period.Fine-tuning on up-to-date data is described as a way to address this issue.
  • Limitations: Because Wukong is built from Chinese Internet corpora, it may reflect Chinese cultural and written-language biases and lacks very long texts.The lack of very long texts may limit document-understanding ability.
  • Societal impacts: Models trained on images and texts from unlimited domains may express undesirable and uncontrollable tendencies in image-text correspondence.
  • Maintenance: The authors state that they will maintain Wukong, Wukong-Test, and the proposed and evaluated models over the long term.
  • Terms of use: The datasets are provided for non-commercial purposes under CC BY-NC-SA 4.0 with additional dataset terms.Commercial use, including selling data or using it for commercial gain, is prohibited.
Loading 2202.06767v4…