Source-linked AI summary
Unified Contrastive Learning in Image-Text-Label Space
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, Jianfeng Gao
TL;DR
Visual recognition must balance discriminative learning from annotated image-label data with broad concept coverage from webly crawled image-text pairs. UniCL unifies both sources in an image-text-label space with a single contrastive objective, achieving broad improvements across recognition settings and large-scale benchmarks. The study also finds that UniCL alone performs competitively on pure image-label data.
Problem
Existing visual recognition approaches separately exploit discriminative human-annotated image-label data or broad but noisy webly crawled image-text data.
Method
UniCL represents image-label and image-text data in a common image-text-label space and trains them with one unified contrastive learning objective.
Results
UniCL improves performance across zero-shot, linear-probe, fully finetuning, and transfer learning settings, and scales successfully to billions of image-text-label data across dozens of benchmarks.
Takeaways & Limitations
UniCL provides a generic multimodal learning paradigm and is also a competitive learner on pure image-label data across ResNet and Swin Transformer backbones.
Abstract
from arXiv · showhide
Visual recognition is recently learned via either supervised learning on human-annotated image-label data or language-image contrastive learning with webly-crawled image-text pairs. While supervised learning may result in a more discriminative representation, language-image pretraining shows unprecedented zero-shot recognition capability, largely due to the different properties of data sources and learning objectives. In this work, we introduce a new formulation by combining the two data sources into a common image-text-label space. In this space, we propose a new learning paradigm, called Unified Contrastive Learning (UniCL) with a single learning objective to seamlessly prompt the synergy of two data types. Extensive experiments show that our UniCL is an effective way of learning semantically rich yet discriminative representations, universally for image recognition in zero-shot, linear-probe, fully finetuning and transfer learning scenarios. Particularly, it attains gains up to 9.2% and 14.5% in average on zero-shot recognition benchmarks over the language-image contrastive learning and supervised learning methods, respectively. In linear probe setting, it also boosts the performance over the two methods by 7.3% and 3.4%, respectively. Our study also indicates that UniCL stand-alone is a good learner on pure image-label data, rivaling the supervised learning methods across three image classification datasets and two types of vision backbones, ResNet and Swin Transformer. Code is available at https://github.com/microsoft/UniCL.
1. Introduction
Visual recognition uses either annotated image-label data or webly crawled image-text pairs, creating a tension between discriminative representations and broad concept coverage. UniCL addresses this by placing both data types in a shared image-text-label space and learning with one unified paradigm.
- Data sources: Supervised learning uses human-annotated image-label pairs, while language-image contrastive learning leverages large, noisy webly crawled image-text pairs.Image-label data is precise but laborious and difficult to scale, whereas image-text pairs cover many visual concepts.
- Research question: The central question is whether one model can combine discriminative representations with broad visual concept coverage.
- Unified formulation: UniCL defines an image-text-label space that eliminates the boundary between image-label and image-text data.It maps labels to textual concepts and assigns unique labels to textual descriptions, placing both learning paradigms in a common space.
- Proposed method: UniCL is a unified contrastive learning method that learns from image-label data, image-text data, or both.
- Empirical scope: UniCL improves image recognition across zero-shot, linear-probe, fully finetuning, and transfer learning settings.
- Scale-up result: At billion-scale, UniCL demonstrates superiority over CLIP and ALIGN across dozens of benchmarks.
2. Related works
Prior work includes supervised image classification and image self-supervised learning, while UniCL extends contrastive learning into a shared image-text-label setting. Its goal is a generic visual-semantic representation learned from combined image-label and image-text data.
- Supervised Learning: Supervised learning maps images to manual labels, with progress spanning datasets, architectures, and objectives such as Cross-Entropy and supervised contrastive loss.
- Supervised Learning: UniCL treats image-label data as image-text-label data by using the textual concepts behind labels as a special language format.
- Unified Contrastive Learning: UniCL applies contrastive learning in an image-text-label setting to learn a generic visual-semantic space from combined image-label and image-text data.
- Self-Supervised Learning: Image self-supervised learning can use unlabelled images but lacks language association, limiting direct applicability to zero-shot recognition.
3. Method
UniCL represents image-label and image-text data as image-text-label triplets, then learns aligned visual and language representations with a bidirectional contrastive objective. Labels determine positive and negative pairs, allowing multiple images sharing a label to align with the same language description.
- 3.1. Preliminaries: UniCL unifies image-label and image-text data as triplets containing an image, language description, and grouped or unique label index.Image-text pairs receive unique labels, while image-label pairs associate each label with its indexed concept name.
- 3.2. Unified Image-Text-Label Contrast: The image encoder and text encoder produce normalized feature vectors whose similarity is computed between every image and language description in a batch.The normalized features lie on a hypersphere, and similarity is their inner product.
- 3.2. Unified Image-Text-Label Contrast: UniCL optimizes a bidirectional contrastive objective combining image-to-text and text-to-image losses.The image-to-text term aligns matched images with text, while the text-to-image term aligns matched texts with images.
- 3.2. Unified Image-Text-Label Contrast: Labels define positive and negative pairs: same-label image-language pairs are positive, whereas unmatched pairs are negative.Image-to-text loss is computed row-wise and text-to-image loss column-wise in the similarity matrix.
- 3.3. Discussions & Properties: Compared with CLIP, UniCL allows any images sharing a language-description label to be positives rather than assuming only diagonal image-text matches.The additional image-to-text term also acts as a regularizer, while the text encoder can process category names or free-form text.
- 3.4. Model Training and Adaptation: During training, UniCL modifies labels so unique language descriptions receive unique indices, and the learned encoders support open-vocabulary recognition or visual-backbone adaptation.The visual backbone can be used for linear probing, full fine-tuning, or object detection.
4. Experiments
The experiments evaluate UniCL across image classification, zero-shot recognition, linear probing, object detection, and data-unification settings. Results indicate that combining image-label and image-text data supports both discriminative representations and broad visual concept understanding.
- 4.1. Results of UniCL on image classification: Experiments compare UniCL with CE and SupCon on CIFAR-10, CIFAR-100, and ImageNet-1K using ResNet and Swin Transformer visual encoders.Models use standard random-crop augmentation, 500 training epochs, and batch size 4096.
- 4.1. Results of UniCL on image classification: UniCL improves over CE particularly when over-fitting is more severe, including by around 1–3 points on CIFAR and 3 points for Swin Transformer on ImageNet-1K.When over-fitting is less severe or stronger augmentation is used, UniCL remains on par with CE.
- 4.1. Results of UniCL on image classification: The bidirectional loss is important: retaining only the L_t2i term causes a 3% performance drop, while performance remains robust across smaller batch sizes.The authors attribute batch-size robustness partly to repeatedly used category-name embeddings and consistent label information.
- 4.2.1 Benefit of image-text to image-label: Adding image-text pairs to ImageNet-1K generally improves zero-shot, linear-probe, ImageNet-1K, and COCO detection performance across the evaluated metrics.GCC-15M produces improvements of +1.9 on ImageNet-1K, +3.5 on linear probe, and +1.2 on COCO detection.
- 4.2.2 Benefit of image-label to image-text: Unifying half of YFCC-14M with half of ImageNet-21K raises zero-shot performance by absolute 6% on ImageNet-1K and absolute 7% across 14 datasets, while improving linear probe by +7.3%.UniCL wins on 11 of 14 datasets, and the combined data improves performance on almost all reported metrics.
- 4.2.2 Benefit of image-label to image-text: Image-label data makes fine-grained visual features more discriminative, while image-text data improves concept coverage and semantic generalization.The visual-feature analysis shows fine-grained dog breeds are mixed with image-text-only training but become clearly grouped when image-label data is added.
5. Conclusion
UniCL is a unified contrastive learning paradigm built in image-text-label space to combine image-label and image-text data for discriminative, semantically rich representations. The paper reports improvements across recognition settings and finds UniCL effective even on pure image-label data.
- UniCL combines image-label and image-text pairs in a unified image-text-label space for representation learning.
- UniCL brings improvements across zero-shot, linear probe, finetuning and transfer learning benchmarks.
- UniCL stand-alone is a good alternative learner on pure image-label data.
- The paper focuses mainly on vision tasks using public datasets, while Florence covers broader tasks with large-scale private data.
A. Validation dataset details
The validation datasets are documented with dataset statistics and vocabulary sizes. Vocabulary size is typically larger than the number of concepts or classes.
- Table 7 lists statistics for all validation datasets used in the experiments.
- Vocabulary size is typically larger than the number of concepts or classes in each validation dataset.
B.1. Training on image classification data
The image-classification training setup uses ResNet and Swin Transformer visual encoders with a 12-layer Transformer language encoder. Visual and textual features are projected into a shared 512-dimensional space.
- Model architecture: The visual encoder uses ResNet or Swin Transformer architectures.
- Model architecture: A 12-layer Transformer with hidden dimension 512 serves as the language encoder.
- Model architecture: Visual and textual encoder features are projected to the same dimension of 512.
B.2. Training on image-text-label space
The joint image-text-label training protocol uses Swin-Tiny and addresses severe data imbalance between image-label and image-text datasets. Balanced sampling is reported as important for achieving the main-submission performance.
- Training protocol for image-text-label space: The joint image-text-label models use Swin-Tiny and mostly follow the Section 4.1 training settings.
- Training protocol for image-text-label space: Image-label and image-text datasets can be severely imbalanced, with around 1.3M ImageNet-1K images versus above 12M GCC-12M images.
- Training protocol for image-text-label space: Balanced sampling is very important for achieving the reported performance.
- Training protocol for image-text-label space: For fair comparison, the CLIP training protocol uses random-crop augmentation, 32 epochs, batch size 4096, learning rate 1e-3 and weight decay 0.1.
C.1. Results over separate datasets
UniCL benefits from adding image-caption pairs to ImageNet-1K for zero-shot classification across 14 datasets, with especially strong gains on fine-grained tasks.
- UniCL outperforms the baseline on most of the 14 zero-shot datasets after adding image-caption pairs to ImageNet-1K.
- The gains are especially pronounced on fine-grained classification tasks such as Food101 and OxfordPets.
C.2. Results with larger vision backbone
With a larger Swin-Base backbone, combining image-label and image-text data consistently improves zero-shot recognition and object detection transfer performance. The results support UniCL as a model-size-agnostic learning paradigm.
- Combining image-label and image-text data improves zero-shot recognition by 8.6 points on ImageNet-1K and 11.0 points across 14 datasets in average.
- Using the full ImageNet-21K and YFCC-14M datasets further significantly improves both zero-shot performance measures.
- Increasing visual encoder size brings substantial gains in all cases, particularly when both data types are combined.
- Combining the two data types improves object detection performance by around 2 points for both box and mask mAP compared with a CLIP-based YFCC-14M model.
- Using the full data set adds around 1 point for both object-detection metrics.
D.2. Concept coverage
Concept coverage helps characterize zero-shot transfer, but semantic overlap alone does not determine performance. UniCL can achieve reasonable recognition even when training and validation concepts have little or no overlap.
- ImageNet-1K overlaps with CIFAR-100 by 24.0% and Caltech-101 by 24.5%, potentially explaining their relatively higher zero-shot performance.
- Despite less or no concept coverage, ImageNet-1K pretraining still achieves reasonably good zero-shot performance on CIFAR-10, Flowers102, and Oxford Pet.
- ImageNet-21K has zero concept overlap with Hateful Memes and PatchCamelyon, yet the model still achieves reasonable performance on them.
- Image-text datasets generally provide higher concept coverage than image-label datasets, but higher coverage does not necessarily yield better zero-shot performance.
- The authors suggest domain gaps and difficulty grounding multiple concepts in image-text pairs may limit the value of shared semantic coverage.
D.3. Concept visualizations
t-SNE visualizations show that joint training produces concept embeddings that better reflect visually grounded semantic relationships than image-label-only training.
- Joint ImageNet-1K and GCC-15M training understands novel ImageNet-21K concepts better than ImageNet-1K-only training.
- The image-label-only model places “porthole” near “porcupine,” whereas the joint model retrieves “porcuponefish” as a close neighbor.
- The image-label-only model confuses “goblet” and “coverlet,” while the joint model finds “liqueur glass” as a semantically and visually similar concept.
- Across the visualizations, joint training learns visually grounded semantic meanings for diverse concepts.