PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified Contrastive Learning in Image-Text-Label Space

Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, Jianfeng Gao

arXiv:2204.03610v1cs.CVcs.AIcs.LG

TL;DR

Supervised and language-image learning offer different recognition strengths, raising the question of whether one model can provide both discriminative representations and broad visual concept coverage. UniCL unifies image-label and image-text data in a shared image-text-label space, with experiments showing improvements across recognition settings.

  • Problem

    Whether one model can provide both discriminative representations and broad visual concept coverage remains an open question.

    Source: §1 · p. 2

  • Method

    UniCL defines a shared image-text-label space and applies one unified contrastive objective to learn from image-label data, image-text pairs, or both.

    Source: §1 · p. 2

  • Results

    UniCL achieves superior performance across standard zero-shot, linear-probe, fully fine-tuning, and transfer-learning settings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    UniCL is a good alternative learner on pure image-label data, with comparable or better performance across tested datasets and vision architectures.

    Source: §4.1 · p. 5, §5 · p. 8

Abstract

from arXiv · show

Visual recognition is recently learned via either supervised learning on human-annotated image-label data or language-image contrastive learning with webly-crawled image-text pairs. While supervised learning may result in a more discriminative representation, language-image pretraining shows unprecedented zero-shot recognition capability, largely due to the different properties of data sources and learning objectives. In this work, we introduce a new formulation by combining the two data sources into a common image-text-label space. In this space, we propose a new learning paradigm, called Unified Contrastive Learning (UniCL) with a single learning objective to seamlessly prompt the synergy of two data types. Extensive experiments show that our UniCL is an effective way of learning semantically rich yet discriminative representations, universally for image recognition in zero-shot, linear-probe, fully finetuning and transfer learning scenarios. Particularly, it attains gains up to 9.2% and 14.5% in average on zero-shot recognition benchmarks over the language-image contrastive learning and supervised learning methods, respectively. In linear probe setting, it also boosts the performance over the two methods by 7.3% and 3.4%, respectively. Our study also indicates that UniCL stand-alone is a good learner on pure image-label data, rivaling the supervised learning methods across three image classification datasets and two types of vision backbones, ResNet and Swin Transformer. Code is available at https://github.com/microsoft/UniCL.

1. Introduction

The paper introduces an image-text-label space that unifies image-label and image-text data, and proposes UniCL to learn from either source or both. It reports broad downstream performance and later scales UniCL to billions of image-text-label examples.

1. Introduction

  • Supervised learning on annotated image-label pairs supports recognition within given categories and transfer, but precise labels are costly to collect and difficult to scale.Web-crawled image-text pairs offer noisy, free-form coverage of many concepts; CLIP [48] and ALIGN [29] show low-shot recognition, but the paper says such models often lack transfer-learning discrimination.Source: §1 · pp. 1–2
  • UniCL uses images and texts with softened targets derived from labels to align visual and textual features, accommodating image-label data, image-text data, or both.Image-text pairs receive unique labels, while image-label examples use manual labels.Source: §1 · p. 2
  • The image-text-label perspective maps supervised labels to textual concepts and assigns unique labels to image-text descriptions, revealing a common space for both paradigms.Figure 1 presents this unified paradigm as recovering supervised methods such as Cross-Entropy [47] or SupCon [30] and language-image methods such as CLIP [48] or ALIGN [29].Source: Figure 1 · p. 1, §1 · p. 2
  • UniCL is reported to leverage both data types and perform strongly across zero-shot, linear-probe, fully finetuning, and transfer-learning settings.The authors also report scaling UniCL to billions of image-text-label data in Florence, with superiority over CLIP [48] and ALIGN [29] across dozens of benchmarks.Source: §1 · p. 2

2. Related works

UniCL connects supervised recognition, language-image contrastive learning, and self-supervised learning by treating labels as language and unifying image-label and image-text data in a visual-semantic space.

Supervised Learning

  • UniCL treats image-label data as image-text-label data, using label concepts as language to learn a broad visual-semantic representation beyond conventional small-scale fine-grained zero-shot classification.It targets representation learning from combined image-label and image-text pairs.Source: §2 · p. 2

Language-Image Contrastive Learning

  • Unlike CLIP [48] and ALIGN [29], which use webly crawled image-text pairs but ignore image-label data, UniCL unifies both sources in contrastive learning.Prior vision-language research also includes multimodal fusion approaches for downstream tasks and generative or contrastive transfer learning from natural-language supervision.Source: §2 · p. 2

Self-supervised Learning

  • Image SSL learns representations from raw pixels without label or text supervision, but its lack of language association limits zero-shot recognition; its contrastive paradigm also informs CLIP [48] and UniCL.Contrastive SSL aligns augmented views of the same image and separates views of different images; non-contrastive variants consider positive view pairs only.Source: §2 · pp. 2–3

3. Method

UniCL represents image-text and image-label data uniformly as image-text-label triplets, then learns image-language alignment with a bidirectional contrastive objective. The method connects this objective to established learning paradigms and supports joint or visual-backbone-only downstream use.

3.1. Preliminaries

  • UniCL encodes image-text pairs with unique labels and image-label pairs with their category names as text, unifying both as triplets for joint learning.The formulation aims to combine language semantics with the structured organization of labels.Source: §3.1 · p. 3

3.2. Unified Image-Text-Label Contrast

  • UniCL applies bidirectional image-to-text and text-to-image contrastive losses, assigning positive and negative pairs from labels and using temperature τ to control hard-negative penalties.Each direction aligns a query with its matched samples in the batch; the image-to-text loss is computed by row and the text-to-image loss by column.Source: §3.2 · p. 3

3.3. Discussions & Properties

  • Under specified conditions, UniCL’s text-to-image term recovers supervised cross-entropy, while its additional image-to-text term regularizes training and its text encoder can handle free-form descriptions.The stated conditions include a class embedding layer with bias, a batch sufficiently larger than the class count to use all class embeddings, temperature τ = 1, and no ℓ2 normalization.Source: §3.3 · pp. 3–4
  • Unlike SupCon, which contrasts image views using one shared image encoder, UniCL contrasts images with language descriptions encoded by separate modality-specific encoders.Both methods use samples with the same label as positives.Source: §3.3 · p. 4
  • With one-to-one image-text pairs, UniCL reduces to the CLIP training objective; with shared labels, it instead treats all same-label images as positives for a description.The same-label positive structure also applies when comparing the image-to-text and text-to-image terms with their CLIP counterparts.Source: §3.3 · p. 4

3.4. Model Training and Adaptation

  • During training, UniCL assigns image-text pairs label 0, image-label pairs labels 1 through K, makes each distinct batch description’s label unique, and learns temperature τ from an initial value of 1.The learned encoders support joint open-vocabulary recognition of seen or novel categories, while the visual backbone can be used alone for linear-probe feature extraction or full-model finetuning in object detection.Source: §3.4 · p. 4

4. Experiments

Experiments compare UniCL with supervised objectives and test how image-text and image-label data interact in a shared training space across recognition and detection tasks.

4.1. Results of UniCL on image classification

  • UniCL is comparable to or better than CE and SupCon across datasets and ResNet and Swin architectures, outperforming SupCon on CIFAR-10 and CIFAR-100 and matching it on ImageNet-1K.Against CE, it improves by around 1–3 points on small CIFAR datasets and by 3 points for Swin Transformer on ImageNet-1K, while remaining on par when overfitting is less severe.Source: §4.1 · pp. 5–6, Table 2 · p. 6
  • Transformer text encoders outperform linear label embeddings by 1.2%, removing one direction of UniCL’s bidirectional loss drops performance by 3%, and UniCL is robust to smaller batches.The authors link the encoder advantage to category-name semantics and batch robustness to recurring category-name embeddings and label guidance.Source: Table 3 · p. 6, §4.1 · p. 6, Table 4 · p. 6

4.2.1 Benefit of image-text to image-label

  • Adding image-text pairs generally improves all metrics; GCC-15M gains are +1.9 on ImageNet-1K, +3.5 on linear probe, and +1.2 on COCO detection.GCC-3M improves linear probe and COCO detection by 0.9 and 0.5, respectively; YFCC-14M improves ImageNet-1K and linear probe by 1.2 and 2.1. The authors attribute benefits to image-text concept richness and quality compensating for classification data.Source: Table 5 · p. 7, §4.2.1 · p. 7
  • On ImageNet-1K alone, UniCL is directly applicable to zero-shot recognition, and adding GCC-15M improves retrieval of semantically similar ImageNet-21K concepts.The text reports 30.2 for UniCL trained only on ImageNet-1K versus 36.3 for CLIP trained on YFCC-14M; the t-SNE comparison shows improved similarity of retrieved concepts with GCC-15M.Source: §4.2.1 · pp. 6–7, Table 5 · p. 7, Figure 4 · p. 7

4.2.2 Benefit of image-label to image-text

  • Mixing image-label and image-text data yields absolute zero-shot gains greater than 6% on ImageNet-1K and 7% across 14 datasets, plus a 7.3% linear-probe gain over CLIP.UniCL wins on 11 of 14 zero-shot datasets, and using the full sets of both datasets improves performance further.Source: §4.2.2 · p. 7, Table 6 · p. 8, Figure 5 · p. 8
  • With half of YFCC-14M and ImageNet-21K, UniCL beats the multi-task learner across tasks; with ImageNet-22K, gaps shrink for ImageNet-1K finetuning and linear probe but remain for zero-shot.The authors explain that ImageNet-22K covers all ImageNet-1K concepts and many linear-probe categories; unlike the multi-task learner, UniCL does not isolate image-label and image-text pairs.Source: §4.2.2 · pp. 7–8, Table 6 · p. 8
  • Image-label training groups fine-grained dog breeds in the visual feature space, unlike image-text-only training, even though ImageNet-21K contains none of those breed concepts.UniCL trained on ImageNet-21K and YFCC-14M is visualized against CLIP trained on YFCC-14M.Source: Figure 6 · p. 8, §4.2.2 · p. 8

5. Conclusion

UniCL combines image-text and image-label pairs in a shared image-text-label space to learn semantically rich, discriminative representations, improving recognition across zero-shot, linear-probe, and finetuning benchmarks. The paper also presents UniCL as an alternative learner for pure image-label data and recommends its public-dataset experiments as an academic baseline.

5. Conclusion

  • UniCL brings universal improvements on zero-shot, linear-probe, and finetuning benchmarks by unifying image-text and image-label pairs in a shared representation-learning paradigm.The method is built in image-text-label space and aims to learn discriminative, semantically rich representations.Source: §5 · p. 8
  • UniCL stand-alone is a good alternative learner on pure image-label data, and the paper discusses its connections to existing learning methods.Source: §5 · p. 8
  • The study focused on vision tasks such as image recognition and object detection and based its model on public datasets, recommending its experiments as an academic baseline.Source: §5 · p. 8
  • For large-scale pretraining and evaluation on broader tasks including VQA and video understanding, the paper refers readers to Florence [72], which used extensive private data.Source: §5 · p. 8

A. Validation dataset details · B. Experiment details

The appendix reports statistics for all validation datasets, including vocabulary sizes that are typically larger than the number of concepts or classes.

A. Validation dataset details

  • The validation-dataset statistics include vocabulary sizes, which are typically larger than the number of concepts or classes.The authors list statistics for all validation datasets in Table 7.Source: A. Validation dataset details · p. 12

B.1. Training on image classification data

This section specifies the image-classification training setup, including the visual and language encoders, optimization choices, and shared training schedule.

B.1. Training on image classification data

  • ResNet [23] and Swin Transformer [44] serve as visual encoders, paired with a 12-layer, 512-hidden-dimension Transformer [61] following [48].Globally pooled final-layer visual features are used, and two linear projection layers map visual and textual features to dimension 512.Source: §B.1 · p. 12
  • CNN models use SGD with learning rate 0.4 and weight decay 1e-4, while models with Transformers use AdamW [32] at 0.002 and 0.05.Source: §B.1 · p. 12
  • All models train for 500 epochs with batch size 4096 and cosine learning-rate scheduling, warming up for 5 epochs on ResNet and 20 on Swin Transformer.Augmentation and regularization follow [44], without MixUp [75] or CutMix [73] except in the last column of Table 2 in the main submission.Source: §B.1 · p. 12

B.2. Training on image-text-label space · C. More results

For Sec. 4.2.1, UniCL training balances image-label and image-text data through sampling; Sec. 4.2.2 follows CLIP’s training protocol for fair comparison.

B.2. Training on image-text-label space

  • Balanced sampling gives each iteration half image-label data and half image-text data, with 500 total epochs and roughly 40 effective epochs on GCC-12M.At each epoch, the sampler selects an image-text subset equal or similar in size to the image-label data; the authors report this strategy is important for the main submission’s performance.Source: §B.2 · p. 12
  • For Sec. 4.2.2, all models use random-crop augmentation and train for 32 epochs with batch size 4096, initial learning rate 1e-3, weight decay 0.1, and cosine scheduling with 5000 warmup iterations.The protocol follows CLIP for fair comparison.Source: §B.2 · p. 12

C.1. Results over separate datasets

UniCL benefits from the concept coverage of image-text pairs, outperforming the baseline on most of 14 zero-shot classification datasets, especially fine-grained tasks.

C.1. Results over separate datasets

  • On most of 14 datasets, UniCL outperforms the baseline, especially on fine-grained classification tasks such as Food101 and OxfordPets.The comparison adds different image-caption pairs to ImageNet-1K.Source: §C.1 · p. 12

C.2. Results with larger vision backbone … D.1. Concept distribution

UniCL benefits from larger vision backbones and combined image-text-label data, improving zero-shot recognition and object-detection transfer. The analyzed datasets also show long-tailed concept frequencies and relatively low overlap among their most common concepts.

C.2. Results with larger vision backbone

  • Increasing vision encoder size brings substantial gains in all cases, especially when both data types are combined.The results also show further significant gains when using the full ImageNet-21K and YFCC-14M sets.Source: §C.2 · p. 12, Table 8 · p. 13
  • Combining image-text and image-label data improves zero-shot recognition by 8.6 points on ImageNet-1K and 11.0 points on 14 datasets on average.With the full ImageNet-21K and YFCC-14M sets, both results improve further.Source: §C.2 · p. 12, Table 8 · p. 13

C.3. Transfer to object detection

  • Image-text pairs enrich and smooth the semantic space, while image-label data directly pressures models to learn more discriminative representations.The paper suggests these effects may promote distinctive representations for COCO detection concepts.Source: §C.3 · p. 13
  • Representations trained on pure image-text pairs usually lack the discriminative ability needed for transfer to image recognition and object detection.Together with the reported Table 5 results, the paper states that adding either data type to the other can improve representations over either alone.Source: §C.3 · p. 13
  • Around 2 points of box and mask mAP improvement comes from combining data types versus YFCC-14M-only training; the full set adds around 1 point on both metrics.The transfer experiment uses Swin-Tiny backbones and a Mask R-CNN model trained with a 1× schedule.Source: §C.3 · p. 13, Table 9 · p. 14

D.1. Concept distribution

  • The top 10,000 concepts overlap less across GCC-3M, GCC-12M, and YFCC-14M than expected, indicating dataset diversity and complementarity.The overlap ratios are summarized in Table 10.Source: Table 10 · p. 14, §D.1 · p. 14
  • Concept frequencies in GCC-3M, GCC-12M, and YFCC-14M are extremely long-tailed: “view” appears over 185,363 times in GCC-12M, while “candle holder” appears 501 times at rank 10,000.The full set contains more than 584k concepts.Source: §D.1 · p. 14

D.2. Concept coverage

Concept coverage partly explains zero-shot recognition, but good performance despite limited or absent overlap shows that coverage alone does not account for the model’s results. Image-text pairs provide broader concept coverage than image-label data, yet greater coverage does not guarantee better performance.

D.2. Concept coverage

  • ImageNet-1K overlaps with CIFAR-100 by 24.0% and Caltech-101 by 24.5%, which may help explain its relatively higher zero-shot performance on those datasets.It also achieves reasonably good zero-shot performance on datasets such as CIFAR-10, Flowers102, and Oxford Pet despite less or no coverage.Source: §D.2 · p. 14
  • ImageNet-21K covers some validation concepts, but reasonable performance on Hateful Memes and PatchCamelyon despite zero overlap suggests UniCL learns beyond memorized training concepts.The authors attribute this to learning underlying structures of different concepts.Source: §D.2 · p. 14
  • Image-text pairs have higher concept coverage than image-label data on almost all validation sets, with GCC-12M highest among the three, but full CIFAR coverage does not ensure better zero-shot performance.Adding the image-text datasets hurts CIFAR-10 and CIFAR-100 performance; the authors suggest domain gaps and difficulty grounding concepts to specific image contents as possible reasons.Source: §D.2 · p. 14

D.3. Concept visualizations

Visualizations show that joint ImageNet-1K and GCC-15M training yields concept embeddings that better capture visually grounded semantic relationships than ImageNet-1K-only training.

D.3. Concept visualizations

  • The joint-data model places “porcuponefish” near “porthole” and “liqueur glass” near “goblet,” avoiding the mismatches made by the ImageNet-1K-only model.The single-data model confuses “porthole” with porcupine and “goblet” with “coverlet,” while the joint-data model identifies semantically and visually similar neighbors.Source: §D.3 · p. 14
  • Joint ImageNet-1K and GCC-15M training produces more semantically meaningful concept embeddings than ImageNet-1K-only training, as illustrated by the t-SNE visualization.The authors conclude that the combined-data model learns visually grounded semantic meanings for varied concepts.Source: §D.3 · p. 14, Figure 9 · p. 15
  • A second random concept set shows a similar trend, with the combined image-label and image-text model understanding more visual concepts.The paper reports this additional comparison in Figure 10.Source: §D.3 · p. 14, Figure 10 · p. 16

Related papers

Loading 2204.03610v1…