PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Vision from Models Rivals Learning Vision from Data

Yonglong Tian, Lijie Fan, Kaifeng Chen, Dina Katabi, Dilip Krishnan, Phillip Isola

arXiv:2312.17742v1cs.CV

TL;DR

Large-scale curated real-image datasets are costly, motivating the question of whether generative models can supply training data for visual representations. SynCLR trains on synthetic captions and images with caption-based contrastive learning, and reports transfer comparable to general-purpose representation learners, including gains on dense prediction.

  • Problem

    Large-scale curated real-image datasets are costly, and the paper asks whether synthetic data from generative models can train state-of-the-art visual representations.

    Source: §1 · p. 1

  • Method

    SynCLR generates multiple images per synthetic caption and aligns same-caption images with multi-positive contrastive learning, alongside masked image modeling.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    SynCLR transfers well across classification and dense prediction, exceeding ImageNet-pretrained MAE by 6.2 mIoU on ADE20k with ViT-B.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Without real training data, SynCLR learns representations comparable to those of state-of-the-art general-purpose visual representation learners.

    Source: §5 · p. 10

Abstract

from arXiv · show

We introduce SynCLR, a novel approach for learning visual representations exclusively from synthetic images and synthetic captions, without any real data. We synthesize a large dataset of image captions using LLMs, then use an off-the-shelf text-to-image model to generate multiple images corresponding to each synthetic caption. We perform visual representation learning on these synthetic images via contrastive learning, treating images sharing the same caption as positive pairs. The resulting representations transfer well to many downstream tasks, competing favorably with other general-purpose visual representation learners such as CLIP and DINO v2 in image classification tasks. Furthermore, in dense prediction tasks such as semantic segmentation, SynCLR outperforms previous self-supervised methods by a significant margin, e.g., improving over MAE and iBOT by 6.2 and 4.3 mIoU on ADE20k for ViT-B/16.

1. Introduction

Collecting large-scale real datasets is costly, while uncurated data scales poorly and curated datasets are narrow. SynCLR learns from synthetic captions and images, using caption-defined visual classes to produce representations that transfer across tasks.

1. Introduction

  • Large-scale uncurated real data scales poorly for self-supervised learning, curated datasets are narrow, and large-scale curated real-image datasets are costly to pursue.The paper asks whether synthetic data sampled from off-the-shelf generative models can provide a viable path to large-scale curated training data.Source: §1 · p. 1
  • SynCLR learns from synthetic text and synthetic images without real training data, matching CLIP’s ImageNet linear transfer performance.The paper frames this as “learning from models,” contrasting it with learning directly from data.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • SynCLR treats each synthetic caption as a visual class, grouping images by finer concepts such as actions and relationships than a coarse subject label.The authors report that caption-level granularity outperforms SimCLR and supervised training, and can scale to more classes by augmenting data online.Source: §1 · p. 2
  • The method generates multiple images for each synthetic caption, yielding 600M images, then combines multi-positive contrastive learning with masked image modeling.Images generated from the same caption are aligned in the embedding space.Source: §1 · p. 2
  • 80.7% (ViT-B) and 83.0% (ViT-L) ImageNet-1K top-1 linear accuracy are on par with CLIP; SynCLR exceeds MAE by 6.2 and 4.1 mIoU on ADE20k.On fine-grained classification, it exceeds CLIP by 3.3% (ViT-B) and 1.5% (ViT-L), and performs similarly to DINO v2.Source: §1 · p. 2

2. Related Works

Related work spans self-supervised and text-supervised representation learning, generative models as representation learners, and synthetic-data training. SynCLR shares elements with these lines while distinguishing itself through text-to-image-generated data and synthetic-only representation learning.

2. Related Works

  • Self-supervised work includes domain-specific pre-text tasks, contrastive learning, and masked image modeling; text supervision offers flexible, richer supervision for generalizable representations.The passage notes limits of supervised transfer, including dense prediction settings.Source: §2 · pp. 2–3
  • Generative models can learn useful representations for dense tasks, while diffusion-based zero-shot recognition approaches may need adaptation as model architectures or families change.SynCLR instead treats images as universal interfaces in pursuit of broader generality.Source: §2 · p. 3
  • Prior synthetic-data work spans computer-vision tasks and representation learning; the closest work, StableRep, also learns representations but requires a real text dataset.SynCLR diverges by using text-to-image models for representation learning from synthetic data.Source: §2 · p. 3

3. Approach

SynCLR learns a visual encoder without real images or text by generating captions from curated concepts, synthesizing multiple images per caption, and training on the resulting synthetic image set.

3. Approach

  • The approach uses a language model to synthesize captions, a text-to-image model to generate images for them, and synthetic images to train a visual encoder.The target setting excludes real images and textual data.Source: §3 · p. 3

3.1. Synthesizing captions

  • Caption synthesis uses concept-only, concept-plus-background, and concept-plus-relation templates, with GPT-4-suggested backgrounds replacing implausible random pairings.For each inference, three demonstrations are randomly sampled from the selected template; the examples total 176.Source: §3.1 · pp. 3–4, Table 1 · p. 4

3.2. Synthesizing Images

  • Each caption produces four images using different reverse-diffusion noise; CFG 2.5 favors diversity and adherence to the conditional image distribution.The authors choose a lower CFG scale, noting that higher scales improve sample quality and text-image alignment while reducing diversity.Source: §3.2 · p. 4

3.3. Representation Learning

  • SynCLR builds on StableRep, aligning embeddings of same-caption images with multi-positive contrastive learning and adding patch-level masked image modeling.The contrastive assignment allows an anchor image to match multiple candidates from the same caption.Source: §3.3 · pp. 4–5
  • iBOT masks localized patches to predict tokenized representations; EMA encodes contrastive candidates and supplies iBOT targets, improving performance and long-schedule stability.The EMA decay follows a cosine schedule from 0.994 to 1.Source: §3.3 · p. 5
  • Local crops receive contrastive loss only and match same-caption global crops encoded by EMA; reusing global crops saves computation.The global crop receives both contrastive and iBOT losses.Source: §3.3 · p. 5

3.4. Implementation

  • Training batches sample 2,048 captions and all four images per caption, yielding 8,192 global crops; half the images and half the tokens in each selected image are masked.The method uses 65,536 prototypes; the contrastive temperature is 0.08, while the iBOT temperature ramps from 0.04 to 0.07 over 4,000 iterations.Source: §3.4 · pp. 5–6

4. Experiment

SynCLR’s ablations identify effective caption and representation-learning choices, and its scaled synthetic-data training yields strong transfer across classification, segmentation, and fine-tuning. Further analyses test generalization beyond the concept list and examine performance across training scales.

4.1. Study different components

  • GPT-generated backgrounds improve accuracy over random Places backgrounds, and the full caption-synthesis configuration achieves the best ImageNet and fine-grained accuracy.The study reports that this synthesis method scales to hundreds of millions of captions with little duplication, unlike IN-plus-Places concatenation, which has at most 365k unique captions.Source: §4.1 · p. 6
  • Adding teacher EMA, iBOT, and multi-crop improves ImageNet linear accuracy over StableRep by 0.9%, 0.9%, and 1.9%, respectively; the full model reaches 78.8% top-1 accuracy.The full model reaches 88.1% fine-grained accuracy, and iBOT adds 1.0 mIoU over multi-crop on ADE20k.Source: Table 4 · p. 6, §4.1 · p. 7
  • SynCLR’s multi-positive objective outperforms supervised cross-entropy and SimCLR by 3.4% and 11.7% on ImageNet linear accuracy, and by 3.5% and 10.6% on fine-grained accuracy.The objective defines images generated from the same caption as the same class and does not require balanced samples from fixed classes.Source: Table 5 · p. 6, §4.1 · p. 7

4.2. Scaling up

  • 80.7% and 83.0% ImageNet-1K top-1 linear accuracy for ViT-B and ViT-L are similar to CLIP, while fine-grained accuracy is similar to DINO v2.DINO v2 models are distilled from ViT-g, an advantage in this comparison.Source: §4.2 · p. 7, Table 6 · p. 7
  • On ADE20k semantic segmentation, SynCLR exceeded iBOT by 4.3 mIoU after fine-tuning, with UperNet as the task layer and evaluation at a single 512x512 scale.SynCLR was 0.1 lower than DINO v2 for ViT-B and 0.2 higher for ViT-L.Source: §4.2 · p. 8, Table 7 · p. 8
  • SynCLR outperforms models trained on ImageNet or large-scale image datasets in ImageNet fine-tuning, including OpenCLIP ViT-L trained on Laion-2B.The authors conclude that synthetic-image pretraining can yield representations adaptable to downstream tasks with limited real data.Source: §4.2 · p. 8, Table 8 · p. 8

4.3. Further analysis

  • On datasets outside SynCLR’s concept list, SynCLR exceeds DINO v2 by 1.5% for ViT-B and 1.1% for ViT-L, while CLIP achieves the best accuracy.The authors attribute most of CLIP’s gains to Country211, whose similar country flags appear in CLIP’s training data.Source: §4.3 · p. 8, Table 9 · p. 8
  • ViT-B performs robustly at 10M captions with diminishing returns beyond, while ViT-L needs more data and scales better; PCA maps show stronger car and airplane features but slightly worse dog heads than DINO v2.The training-scale results are shown in Figures 6 and 7; the PCA comparison uses ViT-L/14.Source: §4.3 · p. 9, Figure 6 · p. 9, Figure 7 · p. 9

5. Discussions and Conclusion

SynCLR learns visual representations without real data and achieves representations comparable to state-of-the-art general-purpose learners. The authors describe generative models as a flexible source of diverse training images and identify several improvements for future exploration.

5. Discussions and Conclusion

  • Generative models can supply many diverse datasets through textual controls, avoiding the separate effort of collecting datasets for different image categories.The authors compare this flexibility with DINO v2 [68], which curates and combines numerous datasets.Source: §5 · p. 10
  • Richer captions, larger-model distillation, high-resolution training, IN-21k fine-tuning, and architectural changes are proposed for future exploration, constrained by resources and paper scope.The authors state that the paper does not focus on achieving the highest possible metrics.Source: §5 · p. 10
  • Without using real data, SynCLR learns visual representations comparable to those of state-of-the-art general-purpose visual representation learners.The authors describe generative models as able to produce many samples and allow control through textual input, while noting that diversity is finite.Source: §5 · p. 10

A. Concept Sampling

Caption concepts are randomly sampled from names of various datasets, with rough sampling probabilities summarized in Table 11. The authors do not optimize the ratios and warn that IN-21k-heavy sampling may under-sample other datasets.

A. Concept Sampling

  • The concept sampling probabilities are rough, and the authors do not optimize the ratios because different combinations may lead to different results.Table 11 presents the rough concept sampling probabilities.Source: A. Concept Sampling · p. 14, Table 11 · p. 14
  • Caption concepts are randomly sampled from names of various datasets, with IN-21k concepts concatenated with classes from datasets such as Caltech-101 and Pets.The authors use uniform sampling from the concatenated list.Source: A. Concept Sampling · p. 14
  • Because the concatenated list is dominated by IN-21k classes, uniform sampling may under-sample concepts from other datasets.Source: A. Concept Sampling · p. 14

B.1. Pre-training

SynCLR uses different pre-training schedules for final training, ablations, and scaling plots. The final schedule trains for 500k steps with a batch size of 8192 captions.

B.1. Pre-training

  • Final long-schedule pre-training runs for 500k steps with a batch size of 8192 captions.Table 12 summarizes the SynCLR pre-training settings.Source: §B.1 · p. 14, Table 12 · p. 14
  • Ablation models train for 85k steps with a batch size of 2048 captions.Source: §B.1 · p. 14
  • Models used for scaling plots train for 300k steps with a batch size of 2048 captions.Source: §B.1 · p. 14

B.2. ImageNet linear probing

ImageNet linear probing uses the final transformer block’s cls token as the image representation, unlike DINO v2’s representation choice. The classifier uses zero weight decay and a sweep of base learning rates.

B.2. ImageNet linear probing

  • SynCLR uses the final transformer block’s cls token as the image representation, unlike DINO v2, which combines cls and average-pooled patch tokens and sweeps over multiple layers.Source: §B.2 · p. 14
  • The linear classifier uses zero weight decay and sweeps base_lr over {0.1, 0.2, 0.5, 1, 2, 5, 10, 20, 50} × 10−2.The authors follow prior work [11] [14] and cite observations that weight-decay regularization hurts performance [43] [88].Source: §B.2 · p. 14
  • Table 13 summarizes the ImageNet linear probing settings.Source: Table 13 · p. 14

B.3. End-to-End ImageNet fine-tuning

The ImageNet end-to-end fine-tuning setup adds a linear classifier to the final block’s CLS token and fine-tunes the network, using layer-wise learning-rate decay.

B.3. End-to-End ImageNet fine-tuning

  • A linear classifier is appended to the last transformer block’s CLS token, and the whole network is fine-tuned.The setup follows common practice [5] [44].Source: §B.3 · p. 14
  • The fine-tuning uses layer-wise learning-rate decay [20].The settings are listed in Table 14.Source: §B.3 · p. 14, Table 14 · p. 14

B.4. Semantic segmentation on ADE20k

The ADE20k semantic-segmentation setup uses UperNet with a single-scale evaluation configuration; the fine-grained linear-classification setup trains a regularized logistic-regression model on CLS features.

B.4. Semantic segmentation on ADE20k

  • ADE20k semantic segmentation uses UperNet as the task adaptation layer and a common single-scale setup.The experiments follow [5] [44] and use settings summarized in Table 15.Source: §B.4 · p. 14, Table 15 · p. 15
  • 512×512 inputs are used for patch size 16×16, while patch size 14×14 uses 518×518 inputs.Both resolutions follow the single-scale setup.Source: §B.4 · pp. 14–15

B.5. Fine-grained linear classification

  • Fine-grained linear classification trains ℓ2-regularized multinomial logistic regression on the output CLS token without data augmentation.Images are resized to 224 pixels on the shorter side and center-cropped to 224×224; the regularization constant is selected on validation data from 45 logarithmically spaced values between 10−6 and 10^5.Source: §B.5 · p. 15

C. In-context Learning Examples

The section provides in-context caption examples for templates that pair concepts with backgrounds or relations, spanning varied objects, animals, places, textures, settings, and spatial arrangements.

C. In-context Learning Examples

  • Template 1 maps a concept c to a caption, with examples describing objects, animals, scenes, and textures.Examples include a station wagon beneath an oak tree and a textured surface described as crystalline.Source: Table 16 · p. 16, C. In-context Learning Examples · pp. 16, 18
  • Other Template 1 examples turn concepts such as a heliport, aquarium, and formal garden into descriptive captions.The examples situate concepts in detailed scenes, such as helicopters taking off against a city skyline and fish among coral formations.Source: C. In-context Learning Examples · pp. 17–18
  • Template 2 maps a concept c and background bg to a caption, pairing items such as a can opener with a hotel room.Additional examples pair a butterfly with a pond side and a television with a mountain lodge.Source: Table 17 · p. 19, C. In-context Learning Examples · p. 19
  • Template 3 represents each example as a concept and relation paired with a generated caption.The table defines the format as “c,rel → caption,” with c as the concept and rel as the relation.Source: Table 18 · p. 20
  • Examples use spatial relations such as “next to” and “below” to place objects within described scenes.A waffle iron is placed next to a jar of syrup, while a Komodo dragon sprawls below a child on a bridge.Source: Table 18 · p. 20, C. In-context Learning Examples · p. 21
  • Captions expand brief concept-relation pairs into detailed scenes with settings, actions, and descriptive attributes.Examples include foxes gathered in front of a forest and a centipede crawling above fern leaves.Source: Table 18 · p. 20, C. In-context Learning Examples · p. 21
Loading 2312.17742v1…