DINOv2: Learning Robust Visual Features without Supervision

Motivation and approach

• Text-guided pretraining can limit retained image information because captions approximate rich visual content, require aligned text-image corpora, and may miss complex pixel-level information.Self-supervised learning instead trains from images alone and can capture information at both image and pixel levels.Source: §1 · p. 2
• DINOv2 explores whether self-supervised learning on large quantities of curated data can produce general-purpose visual features, while improving scalability through faster and more memory-efficient training.The approach is approximately 2× faster and requires 3× less memory than similar discriminative self-supervised methods, enabling longer training with larger batch sizes.Source: §1 · p. 2

Data

• An automatic pipeline filters and rebalances uncurated images using data similarities rather than external metadata or manual annotation, producing a diverse corpus of 142M images.Naive clustering is used to rebalance concepts and reduce overfitting to dominant modes.Source: §1 · p. 2

Models and validation

• Figure 2 shows that scaling DINOv2 substantially improves over prior self-supervised features and reaches performance comparable with the best weakly-supervised features across eight vision-task types.The figure reports average metrics for image- and pixel-level task categories as model parameters increase.Source: Figure 2 · p. 3, §1 · p. 3
• DINOv2 provides pretrained Vision Transformer models trained on the curated data, releases the models and retraining code, and validates them on image- and pixel-level benchmarks.The models are evaluated as frozen transferable features across varied computer-vision tasks.Source: §1 · p. 3

Intra-image self-supervised training

• Intra-image methods use pretext signals such as context prediction, colorization, transformations, inpainting, and patch re-ordering, but their features require supervised finetuning.Masked autoencoders substantially improve downstream finetuning and have been extended across video, audio, and other modalities.Source: §2 · p. 3

Discriminative self-supervised learning

• Discriminative methods learn from relationships between images or groups through instance-level objectives and clustering, producing performant frozen features but scaling poorly to larger models.DINOv2 revisits these approaches for large datasets and models, building particularly on Zhou et al. (2022a).Source: §2 · pp. 3–4

Scaling self-supervised pretraining

• Scaling studies show self-supervised methods benefit from more data and model capacity, yet typically rely on uncurated data and supervised finetuning rather than producing the best pretrained encoders directly.Goyal et al. (2021) showed model-size scaling benefits given enough pretrained data.Source: §2 · p. 4

Automatic data curation

• Prior curation methods use retrieval, metadata, hashtags, or pretrained encoders, whereas DINOv2 filters images without pretrained encoders, metadata, or supervision using visual similarity.Its pipeline maps images to embeddings, deduplicates uncurated images, matches them to curated images, and augments the dataset through self-supervised retrieval.Source: §2 · p. 4, Figure 3 · p. 4

Data sources

• The image-only pipeline combines curated sources including ImageNet-22k, ImageNet-1k training data, Google Landmarks, and fine-grained datasets without requiring metadata or text.The curated dataset is assembled by retrieving images from the uncurated source that resemble images in these curated datasets.Source: §3 · p. 4
• The uncurated source contains 1.2B unique images collected from crawled web pages after URL filtering, PCA-hash deduplication, NSFW filtering, and face blurring.Images are extracted from <img> tags in publicly available crawled web data.Source: §3 · p. 5

Deduplication

• Near-duplicate removal reduces redundancy and increases diversity, including duplicates of images in benchmark test and validation sets.The method applies the copy detection pipeline of Pizzi et al. (2022).Source: §3 · p. 5

Implementation Details

• Faiss enables GPU-accelerated nearest-neighbor indexing with inverted file indices and product quantization, producing LVD-142M in under two days on 20 nodes with 8 V100-32GB GPUs.The processing stages use batch searches of nearest embeddings and product quantization codes.Source: §3 · p. 5

• The discriminative self-supervised method combines DINO and iBOT losses with SwAV centering, a feature-spreading regularizer, and a short high-resolution training phase.These components are introduced as the core pretraining design.Source: §4 · p. 5
• The image-level objective matches student and teacher class-token prototype distributions from different crops using DINO cross-entropy, with teacher centering and exponential-moving-average updates.The student and teacher pass class tokens through separate DINO heads before softmax normalization; the teacher head uses an exponential moving average (He et al., 2020).Source: §4 · p. 5
• The patch-level objective masks student input patches and predicts corresponding teacher-visible patch-token targets with the iBOT loss and exponentially averaged teacher head.Softmax and centering are applied to the student and teacher iBOT outputs, with masked patch indices defining the loss (Zhou et al., 2022a).Source: §4 · pp. 5–6
• At scale, the method unties the DINO and iBOT MLP projection heads, contrary to the parameter-sharing preference reported by Zhou et al. (2022a).The authors use two separate learnable heads in all experiments.Source: §4 · p. 6
• Teacher outputs use three Sinkhorn-Knopp iterations for centering, while student outputs use softmax normalization, following the SwAV-based recommendation of Ruan et al. (2023).This replaces the teacher softmax-centering step of DINO and iBOT with SwAV batch normalization (Caron et al., 2020).Source: §4 · p. 6
• The KoLeo regularizer encourages uniformly spread, ℓ2-normalized features within each batch using nearest-neighbor distances, while resolution increases to 518×518 late in pretraining.The high-resolution phase supports pixel-level tasks such as segmentation and detection while limiting training cost and memory demands (Touvron et al., 2019).Source: §4 · p. 6

Overall implementation

• The DINOv2 code runs around 2× faster than the iBOT implementation while using only one-third as much memory on the same hardware.Training uses A100 GPUs with PyTorch 2.0, and the code and pretrained models are released under the Apache 2.0 license.Source: §5 · p. 6

Fast and memory-efficient attention

• The custom FlashAttention implementation matches or exceeds the original across tested cases while supporting more use-cases and hardware.Hardware-aligned dimensions motivate ViT-g’s 1536-dimensional embedding with 24 heads instead of 1408 dimensions with 16 heads, without significant final-accuracy differences.Source: §5 · p. 6

Sequence packing

• Sequence packing concatenates large- and small-crop token sequences and uses a block-diagonal attention mask, preserving separate-sequence equivalence while improving compute efficiency.This avoids separate forward and backward passes for the different sequence lengths.Source: §5 · pp. 6–7

Efficient stochastic depth

• Efficient stochastic depth skips dropped residual computations, saving memory and compute approximately in proportion to the drop rate.At the 40% drop rate used here, fused kernels provide a drastic efficiency improvement by computing only (1 − d) × B shuffled batch samples.Source: §5 · p. 7

Fully-Sharded Data Parallel (FSDP)

• FSDP shards four float32 model replicas across GPUs, removes the single-GPU memory bound, and reduces communication costs by approximately 50% versus DDP.Backbone weights are broadcast and gradients reduced in float16, while MLP-head gradients remain float32 for stability.Source: §5 · p. 7

Model distillation

• Distillation from the 1.1B-parameter ViT-g yields better performance than training from scratch, even for ViT-L models.The method uses a frozen teacher, an EMA student as the final model, no masking or stochastic depth, and iBOT loss on two global crops.Source: §5 · p. 7

6.1 Improved Training Recipe

• Incrementally adding the improved training components generally raises k-NN or linear-probe accuracy, while LayerScale and high Stochastic Depth reduce linear-probe performance but improve training stability.The stability benefit includes avoiding NaN loss values during training.Source: §6.1 · pp. 7–8, Table 1 · p. 8

6.2 Pretraining Data Source

• Curated LVD-142M features outperform uncurated data on most benchmarks and ImageNet-22k on all but ImageNet-1k, especially improving unseen domains.The results support data curation, scale, and diversity as complementary benefits for general-purpose features.Source: §6.2 · pp. 8–9, Table 2 · p. 8

6.3 Model Size and Data

• As model size increases, LVD-142M becomes more beneficial than ImageNet-22k; a ViT-g matches ImageNet-1k performance while outperforming the baseline on other benchmarks.Figure 4 directly compares performance across model sizes and the two pretraining datasets.Source: §6.3 · p. 9, Figure 4 · p. 9

6.4 Loss Components

• The KoLeo loss improves instance retrieval by more than 8% without hurting other metrics, while masked image modeling improves dense prediction by almost 3%.These effects correspond to KoLeo spreading features in the output space and MIM benefiting patch-level tasks.Source: §6.4 · p. 9, Table 3 · p. 9

6.5 Impact of Knowledge Distillation

• A ViT-L/14 distilled from a ViT-g/14 outperforms the same architecture trained from scratch on all 12 evaluated benchmarks.The ViT-g/14 teacher is also reported as a topline, and the distilled model sometimes exceeds it.Source: §6.5 · p. 9, Figure 5 · p. 10

6.6 Impact of Resolution

• High-resolution training throughout performs best, but resuming 224×224 training at 416×416 for 10k iterations achieves nearly comparable results at a fraction of the compute.Training entirely at 416×416 costs approximately 3× more compute than training at 224×224.Source: §6.6 · p. 10, Figure 6 · p. 10

7 Results

• Across the evaluation suite, frozen DINOv2 representations deliver strong performance across global classification, fine-grained and video recognition, instance retrieval, segmentation, and depth estimation.The comparisons use publicly available self-supervised baselines and open-source weakly supervised models, including OpenCLIP and SWAG.Source: §7 · p. 11

7.1 ImageNet Classification

• DINOv2 improves linear ImageNet-1k accuracy by +4.2% over iBOT ViT-L/16 and surpasses OpenCLIP-G/14 by +0.3% and EVA-CLIP-g/14 by +0.1%.ImageNet-V2 performance is also +1.1% over EVA-CLIP, indicating stronger generalization.Source: §7.1 · p. 11, Table 4 · p. 12

7.2 Additional Image and Video classification Benchmarks

• On downstream classification, DINOv2 beats OpenCLIP on iNaturalist 2018 and 2021 by +8.6% and +9.7%, matches it on UCF101 and Kinetics, and leads on Something-Something v2 by +2.5%.It lags OpenCLIP on Places205 by −2.3%, while exceeding SSL baselines on fine-grained benchmarks, including +14.8% on Stanford Cars and FGVC Aircraft.Source: §7.2 · pp. 13–14, Table 7 · p. 13

7.3 Instance Recognition

• DINOv2 features significantly outperform SSL and weakly supervised methods for instance recognition, achieving +41% mAP over SSL and +34% over weak supervision on Oxford-Hard.The evaluation covers Paris, Oxford, Met, and AmsterTime using cosine-similarity retrieval.Source: §7.3 · p. 14, Table 9 · p. 14

7.4 Dense Recognition Tasks

• Frozen DINOv2 features surpass the best SSL and weakly supervised depth features, while a frozen-backbone DPT decoder matches or exceeds recent state-of-the-art performance and transfers from NYUd to SUN-RGBd.For segmentation, the boosted recipe reaches 86.2 mIoU on Pascal VOC, and the frozen-backbone ViT-Adapter setup reaches 60.2 mIoU on ADE20K versus 62.9 state of the art.Source: §7.4 · pp. 15–16, Table 10 · p. 15, Table 11 · p. 16

7.5 Qualitative Results

• Qualitative evaluations show smoother DINOv2 segmentation and depth predictions than OpenCLIP, strong out-of-distribution transfer, and PCA components that delineate objects and align their parts across related images.Patch matching also captures semantically corresponding regions across different objects, styles, and poses, such as plane and bird wings.Source: §7.5 · pp. 16–18, Figure 7 · p. 17, Figure 8 · p. 17

8.1 Geographical Fairness

• DINOv2 is slightly fairer than SEERv2 across regions and incomes, and significantly outperforms the supervised baseline, but its performance drops 25.7% in Africa versus Europe and differs 31.7% between high- and low-income households.The evaluation uses 16,073 Dollar Street images from 289 households across 54 countries and 94 concepts.Source: §8.1 · pp. 18–19, Table 12 · p. 19

8.2 Gender, Skintones and Age

• The label-association evaluation trains a linear classifier on 619 ImageNet-22k classes, keeps the backbone frozen, and evaluates top-5 labels on 2,955 Casual Conversations images.Labels are grouped into Human, Possibly Human, Non-Human, and Crime; Non-Human and Crime are treated as harmful, and multiple labels may be assigned per image.Source: §8.2 · p. 19, Table 13 · p. 20
• DINOv2 usually classifies people across groups as Human without large skin-tone deviations, predicts almost no harmful Non-Human or Crime labels, and shows no clear bias against a particular group.The only two harmful-label instances involved backgrounds containing bars visually similar to prison bars.Source: §8.2 · p. 20
• DINOv2 frequently predicts Possibly-Human classes, particularly for men because of the prevalence of the Beard class, and the authors caution that broader bias evaluations may reveal flaws.Possibly-Human includes objects such as Scarf, Glasses, and Beard that are often related to humans.Source: §8.2 · p. 20

Estimation methodology

• Using fixed US-average assumptions, the methodology estimates potential energy use and carbon emissions for retraining a similar model in an average data center.The analysis sets PUE to 1.1, carbon intensity to 0.385 kg CO2eq/KWh, and A100-80GB power to 400W, following Patterson et al. (2021) and the setup of Touvron et al. (2023).Source: §9 · p. 20

Model retraining comparison

• Retraining OpenCLIP ViT-L and ViT-G would require 22.4 MWh and 118.9 MWh, respectively, and is reported as 10× more carbon-intensive than retraining DINOv2 ViT-g.The comparison uses the same data center but is not fully fair because OpenCLIP also trains a text encoder; for visual-only features, self-supervised training is preferable in carbon emissions.Source: §9 · p. 21

Carbon footprint of the whole project

• The whole project emitted an estimated 0.5k–1k tCO2eq, corresponding to roughly 200k GPU-days, with self-supervised pretraining as the primary emissions source.A ViT-g pretraining consumed 22k GPU-hours and emitted 3.7 tons of CO2eq, versus 0.2 tons for a 1k-GPU-hour ImageNet-1k finetuning; GPU manufacturing and disposal were excluded.Source: §9 · p. 21

Discussion

• DINOv2 closes the performance gap with weakly supervised alternatives across diverse benchmarks without finetuning.The models are pretrained on large curated datasets with no supervision.Source: §10 · p. 21
• Its strong performance stems from improved training, larger models and datasets, and distillation from the strongest ViT-g model.The paper associates these factors with results in Tables 1 and Figures 4–5.Source: §10 · p. 21

Future work

• The models capture object parts and scene geometry across image domains, with more such properties expected at larger model and data scales.The authors plan to continue scaling both axes.Source: §10 · p. 21
• Because the features work with simple linear classifiers, the authors plan a language-enabled AI system that treats visual features like word tokens for grounding.This compatibility suggests the needed information is readily available in the representations.Source: §10 · p. 21

A.1 Data selection

• The selected datasets for LVD-142M are intended to cover downstream vision tasks across both image-level and dense recognition.The dataset selection is detailed in Tab. 15.Source: §A.1 · p. 29

A Data Processing

• The pipeline combines curated dataset selection, feature-based similarity comparison, deduplication, and retrieval to construct a diverse training collection while controlling redundancy.The supplied passages specify these data-processing operations but provide no separate implementation details for section B.Source: §A.1 · p. 29, §A.2 · p. 29, §A.3 · p. 29

A.2 Image similarity

• Image similarity compares feature representations with cosine similarity, m(s, r) = cosine-similarity (f(s), f(r)) = f(s) · f(r) / (∥f(s)∥2∥f(r)∥2).Here, s and r are the image pair being compared, and f is the feature-generating model.Source: §A.2 · p. 29

A.3 Deduplication

• 1.3B images are reduced to 1.1B by self-deduplicating connected components whose k = 64 nearest neighbors exceed similarity >0.6, retaining one representative per duplicate component.Embeddings from Pizzi et al. (2022) define the nearest-neighbor graph, processed with a scalable disjoint set data structure.Source: §A.3 · p. 29
• 744M images remain after removing self-deduplicated images too similar to evaluation train and test splits, using duplicate components identified with a stricter similarity >0.45.The relative-deduplication procedure discards entire components containing matching reference images to reduce redundancy and support proper evaluation.Source: §A.3 · p. 29

A.4 Retrieval

• Retrieval augments datasets either by collecting k nearest images per sample or by selecting images from 100,000 content clusters, with retrieval capped at 1M images for balance.Sample-based retrieval uses k = 4 for Google Landmarks v2 and ImageNet-22k, and k = 32 for LVD-142M; cluster-based retrieval selects 10,000 images from qualifying clusters.Source: §A.4 · p. 29

B.1 Unsupervised pre-training

• Unsupervised pre-training builds on the DINO and iBOT codebases using the hyperparameters and ViT architectures specified for each model.Source: §B.1 · p. 29
• KoLeo regularization is applied between first-global-crop class tokens within each GPU, with weight 0.1 and no cross-GPU communication.Source: §B.1 · p. 29
• Training runs for 625k iterations with AdamW, LayerScale initialization, cosine weight-decay and teacher-momentum schedules, learning-rate warmup, and float16 precision except for DINO-head gradient reduction.Source: Table 16 · p. 31
• The teacher starts from the student state and is updated after every step as an exponential moving average with cosine-scheduled momentum from 0.994 to 1.0.Source: §B.1 · p. 31

B.2 High-Resolution adaptation

• High-resolution adaptation initializes from pretrained weights and compresses the original training procedure and schedules into 10k iterations while reducing only the base learning rate.Source: §B.2 · p. 31

B.3 Linear probing evaluation

• Linear probing trains an SGD classifier for 12500 iterations with random-resized-crop augmentation while grid-searching learning rate, output-layer count, and token-feature concatenation.The concatenation choice determines whether average-pooled patch tokens are combined with the class token or the class token is used alone; output layers are selected from {1, 4}.Source: §B.3 · p. 31
• The reported probing result is the highest validation accuracy, and the search remains inexpensive because the backbone runs once per iteration before classifier-specific matrix multiplications.Source: §B.3 · p. 31

C List of Datasets used

• Table 18 lists the benchmarks and datasets used and describes their purposes.The table provides the study’s dataset inventory.Source: C List of Datasets used · p. 31, Table 18 · p. 32