PaperlayerView exact revision on arXiv

Source-linked AI summary

DINOv2: Learning Robust Visual Features without Supervision

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, Piotr Bojanowski

arXiv:2304.07193v1cs.CV

TL;DR

Computer vision lacks broadly transferable visual features that work across tasks without finetuning, especially when text-guided pretraining can miss image information and requires aligned image-text data. DINOv2 scales self-supervised pretraining with large curated image data and efficient training, producing frozen features competitive with weakly supervised models across many benchmarks.

  • Problem

    Existing visual pretraining has limited evidence for producing all-purpose features from large, curated data without aligned text-image supervision.

  • Method

    DINOv2 scales discriminative self-supervised learning with an image-only curation pipeline, larger models, training improvements, and distillation into smaller Vision Transformers.

  • Results

    Self-supervised pretraining alone produces transferable frozen features that match or surpass weakly supervised models across a substantial range of image- and pixel-level tasks.

  • Takeaways & Limitations

    DINOv2 shows that curated-data self-supervision is a strong candidate for broadly transferable visual features without finetuning.

  • Takeaways & Limitations

    Text-guided pretraining requires aligned text-image corpora and may fail to retain complex pixel-level information.

Abstract

from arXiv · show

The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision. These models could greatly simplify the use of images in any system by producing all-purpose visual features, i.e., features that work across image distributions and tasks without finetuning. This work shows that existing pretraining methods, especially self-supervised methods, can produce such features if trained on enough curated data from diverse sources. We revisit existing approaches and combine different techniques to scale our pretraining in terms of data and model size. Most of the technical contributions aim at accelerating and stabilizing the training at scale. In terms of data, we propose an automatic pipeline to build a dedicated, diverse, and curated image dataset instead of uncurated data, as typically done in the self-supervised literature. In terms of models, we train a ViT model (Dosovitskiy et al., 2020) with 1B parameters and distill it into a series of smaller models that surpass the best available all-purpose features, OpenCLIP (Ilharco et al., 2021) on most of the benchmarks at image and pixel levels.

1 Introduction

DINOv2 investigates whether self-supervised learning can produce transferable, task-agnostic visual features from large quantities of curated image data, without aligned text-image supervision. It introduces scalable data and training techniques, releases ViT-based models and retraining code, and finds that self-supervised pretraining alone can yield frozen features competitive with leading openly available weakly supervised models.

  • Motivation: Text-guided image encoders depend on aligned text-image corpora, limiting their flexibility compared with learning from raw image data alone.The introduction contrasts this limitation with self-supervised learning, which learns from images without textual supervision.
  • Motivation: Self-supervised learning is studied as a route to all-purpose visual features that operate at both image and pixel levels using images alone.The work revisits discriminative approaches such as iBOT (Zhou et al., 2021) while reconsidering their design choices for larger datasets.
  • Contributions: DINOv2 builds an automatic pipeline that filters and rebalances extensive collections of uncurated images without manual annotation or external metadata.The pipeline addresses the risk that dominant concepts and modes in images from the wild cause imbalance and overfitting.
  • Contributions: The authors train and release DINOv2 models across Vision Transformer architectures, alongside code for retraining them on any data, and evaluate them on image- and pixel-level benchmarks.The models are validated across various computer vision benchmarks as their scale increases.
  • Results: Self-supervised pretraining alone produces transferable frozen features competitive with the best openly available weakly supervised models.The evaluation summarizes performance across eight types of vision tasks and compares DINOv2 with self-supervised and weakly supervised methods.

2 Related Work

Related work spans intra-image pretext tasks, discriminative self-supervised learning, scalable pretraining, and automated data curation. This work revisits discriminative approaches for large datasets and models, building particularly on Zhou et al. (2021).

  • Intra-image self-supervised training: Intra-image self-supervised methods learn from signals extracted within an image, including context prediction, recolorization, transformation prediction, and inpainting.
  • Discriminative self-supervised learning: Discriminative self-supervised learning learns features from signals between images or image groups, evolving from early metric-learning work to instance-classification methods.
  • Discriminative self-supervised learning: This work revisits discriminative self-supervised training for large pretraining datasets and models, building on Zhou et al. (2021) as a particularly scalable approach.
  • Scaling self-supervised pretraining: Scaling studies examine self-supervised learning with larger datasets and models, often using uncurated data and reporting evidence that discriminative methods scale with data.
  • Automatic data curation: Automatic data-curation methods draw on image retrieval, retrieval-based semi-supervised augmentation, and metadata such as hashtags to filter uncurated datasets.

3 Data Processing

The LVD-142M dataset is built by retrieving diverse images from uncurated web data that resemble images in curated datasets, using an image-only pipeline. The process removes near-duplicates, including benchmark evaluation images, and scales retrieval with self-supervised embeddings, clustering, and GPU-accelerated indexing.

  • Data sources: The pipeline constructs LVD-142M by retrieving images from a large uncurated pool that are close to images in several curated datasets, without using metadata or text.Curated sources include ImageNet-22k, ImageNet-1k training data, Google Landmarks, and fine-grained datasets; the uncurated source consists of crawled web images.
  • Deduplication: Near-duplicate removal reduces redundancy, increases image diversity, and excludes near-duplicates of test or validation images from the benchmarks.The deduplication stage uses the copy detection pipeline of Pizzi et al. (2022).
  • Self-supervised image retrieval: Retrieval uses cosine similarity between self-supervised ViT-H/16 embeddings pretrained on ImageNet-22k, followed by k-means clustering of the uncurated data.For a sufficiently large query dataset, the system retrieves N images per query image; the supplied passage truncates the remaining procedure.
  • Implementation Details: Faiss enables efficient nearest-embedding searches through GPU-accelerated inverted-file indices with product-quantization codes.The implementation cites Johnson et al. (2019) for Faiss and Jegou et al. (2010) for product quantization, and distributes processing across 20 nodes with 8 V100-32GB GPUs each.

4 Discriminative Self-supervised Pre-training

The method combines DINO and iBOT discriminative self-supervised losses with SwAV centering, feature-spreading regularization, and brief high-resolution training. It uses image- and patch-level objectives, separate objective heads, Sinkhorn-Knopp normalization, KoLeo regularization, and late resolution adaptation.

  • 4 Discriminative Self-supervised Pre-training: The pretraining method combines DINO and iBOT losses with SwAV centering, a feature-spreading regularizer, and a short high-resolution training phase.These components are designed to support robust features at both image and pixel levels.
  • Image-level objective: The image-level objective applies cross-entropy between student and teacher class-token features from different crops, with the teacher updated by exponential moving average.The student and teacher process different crops of the same image.
  • Patch-level objective: The patch-level objective masks student input patches, leaves teacher patches unmasked, and applies cross-entropy to corresponding masked-patch features alongside the image-level loss.This combines global image supervision with local patch supervision.
  • Untying head weights: Untying the heads for image- and patch-level objectives prevents patch-level underfitting and image-level overfitting, improving performance at both scales.Tied weights produced the opposite imbalance across the two objectives.
  • Normalization, regularization, and resolution: Sinkhorn-Knopp centering runs for 3 iterations on the teacher while the student uses softmax normalization; KoLeo spreads ℓ2-normalized batch features, and resolution rises to 518 × 518 late in pretraining.The high-resolution phase is brief because high-resolution training is time- and memory-intensive, while increased resolution helps pixel-level tasks involving small objects.

5 Efficient implementation

DINOv2 combines implementation changes that make training substantially faster and more memory-efficient than iBOT with sharding and distillation strategies that enable larger and smaller models. These include optimized attention, nested tensors, stochastic depth, FSDP, and distillation from ViT-g.

  • Efficient implementation: Using the same hardware, DINOv2 runs around 2× faster than iBOT while using only one-third of the memory.The implementation targets A100 GPUs with PyTorch 2.0, and the code and pretrained models are available for feature extraction.
  • Fast and memory-efficient attention: Custom FlashAttention improves self-attention speed and memory usage, matching or exceeding the original across tested cases while supporting more use-cases and hardware [Dao et al., 2022].Efficiency is best when the per-head embedding dimension is a multiple of 64, with further gains when the full embedding dimension is also a multiple of 64.
  • Nested tensors in self-attention: Nested tensors process global and local crops in one forward pass, yielding significant compute-efficiency gains over separate forward and backward passes.The lower-level components are available in xFormers [Lefaudeux et al. (2022)].
  • Efficient stochastic depth: Improved stochastic depth skips dropped residual computations instead of masking their outputs, saving memory and compute approximately in proportion to the drop rate [Huang et al., 2016].At the 40% drop rate used here, fused kernels provide a drastic improvement in compute efficiency and memory usage.
  • Fully-Sharded Data Parallel (FSDP): FSDP shards the 16 GB of replicas required by AdamW for a billion-parameter ViT-g across GPUs, reducing per-GPU memory and removing the model-size bound.The replicas comprise the student, teacher, and two optimizer moment states in float32 precision.
  • Model distillation: Distilling smaller models from ViT-g achieves better performance than training from scratch, even for ViT-L, while evaluating the student’s EMA [Hinton et al., 2015; Duval et al. (2023)].The method remains close to Duval et al.’s approach but does not modify the distillation loss terms.

6 Ablation Studies

The ablations validate the pipeline’s technical components, curated pretraining data, model–data scaling, loss terms, distillation, and training resolution across downstream tasks. Curated LVD-142M generally provides the strongest overall features, while individual components improve complementary evaluation settings.

  • 6.1 Technical Improvements: Incrementally adding the proposed technical components generally improves ImageNet-1k k-NN accuracy, linear-probe accuracy, or both, with LayerScale and Stochastic-specific exceptions noted in the evaluation.The ablation starts from a baseline iBOT model and adds components described in Section 4.
  • 6.2 Pretraining Data: LVD-142M outperforms uncurated data on most benchmarks and surpasses ImageNet-22k except on ImageNet-1k, providing the best overall balance across image domains.The results attribute this advantage to curated, diverse data that improves features in domains not covered by ImageNet-22k.
  • 6.3 Scaling Data and Model Size: As model size increases, LVD-142M becomes more beneficial than ImageNet-22k: ViT-g matches ImageNet-1k performance while significantly outperforming it on other benchmarks.Figure 4 compares model scaling across ImageNet-22k with 14M images and LVD-142M with 142M images.
  • 6.5 Model Distillation: The distilled ViT-L/14 outperforms a scratch-trained ViT-L/14 on 10 of 12 benchmarks, validating distillation from the larger ViT-g/14 model.The ViT-g/14 teacher is also reported as a topline reference.
  • 6.4 Loss Ablations: The KoLeo loss improves nearest-neighbor retrieval, whereas the iBOT masked-image-modeling loss improves patch-level tasks such as segmentation.Table 3 evaluates ImageNet-1k/A, ADE-20k segmentation, and Oxford-M retrieval under matched training iterations.

7 Results

Across global and local image-understanding evaluations, DINOv2 features outperform self-supervised state of the art by a large margin and match or surpass weakly supervised features on many tasks. They generalize across classification, retrieval, action recognition, segmentation, depth, and out-of-distribution settings, including with frozen backbones.

  • Overall evaluation: DINOv2 features outperform the current self-supervised state of the art by a very large margin and match or surpass weakly supervised features on many evaluated tasks.The evaluation spans global and local representations, category- and instance-level recognition, semantic segmentation, monocular depth prediction, and action recognition.
  • Holistic image representation: On ImageNet-1k linear evaluation, DINOv2 generalizes better than EVA-CLIP by 1.1%, while OpenCLIP-G is used thereafter as the weakly supervised reference.The comparison uses frozen backbones and a simple linear classifier; the evaluation also compares against several off-the-shelf weakly supervised architectural variants in Table 4.
  • Downstream classification and action recognition: DINOv2 significantly outperforms state-of-the-art features on transfer classification benchmarks while remaining competitive with OpenCLIP on most datasets, except notably SUN (−5.3%) and Cars (−4.7%).The evaluations cover iNaturalist, Places205, video action recognition on UCF101, Kinetics-400, and Something-Something v2, plus 12 transfer classification tasks spanning scenes, objects, and textures.
  • Instance-level recognition: For instance recognition, DINOv2 is evaluated with cosine-similarity retrieval on landmark, artwork, and archival street-view benchmarks, demonstrating strong performance across datasets and setups.The evaluated datasets are Paris, Oxford, Met, and AmsterTime.
  • Semantic segmentation and depth estimation: Frozen DINOv2 backbones produce strong dense predictions: +ms reaches 53.0 versus 53.6 mIoU for fully finetuned MAE, while the boosted recipe reaches 86.2 versus 89.0 mIoU on Pascal VOC and adapter tuning reaches 60.2 versus 62.9 mIoU on ADE20k.The adapter experiment tunes only the adapter and head, keeping the backbone frozen; the qualitative segmentation and depth results are better than OpenCLIP under the linear evaluation setup.
  • Out-of-distribution generalization and emergent structure: DINOv2 transfers across domains and captures semantic object structure, producing good segmentation and depth on animals and paintings while matching corresponding parts across objects despite style and pose variation.The model was not trained to parse object parts, yet its components delineate main-object boundaries and align semantically corresponding regions such as a plane wing and a bird wing.

8 Fairness and Bias Analysis

DINOv2’s largest model is evaluated for geographical fairness and harmful label associations. It remains biased toward Western and wealthier households, while generally avoiding harmful classifications across demographic groups.

  • Geographical fairness: Performance drops 25.7% in Africa versus Europe and differs by 31.7% between high- and low-income households, revealing persistent Western and wealth-related biases.The geographical evaluation uses Dollar Street’s 16,073 images from 289 households across 54 countries and compares 94 concepts across countries and income levels.
  • Label association fairness: Across gender, skin-tone, and age groups, DINOv2 usually classifies images as Human without large skin-tone deviations and does not predict harmful Non-Human or Crime labels.The evaluation uses a frozen backbone with a linear classifier trained on 619 ImageNet-22k classes, grouped into Human, Possibly Human, Non-Human, and Crime categories, and compares DINOv2 with SEERv2.
  • Label association fairness: DINOv2 frequently triggers Possibly-Human labels, which include objects such as scarves, glasses, and beards that are often related to humans.The reported harmful-label exceptions involve two images whose backgrounds contain bars visually similar to prison bars.

9 Estimating the Environmental Impact of Training our Models

The authors estimate training-related energy use and carbon emissions using Patterson et al.’s (2021) methodology, adapted to their data-center and power-grid assumptions. They report emissions for retraining DINOv2 and estimate the project’s total footprint, identifying self-supervised pretraining as the main source.

  • Carbon estimation methodology: The estimate uses Patterson et al.’s (2021) formula, a 400W A100-80GB thermal design power, and the US-average intensity of 0.385 kg CO2eq/KWh.The methodology relates emissions to data-center and power-grid characteristics.
  • Retraining footprint: Retraining OpenCLIP ViT-L and OpenCLIP ViT-G would require 22.4 MWh and 118.9 MWh, respectively, in the same data center.The passage states that Table 14 reports potential carbon emissions for retraining a DINOv2 ViT-g, but does not provide its value here.
  • Whole-project footprint: The whole project is estimated at 0.5k–1k tCO2eq, corresponding to approximately 200k GPU-days, with self-supervised pretraining as the primary emissions source.A ViT-g pretraining emits 3.7 tons of CO2eq over 22k GPU-hours, compared with 0.2 tons for ImageNet-1k finetuning over 1k GPU-hours.

10 Future work and Discussion … B Implementation Details

DINOv2 produces broadly applicable, unsupervised visual features from large curated data, while the appendix details dataset selection, similarity computation, deduplication, and retrieval procedures for constructing LVD-142M.

  • 10 Future work and Discussion: DINOv2 closes the performance gap with weakly supervised alternatives across diverse benchmarks without finetuning, while capturing object parts and scene geometry across image domains.The authors expect these properties to support further development of general-purpose visual features.
  • A.1 Data selection: LVD-142M selects datasets intended to cover downstream vision tasks at both image-level and dense recognition.The dataset selection is detailed in Table 15.
  • A.2 Image similarity: Image similarity is computed with cosine similarity between feature vectors generated by a model for two images.The similarity function is m(s, r) = cosine-similarity(f(s), f(r)).
  • A.3 Deduplication: Self-deduplication processes 1.3B uncurated images using k = 64 nearest neighbors, a similarity threshold >0.6, and connected components to retain one representative per duplicate group.The connected components are computed with a scalable disjoint-set data structure implementation.
  • A.3 Deduplication: Relative deduplication removes images too similar to evaluation train and test splits using duplicate-component matching with a stricter similarity threshold >0.45.The procedure discards an entire duplicate component when it contains a reference evaluation image.
  • A.4 Retrieval: Retrieval augments datasets through sample-based or cluster-based approaches, using k = 4 for Google Landmarks v2 and ImageNet-22k and k = 32 for LVD-142M.Sample-based retrieval collects nearest images for each dataset image, multiplying larger datasets by a fixed retrieval count; it applies to datasets larger than 1M images.

B.1 Unsupervised pre-training · B.2 High-Resolution adaptation · B.3 Linear probing evaluation

The appendix specifies DINOv2’s unsupervised pre-training, high-resolution adaptation, and linear-probing evaluation procedures. It details training schedules, teacher updates, adaptation, classifier-selection grids, and validation reporting.

  • B.1 Unsupervised pre-training: Unsupervised pre-training builds on the DINO and iBOT codebases, using the hyperparameters and ViT architectures specified in Tables 16 and 17.The architecture table distinguishes MLP feed-forward networks for distilled models from SwiGLU (Shazeer, 2020) for models trained from scratch.
  • B.1 Unsupervised pre-training: The LVD-142M dataset combines multiple datasets and splits, incorporating data directly or through sample-based and cluster-based retrieval.For retrieved data, the dataset construction reports both the number of images retrieved and the final number included.
  • B.1 Unsupervised pre-training: DINOv2 models are pretrained for 625k iterations with AdamW, scheduled weight decay and teacher momentum, 100k-step learning-rate warmup, and float16 precision.The weight-decay schedule runs from 0.04 to 0.2, while teacher momentum follows a cosine schedule from 0.994 to 1.
  • B.1 Unsupervised pre-training: The teacher starts equal to the student and is updated after every step as an exponential moving average with cosine-scheduled momentum in [0.994, 1.0].KoLeo regularization is also applied between first-global-crop class tokens within each GPU, with weight 0.1 and no cross-GPU communication.
  • B.2 High-Resolution adaptation: High-resolution adaptation initializes from pretrained weights and runs the original training procedure for 10k iterations with compressed schedules and a reduced base learning rate.All other hyperparameters and schedules remain the same as in pretraining, with schedules compressed to fit the shorter run.
  • B.3 Linear probing evaluation: Linear probing tunes learning rate, output-layer count, and whether to concatenate average-pooled patch tokens with the class token, training an SGD classifier for 12,500 iterations.The grid searches output layers in {1, 4} and token concatenation in {yes, no}, with random-resized-crop augmentation.
  • B.3 Linear probing evaluation: Evaluation reports the highest validation accuracy from the linear-probing grid search, whose cost is limited by one backbone inference per iteration followed by classifier matrix multiplications.This procedure evaluates all linear classifiers using the same backbone output at each iteration.

C List of benchmarks used for evaluations

This section identifies the benchmarks and datasets used to evaluate the models, with the complete list provided in Table 18.

  • C List of benchmarks used for evaluations: Table 18 lists the benchmarks and datasets used for evaluation.The passage introduces the table as the source for the evaluation benchmark list.
Loading 2304.07193v1…