PaperlayerView exact revision on arXiv

Source-linked AI summary

Data Density Scaling Laws for Image Self-Distillation

Alvard Barseghyan, Ani Vanyan, Hakob Tamazyan, Hrant Khachatrian

arXiv:2610.19013v1cs.LG

TL;DR

Scaling laws leave uncertain how unique samples and repeated exposure trade off in visual self-distillation, especially in redundant datasets. The paper tests that trade-off across seven datasets and extends a Chinchilla-style law to represent density interactions; results vary by dataset, including a strong smaller-subset result on Maxar.

  • Problem

    Existing compute-only laws collapse unique support and repetition into exposure, while data-constrained assumptions do not capture every observed visual self-distillation trade-off.

    Source: §1 · p. 2

  • Method

    The paper compares existing laws with a Chinchilla-style density-interaction extension that accounts for unique images and repetitions.

    Source: §1 · p. 2

  • Results

    A random subset containing 1.5% of the Full Maxar pool improves average downstream kNN performance by 3.11 percentage points over the full 50M-image pool at 200M image presentations, while density effects vary across datasets.

    Source: §1 · p. 1, Conclusion · p. 10

  • Takeaways & Limitations

    Scaling behavior depends on how training budget is allocated between unique support and repeated exposure, and the density-interaction law better describes tested extrapolation behavior.

    Source: §1 · p. 2, Conclusion · p. 10

    Under their parameter constraints, the baseline scaling laws cannot predict a smaller subset strictly outperforming a larger one at the same compute budget.

    Source: §5 · p. 8

Abstract

from arXiv · show

Current scaling law formulations suggest that increasing the number of unique samples in a dataset improves model performance, while repeated exposure to the same samples provides diminishing value. In this work, we examine this assumption for image self-distillation models. Through systematic experiments across diverse datasets, namely Maxar, SATLAS, Waymo, nuPlan, Ego4D, ImageNet, and YFCC15M, we demonstrate that the effects of sample repetition on downstream performance are non-trivial and dataset-dependent. We identify settings where repeatedly training on a smaller random subset can outperform training on a larger set of unique samples. This finding highlights a limitation of existing scaling laws. To address this gap, we propose a modified version of a Chinchilla-style scaling law that explicitly accounts for two factors: the number of unique samples and repetition per sample. This formulation provides a framework for understanding data-compute trade-offs and improves prediction in some regimes where repetition plays a central role, although its benefits are not consistent across datasets. Finally, we examine how subset selection and augmentation affect downstream performance.

1 Introduction

Scaling laws often treat more unique images as beneficial and repeated exposure as diminishing in value, but visual self-distillation shows dataset-dependent trade-offs between unique support and repetition. The paper studies these effects and proposes a density-interaction extension to a Chinchilla-style law.

1 Introduction

  • Existing scaling laws often assume that increasing unique image counts improves representation quality, an assumption that may be less reliable in domain-specific visual corpora with redundancy and uneven information density.The paper situates the question within scaling-law work on model, dataset, and compute growth.Source: §1 · p. 1
  • Across seven datasets, the interaction between unique sample count and repetition density is highly dataset-dependent.Experiments cover Maxar, SATLAS, Waymo, nuPlan, Ego4D, ImageNet, and YFCC15M.Source: §1 · p. 1
  • A random subset containing 1.5% of the Full Maxar pool improves average downstream kNN performance by 3.11 percentage points over the full 50M-image pool at the same 200M image presentations.The result challenges the assumption that more unique samples always improve performance at fixed compute.Source: §1 · p. 1
  • The study distinguishes unique data size U, repetition R, and compute budget C; encountered support is min(C, U), with R = C/Useen and C = UseenR.C is the total number of source-image presentations and serves as a compute proxy within a fixed model and training recipe.Source: §1 · p. 2
  • Visual self-distillation differs from token replay because stochastic augmentations and an EMA-evolving teacher give repeated image views changing supervisory signals.These differences motivate the study but do not establish the mechanism behind the subset advantage.Source: §1 · p. 2
  • The proposed density-interaction extension captures interactions between unique images and repetitions, improving held-out prediction on some datasets but not consistently across all seven.The paper also examines how the subset advantage depends on model size, self-distillation method, subset selection, and augmentation.Source: §1 · p. 2

2 Related Work

Prior scaling laws and data-selection research motivate this study, but largely leave total exposure as the principal scaling axis. This work instead examines how repetition density and unique sample count interact in visual self-distillation.

2 Related Work

  • Repeated views in visual self-distillation carry changing signals because augmentations are stochastic and the teacher evolves through EMA updates.The passage says this distinction may partially explain different repetition behavior from data-constrained language modeling.Source: §2 · p. 2
  • Scaling-law work commonly models performance through model size, unique tokens, and compute, with repeated language-model exposure yielding diminishing returns [4].Visual representation learning has adopted similar expectations for larger image collections.Source: §2 · p. 2
  • Data filtering and constrained-compute studies report that curated subsets can outperform larger unfiltered corpora, but existing formulations still primarily treat total exposure as one scaling axis.Prior work also connects subset utility with data quality, filtering, and selection strategy.Source: §2 · p. 3
  • Traditional scaling laws collapse unique sample count and repetition into a single exposure term, while the paper reports that these factors play different roles in highly redundant visual domains.Its stated objective differs from data pruning: studying their interaction under fixed compute budgets.Source: §2 · p. 3
  • The paper reports conventional scaling behavior for ImageNet but regimes in highly redundant domains such as Maxar where repeated training on curated subsets outperforms larger unique collections.It frames this as a change in scaling behavior across datasets with different redundancy structures.Source: §2 · p. 3
  • Prior research links nominal dataset size to duplicates, semantic overlap, and low-information samples, while this paper hypothesizes that harmful sample co-occurrences may interact under self-distillation.It measures semantic coverage using Vendi scores over DINOv2 embeddings.Source: §2 · p. 3

3 Experimental Design

The experimental design spans seven datasets and four visual domains, varying unique-data support, compute, and subset-selection strategy. Frozen representations are assessed on domain-specific classification and tracking tasks, with kNN and logistic-regression aggregates kept separate.

3.1 Pretraining Datasets

  • Pretraining spans seven datasets across remote sensing, driving, egocentric video, and natural images; the experimental Full pools include 50M Maxar images and datasets of varied scale and content.Full denotes the complete experimental training pool, which may differ from the complete source dataset.Source: §3.1 · pp. 3–4

3.2 Sampling Strategies

  • The design varies uniformly sampled random-subset support at fixed budgets and compares subset-selection methods at matched sizes and budgets.It also tests hierarchical representation-space clustering, DASSOT, and NIQE-based quality selection against random sampling; the latter comparison uses 0.75M-image Maxar subsets.Source: §3.2 · p. 4

3.3 Model Pretraining

  • iBOT pretraining uses a ViT-B/16 backbone, and compute budgets range from 10M to 400M image presentations across random and hierarchical sampling runs.With batch size 1,600, optimization steps are calculated as S = C/B; Figure 2 maps runs by U and C, with dataset colors, strategy markers, and constant-R diagonals.Source: §3.3 · p. 4, Figure 2 · p. 5

3.4 Downstream Evaluation

  • Frozen normalized final-layer [CLS] embeddings are evaluated with kNN and logistic regression across tasks for remote sensing, driving, general recognition, and egocentric recognition.Scores are averaged across available tasks and seeds, keeping kNN and logistic regression separate; main results use kNN, while logistic-regression results appear in Appendix D.Source: §3.4 · p. 5

4 Empirical Observations

The fixed-compute trade-off between unique support and repetition varies across datasets and compute budgets. Maxar can benefit from repeatedly training on a small subset, while ImageNet and Waymo show different patterns.

Motivation

  • Unlike autoregressive language modeling’s repeated static targets, image self-distillation supplies repeated images with stochastic views and a changing EMA teacher.The data-constrained language-model assumption is that repeated data has diminishing value, but this behavior is not obviously valid for image self-distillation.Source: §4 · pp. 5–6

Observation 1: the fixed-compute U/R trade-off is dataset-dependent.

  • At fixed compute, ImageNet loses performance as unique support shrinks, Waymo tolerates 4-to-16-fold reductions, and Maxar peaks near U ≈F/50 to F/100.At C = 100M, ImageNet falls from 0.6956 with the full dataset to 0.6545 at roughly half the images.Source: §4 · p. 6

Observation 2: the trade-off changes with compute.

  • The support–repetition trade-off also shifts with compute: ImageNet is relatively flat at C = 20M but support-limited at higher budgets, while Maxar’s small-subset advantage grows with compute.Waymo remains comparatively flat across the tested non-spike budgets.Source: §4 · p. 6

Observation 3: more unique data can be harmful, not merely redundant.

  • At C = 100M, Maxar subsets near U = 0.5M–1M reach about 0.921–0.922, versus about 0.895–0.897 for full-data runs.At C = 400M, full-data Maxar falls to about 0.886, while U = 0.4M remains around 0.914.Source: §4 · pp. 6–7

Observation 4: repetition can help, saturate, or hurt.

  • Repetition can hurt, be tolerated, or improve kNN performance across ImageNet, Waymo, and Maxar, and prolonged training at large or full Maxar support can degrade it.The Maxar degradation is less visible under logistic-regression evaluation, where performance more often appears to saturate.Source: §4 · p. 7

Caveat

  • Known training-loss-spike runs are excluded from the scaling observations because they reflect optimization instability, not the data-density trade-off.For example, the Waymo full-data run falls from 0.7714 at C = 100M to 0.5804 at C = 200M after a spike.Source: §4 · p. 7

5 Density-Aware Scaling Law

The section tests whether scaling laws capture compute, unique support, and repetition, and whether low-compute fits predict performance at higher compute. Density interaction improves held-out prediction on some datasets, but not consistently.

Fitting protocol

  • Laws are fitted separately by dataset on runs with C ≤50M presentations and evaluated on higher-compute runs, predicting average downstream accuracy across frozen-representation tasks.The target is transfer performance rather than pretraining loss.Source: §5 · p. 7

Existing laws

  • Under the imposed parameter constraints, none of the three existing baselines can predict a smaller subset strictly outperforming a larger one at the same compute budget.The compute-only law predicts the same for all subsets at fixed compute; the other two allow repetition benefits or declines but not the smaller-subset advantage.Source: §5 · p. 8

Density interaction

  • The density-interaction law extends a compute-only baseline with a joint correction for unique support and repetition, interpreted as a phenomenological density penalty rather than a first-principles law.The interaction adjusts predictions according to how total exposure is divided between unique images and repetitions; a proposed mechanism remains a hypothesis.Source: §5 · p. 8

Main results

  • Density interaction has the lowest held-out MAE on Maxar, SATLAS, and Ego4D, is near the best baseline on Waymo, and performs worse on nuPlan, ImageNet, and YFCC15M.Table 1 reports kNN scaling-law MAE for the held-out range.Source: §5 · p. 9, Table 1 · p. 9

Model size and self-distillation method

  • At 100M presentations, the 0.75M-image subset exceeded full-data average kNN accuracy for ViT-S (0.9207 vs 0.8994), ViT-L (0.9196 vs 0.8796), and DINOv2 (0.9182 vs 0.9095).The subset advantage also appeared under logistic-regression evaluation; the DINOv2 result persisted through 300M presentations.Source: §5 · p. 9

Subset selection

  • At 50M presentations, random, hierarchical, DASSOT, and NIQE-based 0.75M-image subsets achieved average kNN accuracies of 0.9202, 0.9187, 0.9202, and 0.9216, versus 0.9015 for full-data training.The comparisons do not fully separate repetition, redundancy, image quality, and semantic coverage.Source: §5 · p. 9

Augmentations

  • With reduced augmentation, average kNN accuracy fell from 0.9251 to 0.8910 at 100M presentations, but rose from 0.8400 at 10M to 0.8969 at 400M.Crops and masks still provided variation; without a reduced-augmentation Full control, persistence of the subset advantage could not be determined.Source: §5 · p. 9

Conclusion

Data density matters in image self-distillation, but its effect depends on the dataset. The density-interaction law better describes tested extrapolations, though not consistently across datasets.

Conclusion

  • At matched compute, reducing unique support can hurt early on ImageNet, remain nearly neutral on Waymo, and substantially improve Maxar.The study varies total compute, unique support, and repetition independently.Source: Conclusion · p. 10
  • Compute-only and data-constrained laws do not consistently capture the observed behaviors, while a density-interaction law better describes the tested extrapolation range.Its empirical advantage is not stated to hold consistently across datasets.Source: Conclusion · p. 10
  • Data density is not a minor implementation detail in the tested image self-distillation settings.The observed effects range from early harm to near neutrality to substantial improvement as unique support is reduced.Source: Conclusion · p. 10

A Pretraining Dataset Vendi Scores

Table 2 reports Vendi scores across datasets, sampling strategies, and target subset percentages; subset sizes may use dataset-specific matching rules.

A Pretraining Dataset Vendi Scores

B Downstream Tasks

The downstream evaluations span remote-sensing, driving, general, and egocentric tasks, with fixed-reference kNN settings and separate logistic-regression aggregates. The logistic-regression density curves compare accuracy across unique-support, compute, and repetition conditions.

B Downstream Tasks

  • Maxar and SATLAS use six remote-sensing classification tasks; Waymo and nuPlan use three driving classification tasks, with two BDD100K tracking scores added to their kNN aggregates.Source: B Downstream Tasks · p. 15
  • Ego4D, ImageNet, and YFCC15M share 24 classification tasks, while Ego4D also evaluates EPIC-KITCHENS-100 noun recognition with single- and five-frame pooled features.Source: B Downstream Tasks · p. 15
  • For each pretraining group and classification task, k is selected using Full-50M reference runs and held fixed across evaluated models; kNN and logistic-regression scores are aggregated separately.The allowed k values are 1, 5, 10, 30, and 50; ties select the smallest k.Source: B Downstream Tasks · p. 15

C Logistic-regression density curves

  • Figure 5 plots mean downstream accuracy as unique images vary at fixed compute, compute varies at fixed unique support, and repetition varies at fixed unique support.Rows cover Maxar, SATLAS, Waymo, nuPlan, Ego4D, ImageNet, and YFCC15M; horizontal axes are logarithmic.Source: Figure 5 · p. 17

D Logistic-regression scaling-law results

The logistic-regression figures show density trade-offs and compare scaling-law fits across seven datasets. Fits use low-compute runs and are evaluated on higher-compute runs.

D Logistic-regression scaling-law results

  • Figure 6 compares four logistic-regression scaling laws across seven datasets, including the density-interaction law.The laws are fitted on runs with C ≤50M and evaluated on runs with C >50M; panels show fitted equations and held-out MAE.Source: Figure 6 · p. 18

E k-NN density curves

  • Figure 7 plots kNN accuracy as unique images vary at fixed compute and as compute or repetition varies at fixed unique support.Rows cover the seven datasets, and k is selected from Full-data 50M reference runs and held fixed across evaluated models.Source: Figure 7 · p. 19
  • In Figure 7, random-subset and Full-data runs are shown as points connected by lines, with logarithmic horizontal axes and image counts in millions.Source: Figure 7 · p. 19

F k-NN scaling-law results

The kNN scaling-law figure compares four fits across seven datasets, fitting on low-compute runs and evaluating on higher-compute runs. A separate logistic-regression comparison table specifies its log-transformed unique-support and repetition quantities.

F k-NN scaling-law results

  • Figure 8 compares compute-only, data-constrained effective-data, prescriptive, and density-interaction laws under kNN evaluation across seven datasets.Source: Figure 8 · p. 20
  • The kNN scaling-law curves are fitted on runs with C ≤50M and evaluated on runs with C >50M, with fitted equations and held-out MAE shown in each panel.Source: Figure 8 · p. 20
  • Table 6 compares logistic-regression scaling laws using ℓU = log(1 + U) and ℓR = log(1 + R), with lower MAE indicating better fit.Each law is fitted on runs with C ≤50M and evaluated on runs with C >50M.Source: Table 6 · p. 21

G Scaling-law ablation plots

The section presents kNN and logistic-regression ablation plots for the density-interaction law, fitted coefficients by dataset and evaluation protocol, and a Maxar comparison of subset-selection strategies.

G Scaling-law ablation plots

  • The density-interaction law is examined with kNN and logistic-regression ablation plots; the kNN rows compare it with five model variants.The variants are unique-only, repeat-only, separate-additive, raw-U, and raw-R.Source: Figure 9 · p. 22, Figure 10 · p. 23

H Fitted density-interaction coefficients

  • Table 7 reports density-interaction-law coefficients fitted independently for each dataset and evaluation protocol.The fitted form is Perf = L−A/C^α−D[log(1+U)]^β[log(1+R)]^γ.Source: Table 7 · p. 24

I Subset Selection

  • Figure 11 compares hierarchical-clustering selection with random selection on Maxar, plotting downstream-performance differences as compute increases for fixed subset sizes.The plotted difference is hierarchical minus random.Source: Figure 11 · p. 25
Loading 2610.19013v1…