PaperlayerView exact revision on arXiv

Source-linked AI summary

Growing to See: Efficient Pretraining of Self-Supervised Vision Transformers

Amir Mohammad Karimi Mamaghan, Ferdinand Kapl, Andrea Dittadi, Karl Henrik Johansson, Kaitlin Maile, Johannes von Oswald, Stefan Bauer

arXiv:2610.16817v1cs.CVcs.AIcs.LG

TL;DR

The paper addresses limited understanding of how progressive growth shapes representations and computation in self-supervised vision transformers. It studies growth in MAE and DINOv3, finding improved performance across global and dense tasks with less pretraining compute, alongside changes in intermediate representations and attention patterns.

  • Problem

    How progressive growth shapes representations and computational patterns in self-supervised vision models remains insufficiently understood, including across masked-image modeling and self-distillation objectives.

    Source: §1 · p. 2

  • Method

    The study progressively inserts copied layers into MAE and DINOv3, comparing growth strategies and analyzing downstream tasks, intermediate representations, and internal computation.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Across MAE and DINOv3, growing improves classification, segmentation, and depth estimation while reducing pretraining FLOPs by 13.9–24.8%.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Simple block copying is a practical pretraining approach that influences both representation quality and computational organization.

    Source: §1 · p. 2, §5 · p. 10

    Comparisons also differ in learning-rate schedule, so they do not isolate the effect of the teacher setting.

    Source: §4.1.2 · p. 7

Abstract

from arXiv · show

Progressive depth growth builds large networks by starting with shallow models and inserting layers during training. Recent advances in large language models have renewed interest in this longstanding approach, demonstrating benefits for training efficiency and reasoning alongside changes in internal computations. However, how growing affects representation quality and computational patterns in self-supervised vision transformers remains unclear. In this work, we systematically investigate progressive depth growth in two popular self-supervised families, namely masked autoencoders (MAE) and DINOv3, and analyze how growing affects downstream performance, intermediate representations, and internal computations. Across global and dense tasks, we show that growing improves downstream performance while reducing training FLOPs by 13.9--24.8%, with ImageNet linear-probe accuracy gains of up to 3.36 percentage points. Growing also improves downstream performance at intermediate layers and reshapes internal computational patterns, introducing recurring alternations between global and localized attention in MAE and earlier local-to-global attention progression in self-distilled DINOv3. Together, these findings establish progressive depth growth as an effective and efficient pretraining strategy for improving downstream performance across global and dense visual tasks with less computation, and advance our understanding of how growing shapes visual computation.

1 INTRODUCTION

The paper investigates how progressive depth growth affects self-supervised vision representations and computation, comparing MAE and DINOv3 across global and dense tasks. It reports improved downstream performance with less pretraining compute and changes in intermediate representations and attention patterns.

1 INTRODUCTION

  • How growth changes representations and computation in self-supervised vision models remains insufficiently understood, especially across masked-image modeling and self-distillation objectives.Prior analyses of pretrained ViTs describe local-to-global transitions and repeated or phase-like behavior across layers, while attention and representations vary across SSL objectives.Source: §1 · pp. 1–2
  • The study compares Stacking, MIDAS, and LIDAS across model sizes, growth schedules, training budgets, and distillation settings, using matched-step and data-exposure fixed-depth baselines.It evaluates classification, segmentation, and depth estimation alongside layer-wise readouts, attention analyses, and layer-skipping interventions.Source: §1 · p. 2
  • Across global and dense tasks, growing improves final-layer performance while reducing pretraining FLOPs by 13.9–24.8%, with unchanged final architectures and inference costs.ImageNet linear-probe accuracy improves by up to 3.36 percentage points, ADE20K mIoU by up to 1.44 points, and NYUv2 depth-estimation RMSE decreases by up to 0.0538 meters.Source: §1 · p. 2
  • Growing improves frozen intermediate representations, often with larger gains than at the final layer, and changes attention patterns in MAE and DINOv3.MAE develops recurring alternations between global and localized attention; self-distilled DINOv3 shows earlier local-to-global progression.Source: §1 · p. 2

2 RELATED WORK

Prior work spans self-supervised visual objectives and neural-network growth, with recent language-model studies linking growth to efficiency and changes in learned computation. Vision growth studies have primarily targeted efficiency, leaving limited analysis of internal computation across SSL objectives.

Self-supervised Visual Representation Learning

Growing Neural Networks

  • Neural-network growth has a long history of building larger models by reusing what smaller models have learned, including function-preserving expansion in Net2Net [Chen et al., 2015].Net2Net was later extended to transformers [Gesmundo & Maile, 2023].Source: §2 · p. 2
  • Recent language-model growth work reports efficient depth-wise stacking and reasoning gains, while associating middle stacking with increased late-layer utilization and permutable computational blocks [Du et al., 2024; Saunshi et al., 2024; Kapl et al., 2026a].MIDAS improved reasoning despite similar or slightly worse perplexity, and LIDAS was introduced as a layer-wise middle-duplication strategy with further reasoning gains [Saunshi et al., 2024; Kapl et al., 2026a].Source: §2 · p. 3

Growing in Vision

  • Vision growth research has largely pursued training acceleration in supervised ViTs or image–text models, with limited analysis of internal computation across SSL objectives.Prior approaches include supernet search, architecture-specific expansion, learned mappings, and staged width or depth growth; SCG studies small self-supervised JEPA encoders on synthetic environments [Li et al., 2022; Wang et al., 2023; Berenz, 2026].Source: §2 · p. 3

3 PROGRESSIVE DEPTH GROWING OF VISION TRANSFORMERS

Progressive depth growth starts with a shallow transformer and inserts copied layers until it reaches a target depth. The paper separates the choice of copied layers from the schedule for increasing depth, and compares three growth strategies and proportional schedules.

3 PROGRESSIVE DEPTH GROWING OF VISION TRANSFORMERS

  • Progressive depth growth inserts layers into a shallow network, allocating part of training to smaller models to reduce pretraining FLOPs at the same steps and batch size.The paper studies masked-image reconstruction with MAE and self-distillation with DINOv3, separating which layers to copy from when to increase depth.Source: §3 · p. 3
  • Each growth event increases depth from N to N+B by copying a contiguous slice of B layers and inserting the copies after their source.The layer sequence before a growth event is ordered with zero-based indices; N is a multiple of B.Source: §3 · p. 3
  • Stacking copies the final B layers, MIDAS duplicates a middle group of B layers, and LIDAS duplicates a central B-layer slice without group-boundary alignment.Copied blocks are inserted after their source layers and initialized from them; source and copied layers then have separate trainable parameters and update jointly without weight sharing.Source: §3 · p. 3
  • The default setup grows models from four layers in increments of four to 24 layers for ViT-L or 32 for ViT-H, keeping width, attention-head count, and patch size unchanged.MAE grows only its encoder while keeping the decoder fixed; in self-distilled DINOv3, the teacher backbone is reset to the expanded student backbone at each growth event before EMA updates resume.Source: §3 · pp. 3–4
  • Prop-α schedules allocate a K-stage growth horizon across stages of depths iB, with each stage duration proportional to i^α.Larger α allocates more steps to deeper stages and increases pretraining FLOPs for a fixed number of training steps; the schedule trades compute saved at shallow depths against training at deeper models.Source: §3 · p. 4

4 EXPERIMENTS

The experiments compare grown and fixed-depth MAE and DINOv3 models under matched training steps and data exposure, then evaluate downstream tasks, intermediate representations, and computational patterns. Growing improves global and dense task performance with lower pretraining compute, while producing distinct attention changes in the two model families.

4.1 GROWING IMPROVES DOWNSTREAM PERFORMANCE WITH LESS COMPUTE

  • For MAE ViT-L, Stacking uses 79.6% of baseline FLOPs, raising linear accuracy from 60.28 to 63.64 and mIoU from 26.29 to 27.73 while lowering RMSE from 0.6393 to 0.5855.For DINOv3, LIDAS uses 86.1% of baseline FLOPs and reports 83.73 linear accuracy, 45.77 mIoU, and 0.4523 RMSE versus 83.40, 44.70, and 0.4650 for the fixed-depth baseline.Source: §4.1 · pp. 4–5
  • Across MAE and DINOv3, growing improves downstream performance across global and dense tasks while reducing pretraining compute.Table 1 reports FLOPs relative to each fixed-depth baseline; masking experiments also find robustness gains under random and semantic masking for specified models and tasks.Source: §4.1 · pp. 4–5, Table 1 · p. 5

4.1.1 EXTENDED MAE TRAINING

  • After growing for the first 50k of 500k training steps, MAE Stacking retains an advantage over the fixed-depth baseline through 500k steps.Final gains are approximately 1.9 and 1.1 percentage points in k-NN and linear accuracy, 0.9 points in mIoU, and a 0.01 reduction in RMSE.Source: §4.1.1 · p. 6

4.1.2 DISTILLATION FROM A PRETRAINED TEACHER

  • With a frozen full-depth teacher, grown students reduce FLOPs by 23%, retain similar classification performance, and improve dense prediction scores.MIDAS and LIDAS slightly lower classification scores but raise ADE20K mIoU and reduce NYUv2 RMSE; the comparison does not isolate teacher setting because learning-rate schedules differ.Source: §4.1.2 · pp. 6–7

4.2 GROWING IMPROVES REPRESENTATIONS IN INTERMEDIATE LAYERS

  • Growing improves layer-wise downstream performance across both model families and tasks, with particularly large gains in middle layers that narrow toward the final layers.MAE Stacking approaches final classification and segmentation performance around layers 16–18, while the baseline improves primarily in later layers.Source: §4.2 · p. 7, Figure 3 · p. 7

4.3.1 MAE: REPEATED CYCLES OF GLOBAL AND LOCAL ATTENTION

  • MAE Stacking develops a recurring four-layer spatial-attention cycle, shifting from global attention to localized, query-dependent attention and back to broader attention.The CLS–patch attention pattern is weaker and less regular, suggesting the four-layer cycle is mainly a feature of spatial attention.Source: §4.3.1 · pp. 7–8, Figure 4 · p. 8, Figure 5 · p. 8

4.3 GROWING CHANGES COMPUTATIONAL PATTERNS

  • Layer-skipping interventions show stronger effects from skipping early layers and smaller effects from skipping later layers in both MAE and DINOv3.The analyses also measure spatial attention distance, entropy, query–key NMI, and CLS–patch attention mass.Source: §4.3 · p. 7, §4.3.1 · p. 8, §4.3.2 · p. 9

4.3.2 DINOV3: EARLIER TRANSITION FROM LOCAL TO GLOBAL ATTENTION

  • In DINOv3, growing brings localized, query-dependent attention earlier, followed by broader, less query-dependent attention across much of the middle layers.CLS–patch profiles and layer-skipping interventions likewise indicate earlier changes in token interactions and layer influence; LIDAS also shows clearer object-region similarity patterns at earlier layers.Source: §4.3.2 · p. 9, Figure 6 · p. 9

5 DISCUSSION AND CONCLUSION

The discussion concludes that progressive depth growth can improve representation quality and alter computation in MAE and DINOv3 while reducing pretraining compute. It presents possible uses for stronger intermediate representations and documents the training setup, reproducibility materials, and AI assistance used in the work.

5 DISCUSSION AND CONCLUSION

  • Across MAE and DINOv3, growing improves classification, segmentation, and depth estimation with reduced compute, improves intermediate representations, and changes internal processing.The discussion reports recurring attention patterns in MAE and earlier local-to-global progression in DINOv3, and says MAE’s final-layer gains and recurring patterns persist after prolonged full-depth training.Source: §5 · pp. 9–10
  • The authors identify early-exit inference, cheaper feature extraction, adaptive dense prediction, and self-speculative decoding as potential uses of stronger intermediate representations.These are presented as possibilities for exploration, not demonstrated outcomes.Source: §5 · p. 10

REPRODUCIBILITY STATEMENT

  • The paper states that experiment details are provided in the main text and appendix, including architectures, growth procedures, evaluation protocols, and analysis procedures.The work builds on publicly available MAE and DINOv3 implementations and uses publicly available datasets.Source: REPRODUCIBILITY STATEMENT · p. 10

A TRAINING AND GROWING SETUP

  • Experiments use matched steps, batch size, and data exposure, while grown models reach the fixed-depth counterparts’ final architecture and use different intermediate depths and pretraining compute.The setup builds on original MAE and DINOv3 implementations; generative AI tools assisted implementation, result interpretation, literature retrieval, manuscript drafting, and plotting code, with the authors taking responsibility for the final content.Source: AI USE STATEMENT · p. 10, A TRAINING AND GROWING SETUP · p. 14

A.1 TRAINING SETUP

The section documents the MAE and DINOv3 training recipes, growth procedures, distillation setup, and evaluation pipeline.

A.1 TRAINING SETUP

  • All models use the ImageNet-1k training split, and training uses 32 NVIDIA GH200 GPUs.Source: §A.1 · p. 14

A.1.1 MAE

  • MAE encoders process visible patches after 75% masking, while an eight-layer decoder reconstructs normalized pixels on masked patches.Source: §A.1.1 · p. 14

A.1.2 DINOV3

  • DINOv3 follows its public ImageNet recipe with DINO and iBOT objectives, KoLeo regularization, and two global plus eight local crops per image.Source: §A.1.2 · pp. 14–15

A.1.3 TRAINING HYPERPARAMETER SELECTION

  • MAE hyperparameters are selected through schedule and learning-rate sweeps, while DINOv3 settings are selected using the same validation split used for reporting.Source: §A.1.3 · p. 16

A.2 GROWING SETUP

  • Growth begins at four layers and adds four per stage; MAE uses Stacking, MIDAS, and LIDAS, while DINOv3 uses MIDAS and LIDAS under its selected schedule.Source: §A.2 · p. 16
  • Copied layers are inserted after their source slice, and transitions preserve existing optimizer state and global schedules without restarting warmup.Source: §A.2 · pp. 16–18

A.3 DISTILLATION SETUP

  • For frozen-teacher distillation, randomly initialized students use targets from a fixed pretrained DINOv3 teacher, while grown students follow a six-stage schedule.Source: §A.3 · p. 19

B EVALUATION SETUP

  • Evaluation covers classification, segmentation, and depth estimation, using frozen backbones and layer-specific readouts for intermediate-layer assessment.Source: B EVALUATION SETUP · p. 19

B.1 DOWNSTREAM EVALUATION

The evaluation sections specify frozen-feature downstream protocols and additional analyses of attention, interventions, masking robustness, and representation similarity.

B.1 DOWNSTREAM EVALUATION

  • Frozen backbones are evaluated on classification, segmentation, and depth tasks, with separate trained task heads for each evaluated layer.Source: §B.1 · p. 19

B.1.1 CLASSIFICATION

  • ImageNet classification uses top-1 accuracy; k-NN uses cosine similarity of CLS features, while linear probes use concatenated CLS and mean-pooled patch features.Source: §B.1.1 · p. 20

B.1.2 SEMANTIC SEGMENTATION; B.1.3 DEPTH ESTIMATION

  • ADE20K segmentation reports validation mIoU, while NYUv2 depth estimation reports per-image RMSE in meters using a linear binned depth head.Source: §B.1.2 · p. 20, §B.1.3 · p. 21

B.2 COMPUTATIONAL PATTERN ANALYSES

  • Attention analyses measure spatial distance and entropy, query–key NMI, and raw attention mass exchanged between CLS and patch tokens.Source: §B.2 · pp. 21–22
  • Layer-skipping interventions bypass one block and track subsequent residual-stream changes, while feature maps compare CLS–patch and patch–patch cosine similarities.Source: §B.2 · p. 22

B.3.1 ROBUSTNESS TO MISSING EVIDENCE

  • Masking tests compare random patch removal and semantic foreground or context masking, reporting task scores and masked-to-clean performance retention.Source: §B.3.1 · pp. 22–23

B.3.2 MAE MLP PROBES; B.3.3 MAE FINE-TUNING

B.3.4 REPRESENTATION SIMILARITY; B.3.5 WEIGHT SIMILARITY; C.1 ABLATION STUDY

  • Representation similarity compares layer-pair features, while weight similarity compares checkpoint parameters and source–copy blocks; ablations examine strategy, schedule, and configuration trade-offs.Source: §B.3.4 · p. 23, §B.3.5 · pp. 23–24, §C.1 · p. 24

C.1.1 ABLATIONS ON GROWING SCHEDULE AND STRATEGY

Under matched Prop-1 schedules, the strongest growing strategy varies by model family and scale. Changing the schedule can further improve some tasks, but no configuration is best across all objectives.

Growing strategy

  • At 79.6% of baseline FLOPs, MAE ViT-L Stacking improves linear accuracy by 3.36 points and mIoU by 1.44 points; LIDAS leads on k-NN accuracy.At ViT-H, LIDAS leads linear accuracy and mIoU, while Stacking leads k-NN and depth estimation; growing uses 24.8% fewer FLOPs.Source: §C.1.1 · pp. 24–25
  • With 27.5% less compute, all three Prop-1 DINOv3 strategies fall below baseline on classification and segmentation; MIDAS has the lowest depth RMSE.LIDAS comes closest on linear accuracy, reaching 83.31% versus 83.40% for the baseline.Source: §C.1.1 · p. 25

Growing schedule

  • For MAE Stacking, Prop-2 raises k-NN accuracy from 33.86% to 36.33% and mIoU from 27.73% to 28.27%, while depth estimation favors Prop-1.Prop-3 uses more computation than Prop-2 but has lower k-NN accuracy and mIoU; all three schedules outperform the fixed-depth baseline on every task.Source: §C.1.1 · p. 25, Figure 8 · p. 25
  • For DINOv3, Prop-2 and Prop-3 improve classification and segmentation over Prop-1; Prop-3 MIDAS exceeds baseline on all four metrics, while Prop-3 LIDAS leads linear accuracy and mIoU.Prop-3 LIDAS k-NN accuracy is 0.01 percentage points below baseline; Prop-2 MIDAS achieves the lowest RMSE, and the compared configurations use 86.1% and 81.0% of baseline FLOPs, respectively.Source: §C.1.1 · pp. 25–26
  • Schedule and strategy both matter: greater allocation to deeper stages raises pretraining FLOPs, and MAE shows a clearer classification/segmentation versus depth-estimation trade-off.The preferred configuration depends on model family and task.Source: §C.1.1 · p. 26

C.1.2 ABLATIONS ON TRAINING CONFIGURATION

Training-configuration comparisons show that preferred learning rates and interventions depend on schedule, strategy, and downstream task. Growing remains beneficial under a stronger MAE WSD reference, while DINOv3 ablations yield mixed task-specific trade-offs.

MAE learning rate and schedule

  • For fixed-depth MAE, WSD at 10^-3 is strongest across all four metrics, while the preferred cosine configuration varies by task.The WSD configuration achieves 35.71% k-NN accuracy, 64.02% linear accuracy, 27.07% mIoU, and 0.6214 RMSE; increasing its learning rate to 2.4 × 10^-3 worsens all four metrics.Source: §C.1.2 · p. 27
  • Under WSD at 10^-3, MAE Stacking improves k-NN accuracy by 4.78 points, linear accuracy by 0.52 points, and mIoU by 1.13 points, while reducing RMSE from 0.6214 to 0.5803.Both grown configurations use 79.6% of the corresponding fixed-depth FLOPs, and growing improves all four metrics under both adopted recipes.Source: §C.1.2 · p. 27

DINOv3 learning rate and schedule

  • DINOv3's favorable learning-rate range differs by schedule: cosine results peak near 10^-3, while WSD favors roughly 3.5 × 10^-4 to 5 × 10^-4 depending on task.Raising WSD to 10^-3 worsens classification, segmentation, and depth error, especially without cooldown; replacing linear cooldown with 1 −√s improves the tested WSD recipes.Source: §C.1.2 · p. 27

DINOv3 stochastic depth and layer-wise learning rates

  • Raising stochastic depth from 0.1 to 0.3 reduces DINOv3 classification and segmentation under all three listed WSD variants, but improves cosine k-NN and linear accuracy by 0.23 and 0.18 points.The cosine change slightly worsens segmentation and depth, so the preferred stochastic-depth value does not transfer automatically across schedules.Source: §C.1.2 · p. 27
  • In grown DINOv3, increasing layer-wise learning-rate decay to 0.99 lowers linear accuracy and mIoU for both MIDAS and LIDAS, while depth responses differ by strategy.The grown cooldown change also gives no improvement shared across tasks: cosine cooldown reduces both classification scores and mIoU for both strategies, with mixed depth effects.Source: §C.1.2 · p. 28

DINOv3 iBOT loss weight

  • For cosine-trained grown DINOv3, disabling iBOT in the final stage gives the weakest performance across all four metrics; increasing its weight toward 1 raises mIoU from approximately 41 to 44 for both strategies.Under WSD, final-stage weights from 0.75 to 1 produce smaller, non-monotonic changes, and the fixed-depth curves apply weights throughout training rather than only in the final stage.Source: §C.1.2 · pp. 28–29

C.1.3 TRAINING BUDGET AND GROWTH HORIZON

Changing the growth horizon and training budget produces task- and schedule-dependent outcomes. Frozen-teacher recipe comparisons favor cooldowns for most metrics, while joint MAE encoder–decoder growth saves more compute at the cost of weaker segmentation and depth results.

C.1.3 TRAINING BUDGET AND GROWTH HORIZON

  • At 500k MAE steps, spreading Stacking growth across the full budget yields higher k-NN accuracy than early growth, but other task gains depend on schedule.Under cosine, full-budget growth uses 79.6% of baseline FLOPs and favors k-NN while early growth improves all four metrics; under WSD, full-budget growth improves all four over baseline but has higher RMSE than early growth.Source: §C.1.3 · pp. 29–30, Table 15 · p. 30
  • For DINOv3, extending the final full-depth stage to 160k steps at matched baseline FLOPs raises mIoU by 0.13 for MIDAS and LIDAS but worsens other reported metrics; shortening MIDAS's growth horizon improves classification and segmentation with schedule-specific depth effects.The 100k-horizon runs retain 125k training steps and use 78.0% of baseline FLOPs, versus 72.5% for the original Prop-1 horizon.Source: §C.1.3 · pp. 30–31, Table 16 · p. 30

C.1.4 DISTILLATION FROM A PRETRAINED TEACHER

  • At 5 × 10^-4 and 125k steps, the 1 −√s cooldown gives the strongest listed frozen-teacher student results for k-NN, linear accuracy, and segmentation.Compared with no cooldown, linear accuracy rises from 81.69% to 82.96%, mIoU from 41.35% to 43.46%, and RMSE falls from 0.4854 to 0.4672; the lowest student RMSE instead comes from a lower-rate run without cooldown.Source: §C.1.4 · p. 31, Table 17 · p. 31

C.1.5 JOINT GROWING OF MAE ENCODER AND DECODER

  • Joint MAE encoder–decoder growth reduces compute to 73.3% of baseline FLOPs, but lowers mIoU from 27.73 to 26.16 and raises RMSE from 0.5855 to 0.6356 versus encoder-only growth.It improves k-NN accuracy by 2.44 points while linear accuracy decreases by 0.56 points; encoder-only growth gives a more favorable balance across downstream tasks.Source: §C.1.5 · pp. 31–32, Table 18 · p. 32

C.2 ADDITIONAL LAYER-WISE DOWNSTREAM RESULTS

Additional layer-wise evaluations show that growing often makes useful representations accessible earlier, but the advantage depends on strategy, metric, and training duration. The section also identifies growth strategies, scales, and teacher settings for further computational-pattern analyses.

C.2 ADDITIONAL LAYER-WISE DOWNSTREAM RESULTS

  • For MAE, Stacking gives the clearest intermediate-layer advantages across ViT-L and ViT-H, while MIDAS and LIDAS improve intermediate linear classification and segmentation but can narrow or reverse later.For ViT-H, Stacking leads linear classification and segmentation through much of the middle network, whereas LIDAS gives the strongest final-layer results; MIDAS and LIDAS do not share Stacking's broad k-NN advantage.Source: §C.2 · pp. 32–33
  • After 500k MAE steps, Stacking's intermediate-layer advantage is weaker and its profiles align more closely with the baseline, though it retains a localized k-NN lead and final-layer gains.Both models show similar late-layer non-monotonicity in segmentation and depth estimation.Source: §C.2 · pp. 33–34, Figure 13 · p. 33
  • In DINOv3 self-distillation, MIDAS and LIDAS show early and intermediate-layer advantages that largely converge by the final layer, even where final-layer scores are slightly below baseline.With a frozen teacher, grown students generally reach stronger classification and segmentation readouts earlier, while the teacher has stronger classification and segmentation across much of the later network.Source: Figure 14 · p. 33, §C.2 · p. 34

C.3 ADDITIONAL COMPUTATIONAL-PATTERN ANALYSES

  • The additional computational-pattern analyses cover growth strategies, model sizes, training budgets, and teacher settings, using attention statistics, layer-skipping interventions, and qualitative spatial feature maps.These comparisons are linked to the layer-wise downstream results.Source: §C.3 · p. 34

C.3.1 MAE: GROWING STRATEGIES, MODEL SCALE, AND TRAINING DURATION

MAE growth produces recurring four-layer spatial-attention cycles across strategies and model sizes, while strategies differ in CLS–patch allocation and in how they redistribute layer influence. These patterns persist after extended training, although their strength and downstream relevance vary.

Spatial attention across strategies and sizes

  • Across ViT-L and ViT-H, Stacking, MIDAS, and LIDAS repeatedly alternate between broader and more localized attention, unlike the fixed-depth baselines’ less aligned variation.Distance and entropy peak near layers 5, 9, 13, 17, and 21, while NMI peaks near layers 7, 11, 15, and 19; the pattern also appears in ViT-H’s additional layers.Source: §C.3.1 · p. 34, Figure 15 · p. 35
  • The shared spatial cycles do not predict which strategy performs best: MIDAS and LIDAS show distinct attention extremes, while downstream performance differs across strategies.MIDAS has a deep entropy minimum and large NMI peak near ViT-L layer 7, while LIDAS has a pronounced concentration near ViT-H layer 15.Source: §C.3.1 · p. 34, Figure 15 · p. 35

CLS–patch attention exchange

  • Stacking reduces patch-to-CLS mass and raises CLS-to-patch mass across much of the middle layers, whereas MIDAS often shifts both quantities oppositely and LIDAS is mixed.ViT-H CLS-to-patch profiles can oscillate sharply, particularly under MIDAS; patch-to-CLS mass shows no consistent strategy-wide increase or decrease.Source: §C.3.1 · pp. 34–36, Figure 16 · p. 35

Persistence after extended training

  • At 500k steps, Stacking retains four-layer attention organization while the baseline develops a localized, low-entropy, high-NMI region near layers 8–9.Stacking’s intermediate-layer downstream advantage shrinks after extended training, but its CLS-to-patch allocation remains higher around layers 9–14.Source: Figure 15 · p. 35, Figure 16 · p. 35, §C.3.1 · p. 36

Propagation of layer-skipping effects

  • Growing shifts layer-skipping effects toward stronger early-layer influence and weaker effects from many late interventions, with distinct patterns by strategy, scale, and training duration.For ViT-H, increases are broad for Stacking and concentrated earlier for MIDAS and LIDAS; after 500k steps, Stacking’s earliest intervention remains influential while other changes are more selective.Source: §C.3.1 · p. 36, Figure 17 · p. 36, Figure 18 · p. 37

C.3.2 DINOV3: GROWING SCHEDULES AND TEACHER SETTINGS

DINOv3 growth changes attention progression and layer influence under both self-distillation and frozen-teacher distillation. Additional examples show earlier or more persistent object-related spatial similarity patterns for LIDAS, especially in CLS–patch maps.

C.3.2 DINOV3: GROWING SCHEDULES AND TEACHER SETTINGS

  • Under self-distillation with Prop-1, MIDAS and LIDAS show localized, query-dependent attention in early layers, ahead of the baseline’s strongest localized phase around layers 8–11.After this early phase, distance and entropy generally rise and NMI falls; CLS-to-patch attention recovers earlier but ends below baseline under Prop-1.Source: §C.3.2 · p. 37, Figure 20 · p. 38
  • With frozen-teacher distillation, MIDAS and LIDAS exhibit four-layer attention oscillations, while the teacher’s progression is smoother and the fixed-depth student’s peaks are less regularly aligned.LIDAS reaches an NMI peak of approximately 0.5 near layer 6; the grown students also assign less patch-to-CLS attention than the teacher across much of the middle layers.Source: §C.3.2 · pp. 37–38, Figure 20 · p. 38, Figure 21 · p. 39
  • Layer-skipping comparisons show redistributed intervention effects; frozen-teacher grown students improve depth estimation over the fixed-depth student but have slightly lower linear classification accuracy.The self-distilled MIDAS comparison also shows stronger effects from early interventions at subsequent middle-layer readouts and smaller effects for many later interventions.Source: §C.3.2 · pp. 39–40, Figure 22 · p. 39

C.3.3 QUALITATIVE SPATIAL FEATURE MAPS

  • Across four additional examples, LIDAS develops earlier or more persistent object-associated spatial similarity patterns, with the clearest differences generally appearing in CLS–patch maps.Examples show clearer patterns around layers 12–16, while patch-to-patch maps are comparatively similar between models.Source: §C.3.3 · pp. 40–41, Figure 23 · p. 40

C.4 ROBUSTNESS TO MISSING VISUAL EVIDENCE

Growing’s robustness to missing visual evidence varies by model strategy and task: MAE Stacking shows clear classification advantages under masking, while DINOv3’s random-mask curves remain closely aligned. Semantic masking reveals additional strategy-specific differences.

Random input masking

  • For MAE ViT-L at 50k steps, Stacking has the clearest random-mask classification advantage; for ViT-H, MIDAS and LIDAS fall below baseline across much of the moderate-to-high masking range.Segmentation retention differences are smaller; for ViT-H, Stacking has the strongest retention while MIDAS and LIDAS generally do not improve on baseline.Source: §C.4 · p. 41, Figure 25 · p. 42
  • At 50% random masking after 500k steps, MAE Stacking retains approximately 75% of clean accuracy, versus approximately 52% for the baseline.Their clean scores differ by only 0.84 percentage points, while Stacking also retains more segmentation performance across much of the intermediate masking range.Source: §C.4 · p. 42, Figure 25 · p. 42
  • DINOv3 baseline, MIDAS, and LIDAS classification curves remain closely aligned under random masking, while segmentation shows small grown-model advantages and similar retention.Classification drops sharply only at the highest masking ratios, whereas segmentation deteriorates more steadily.Source: §C.4 · p. 42, Figure 26 · p. 42

Semantic input masking

  • For MAE ViT-L at 50k steps, target-retained ImageNet-S accuracy rises from 34.20% to 42.58% with Stacking, and retention rises from 52% to 62%.Stacking also reaches 48.12% versus 42.12% for the ViT-H baseline and 63.70% versus 57.80% at 500k steps; other MAE strategies have mixed results.Source: §C.4 · p. 43, Table 19 · p. 43
  • With the ADE20K target masked, DINOv3 mIoU rises from 34.43 to 38.27 for MIDAS and 38.72 for LIDAS, gains exceeding their clean-score advantages.Retention increases from 77% to 84% for both; both strategies also improve target-retained performance over the baseline’s 11.82 mIoU.Source: §C.4 · p. 43, Table 19 · p. 43

Overall

  • Robustness gains vary across strategies and tasks, but improved relative retention shows they are not explained solely by higher clean scores.The clearest reported gains are MAE Stacking classification under both masking types and grown DINOv3 under semantic masking.Source: §C.4 · pp. 43–44

C.5 MAE ADAPTATION THROUGH FINE-TUNING AND LARGER HEADS

MAE growth advantages can persist with nonlinear frozen-feature heads and transfer to fine-tuning, particularly after shorter pretraining. Representation and weight analyses show strategy-specific patterns, with weight signatures of copying weakening after extended training.

C.5 MAE ADAPTATION THROUGH FINE-TUNING AND LARGER HEADS

  • For MAE ViT-L at 50k steps, Stacking’s linear-probe advantage persists with nonlinear heads: gains over baseline are 2.87 points with MLP-2 and 2.84 points with MLP-3.At 500k steps, gains remain 0.97 and 0.90 points with the two MLP heads.Source: §C.5 · p. 44
  • For MAE ViT-H at 500k steps, LIDAS improves over baseline by 2.91 points with the linear head, 3.32 with MLP-2, and 2.94 with MLP-3.Increasing head capacity does not monotonically improve accuracy: MLP-3 falls below the linear probe for every listed model at 500k steps.Source: §C.5 · p. 44
  • After 50k pretraining steps, MAE ViT-L Stacking fine-tunes to 83.77% versus the baseline’s 82.24%, while ViT-H Stacking reaches 85.14% versus 84.64%.After 500k steps, fine-tuned scores are closely matched despite frozen-readout differences, including 86.79%–86.84% across MAE ViT-H models.Source: §C.5 · p. 45

C.6 REPRESENTATION AND WEIGHT SIMILARITY

  • At 50k steps, MAE Stacking has stronger intermediate CLS alignment than baseline; at 500k, both show broad intermediate alignment, while final CLS representations remain more distinct for Stacking.DINOv3’s overall representation-similarity progression remains broadly similar across baseline, MIDAS, and LIDAS, with localized changes and broader depth-range similarity for patch tokens.Source: §C.6 · p. 45, Figure 27 · p. 45, Figure 28 · p. 46
  • Weight alignment peaks at MAE groups 4–5 for Stacking and groups 2–3 for MIDAS and LIDAS; DINOv3 MIDAS and LIDAS also peak in middle groups.These alignments match the source–copy pairs at the final growth event, leaving a visible signature of where copying occurred.Source: §C.6 · pp. 45–46, Figure 29 · p. 46, Figure 30 · p. 47
  • After extended MAE training, Stacking’s strong later-group weight alignment weakens and its similarity patterns become more like baseline, while recurring attention patterns remain visible.The weight signature fades more than the computational pattern, suggesting that growing can leave lasting computation patterns as copied parameters become less aligned.Source: §C.6 · p. 46, Figure 31 · p. 47
Loading 2610.16817v1…