PaperlayerView exact revision on arXiv

Source-linked AI summary

Procedural Core: A Compact Recurrent Initialization for Vision Transformers

Zachary Shinnick, Christian Internò, Hemanth Saratchandran, Anton van den Hengel, Damien Teney

arXiv:2609.37631v1cs.LGcs.CV

TL;DR

Procedural-data pretraining can provide useful structure, but its dedicated training stage must be repeated for each target model. Procedural Core trains a compact recurrent transformer on procedural data and expands its weights to initialize models of arbitrary size. The authors report improvements across vision and language tasks, including a +2.2 percentage-point ImageNet top-1 accuracy gain for ViT-Base over random initialization.

  • Problem

    Procedural-data pretraining requires a dedicated stage that must be repeated for each target model, while the learned structure may admit a compact, transferable representation.

    Source: §1 · pp. 1–2

  • Method

    Procedural Core trains a small recurrent model on procedural data, then expands its compact weights to initialize transformers of arbitrary size.

    Source: §1 · p. 2

  • Results

    Procedural Core improves supervised and self-supervised vision and language modeling; for ViT-Base, ImageNet top-1 accuracy improves by +2.2 percentage points over random initialization.

    Source: §1 · pp. 2–3, §6 · p. 9

  • Takeaways & Limitations

    A compact set of weights can provide a deterministic alternative to random initialization across the evaluated vision and language settings.

    Source: §6 · p. 9

    Experiments focus on standard ViTs; with the heavily optimized DeiT-III recipe, ImageNet classification gains disappear, though segmentation and localization improvements persist.

    Source: §6 · p. 9

Abstract

from arXiv · show

Transformers are typically trained from random initialization, requiring all their capabilities to emerge from large-scale optimization. Recent work showed that a small amount of abstract procedurally generated data can help acquire generic inductive structure at low cost. However, this adds a pretraining stage that must be repeated for every target model. We propose Procedural Core, an initialization strategy that captures this generic structure into a compact set of weights that can be reused across models. We train a minimal recurrent transformer on procedural data, then expand its weights to initialize transformers of arbitrary width and depth. The resulting initialization improves performance on image classification, self-supervised visual learning (DINO), and modeling natural language (FineWeb-Edu) and code (CodeParrot). For image classification, expanding a 1M-parameter core to initialize an 85M-parameter ViT-Base improves ImageNet top-1 accuracy by 2.2 pp over standard random initialization. Our analysis identifies recurrence as essential for learning compact weights that transfer across models. In ViTs, we localize a key benefit in the suppression of high-norm tokens that produces substantial improvements in zero-shot segmentation (ImageNet-S mAP 32.3 to 42.9), object localization (VOC07 CorLoc 9.9 to 18.4), and depth estimation (NYUv2 RMSE 1.104 to 0.998). This demonstrates that transformers need not start from a blank slate, and can be initialized with generic capabilities at low cost with no domain- or task-specific data.

1 INTRODUCTION

Procedural Core aims to make procedural pretraining reusable rather than repeating it for each target model. The paper trains a compact recurrent model on abstract data, expands its weights to initialize larger transformers, and reports benefits across vision and language tasks.

1 INTRODUCTION

  • Existing procedural-data strategies require a dedicated training stage for each target model, motivating the question of whether their learned structure can be compressed into a transferable core.Source: §1 · pp. 1–2
  • The authors hypothesize that depth-wise recurrence can consolidate useful computations into fewer reusable blocks.They cite prior work suggesting that deep transformers may approximate compact recurrent programs.Source: §1 · p. 2
  • The method trains a small recurrent model on procedural data, then expands its compact weights to initialize transformers of arbitrary size.Parameter sharing is presented as a regularizer that encourages compact computations for reuse.Source: §1 · pp. 2–3
  • A ∼1M-parameter core initializing an 85M-parameter ViT-Base improves ImageNet top-1 accuracy by +2.2 percentage points over random initialization, with gains also reported for DINO and language models.Reported language-model gains cover natural language and code.Source: §1 · pp. 2–3
  • Recurrence spreads computation across more directions, while the attention value/output pathway suppresses high-norm token outliers in ViTs.The paper connects this pathway to better segmentation, localization, and depth estimation.Source: §1 · p. 3
  • ImageNet-S mAP rises 32.3→42.9, VOC07 CorLoc 9.9→18.4, NYUV2 RMSE falls 1.104→0.998, and ADE20K mIoU rises 26.6→28.8.Source: §1 · p. 3

2 RELATED WORK

Prior work studies transformer initialization, procedural-data pretraining, and recurrence as ways to shape model structure. Procedural Core combines these threads by using recurrence to distill procedural training into weights reusable across target models.

2 RELATED WORK

  • Unlike approaches that transfer pretrained-model weights or handcrafted patterns, this method constructs compact transferable weights through recurrence and abstract-data training.Source: §2 · p. 3
  • Procedural pretraining can improve data efficiency and generalization, but prior approaches require a new training phase; this work amortizes that phase in reusable weights.Source: §2 · p. 3
  • Abstract visual and non-visual data have been studied for vision training, and this work adds evidence for basic computations shared across tasks and modalities.Source: §2 · p. 3
  • Prior transformer work uses parameter sharing for memory efficiency or iterative computation; here, depth recurrence produces weights that can initialize models of arbitrary depth.Source: §2 · p. 3

3 PROPOSED METHOD

Procedural Core trains a small recurrent ViT on abstract sequences and reuses its learned weights to initialize target models. The method preserves the learned structure while expanding it across target depth and width.

3 PROPOSED METHOD

  • Procedural Core replaces repeated procedural warm-up with one trained auxiliary model whose weights initialize target models in a deterministic expansion step.Training uses nested compositional token sequences and masked prediction of closing tokens with unique valid completions.Source: §3 · p. 4

3.1 LEARNING THE CORE WEIGHTS

  • The auxiliary ViT-Tiny ties blocks 2 through L−1 while leaving the first and last blocks untied; recurrence regularizes against layer-wise specialization.The authors report degraded performance in initial experiments when all layers were tied.Source: §3.1 · p. 4

3.2 EXPANDING THE CORE WEIGHTS TO ARBITRARY DEPTH AND WIDTH

  • Depth expansion copies the first and last blocks and replicates the recurrent middle block; width expansion tiles and Frobenius-rescales core matrices.Replicated middle blocks are untied during target training; biases and normalization scales use 0 and 1 padding, and concatenated attention weights are split into Q, K, and V.Source: §3.2 · p. 4

4 EXPERIMENTS

Procedural Core improves supervised ImageNet classification, DINO representation learning, and language modeling on natural-language and code data.

4.1 SUPERVISED VISION TRANSFORMERS

  • +2.2% top-1 accuracy over default initialization for ViT-Base on ImageNet-1K, with Procedural Core also outperforming the other tested alternatives.The advantage persists throughout training rather than providing only an initial head start.Source: §4.1 · p. 5, Figure 2 · p. 5

4.2 SELF-SUPERVISED VISION TRANSFORMERS

  • +0.6% k-NN accuracy at epoch 100 and +0.3% at epoch 300 with DINO on ImageNet-1K.The advantage narrows but persists through training.Source: Figure 3 · p. 5, §4.2 · p. 6

4.3 TRANSFORMER-BASED LANGUAGE MODELS

  • About 4% lower validation perplexity in both natural-language and code modeling than default random initialization.The models were trained on 2B tokens in each domain.Source: §4.3 · p. 6, Figure 4 · p. 6

5 ANALYSIS OF PROCEDURAL CORE

The analysis links transfer to recurrence and a broader spectrum of useful directions, while identifying suppression of high-norm tokens as a key visual mechanism associated with improved dense-prediction performance.

5.1 ARCHITECTURAL CHOICES AND ABLATIONS

  • A parameter-matched non-recurrent auxiliary model improves CIFAR-100 accuracy by +1.7%, versus +4.7% for the recurrent model; transfer peaks at three unique blocks.Figure 5 also reports effective depth expansion across tested target depths and a further benefit from tiling over zero-padding for width expansion.Source: §5.1 · p. 6, Figure 5 · p. 7

5.2 SPECTRAL STRUCTURE AND ITS FUNCTIONAL IMPORTANCE

  • Procedural Core weights have slower spectral decay than direct-warm-up weights, and rank truncation degrades their transfer more sharply.The results indicate that transferable computations depend on a broader set of singular directions and that this broader spectrum is functionally important.Source: §5.2 · p. 7, Figure 6 · p. 7, Figure 7 · p. 7

5.3 HOW DOES PROCEDURAL CORE ALTER COMPUTATIONS AFTER TRAINING?

  • On 200 ImageNet-1K images, Procedural Core reduces peak token-norm bimodality from 1.29 to 0.63, high-norm tokens from 2.20% to 1.39%, and their peak attention mass from 50.4% to 17.7%.High-norm outliers are mitigated from the middle blocks onwards; shuffling V/proj. restores the default high-norm-token behavior, while shuffling Q/K preserves the effect.Source: §5.3 · pp. 7–8, Figure 9 · p. 8, Figure 8 · p. 8

5.4 IMPROVED REPRESENTATIONS BENEFIT A VARIETY OF VISUAL TASKS

  • Procedural Core improves all four frozen-backbone evaluations, with particularly large gains on ImageNet-S and VOC07, and matches or exceeds direct warm-up on all four primary metrics.With the stronger DeiT-III recipe, it does not improve ImageNet-1K classification but retains gains in zero-shot segmentation and unsupervised object localization.Source: §5.4 · p. 9, Figure 11 · p. 9

6 DISCUSSION

Procedural Core compresses the benefits of abstract-data training into reusable weights, with recurrence acting as a regularizer that supports reuse. The discussion also identifies open questions about procedural data and limits the evidence to standard ViTs and current training settings.

6 DISCUSSION

  • The core replaces a dedicated procedural-training phase with reusable weights for initializing models of arbitrary size.The authors frame this as making the benefits of abstract data more accessible, including improved convergence and generalization.Source: §6 · p. 9
  • Depth-wise recurrence regularizes the model by forcing generic computations into compact weights, which the analysis finds more reusable.Source: §6 · p. 9
  • Results across vision and language tasks support broad effects from training on abstract data, while which data properties produce useful task-specific circuits remains an open question.Source: §6 · p. 9
  • Experiments focus on standard ViTs; larger scales and other architectures remain future work, and optimized DeiT-III removes ImageNet classification gains while segmentation and localization improvements persist.Source: §6 · p. 9

AI USE STATEMENT

  • Generative AI assisted only with grammar and presentation, not the core ideas, experimental design or execution, or results analysis.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The authors state that detailed methods and evaluation procedures are provided in the paper and appendix, and that code will be released upon publication.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 PROCEDURAL CORE TRAINING SETUP

The vision Procedural Core is trained as a recurrent auxiliary model on procedurally generated sequences, with shared intermediate layers. The appendix also specifies the supervised ImageNet and CIFAR-100 fine-tuning protocols.

A.1 PROCEDURAL CORE TRAINING SETUP

  • A ViT-T/16 auxiliary model is trained for 15,000 steps on procedurally generated k-Dyck sequences using masked-token prediction on one NVIDIA H200 GPU.Source: §A.1 · p. 14
  • The auxiliary model has distinct input and output layers, while its intermediate layers share parameters.Source: §A.1 · p. 14
  • Cross-instance sharing uses three parallel instances with different random seeds, but provides only marginal and inconsistent gains; attention and MLP weights are shared while LayerNorm and head parameters remain independent.Source: §A.1 · p. 14
  • Procedural sequences use 64 opening-token types and 64 corresponding closing-token types, with an opening token sampled at p_open = 0.6 whenever structurally permissible.Source: §A.1 · p. 14

A.2 SUPERVISED IMAGE CLASSIFICATION

  • ImageNet-1K training follows a 300-epoch ViT-B protocol with AdamW, cosine decay, a 4096 batch size, and a 50-epoch linear warm-up; CIFAR-100 fine-tuning uses batch size 512 for 50 epochs with five warm-up epochs.ImageNet runs used the same configuration and effective batch size across runs; CIFAR fine-tuning used a base learning rate of 5×10^-4.Source: §A.2 · p. 14

A.3 SELF-SUPERVISED TRAINING (DINO)

The appendix describes DINO training and evaluation, language-model core construction and training, and downstream evaluations. Procedural Core outperforms procedural warm-up on CIFAR-100 and achieves the highest reported accuracy after ImageNet-1K pretraining.

A.3 SELF-SUPERVISED TRAINING (DINO)

  • DINO uses a ViT-S/16 backbone with multi-crop augmentation and momentum-teacher updates; representation quality is evaluated with weighted k-NN and linear evaluation.The k-NN protocol compares L2-normalized features by cosine similarity and uses k = 20 neighbors with temperature τ = 0.07.Source: §A.3 · pp. 14–15

A.4 PROCEDURAL CORE FOR LANGUAGE MODELS

  • For language-model core construction, a 12-layer GPT-2-style transformer is autoregressively trained on k-Dyck sequences, with distinct input and output layers and shared intermediate blocks.Source: §A.4 · p. 15

A.5 LANGUAGE MODELING TRAINING DETAILS

  • GPT-2-style models are trained for 2B tokens on FineWeb-Edu and CodeParrot, using context length 1024; CodeParrot sequences come from its clean splits.Source: §A.5 · p. 15

B.1 COMPARISON WITH PROCEDURAL WARM-UP

  • 74.6% versus 73.9% ViT-Tiny top-1 accuracy on CIFAR-100: Procedural Core improves over procedural warm-up while avoiding procedural training of the target model.Source: §B.1 · p. 15, Table 3 · p. 16

B.2 DOWNSTREAM TRANSFER AFTER IMAGENET PRETRAINING

  • Procedural Core achieves the highest downstream accuracy after fine-tuning ImageNet-1K-pretrained ViT-Base checkpoints on CIFAR-100.Source: §B.2 · p. 15, Table 4 · p. 16

B.3 LINEAR EVALUATION OF DINO REPRESENTATIONS

  • Linear-probe results differ only marginally from k-NN results; the authors attribute this to a trainable classifier partly compensating for weaker representations.They describe k-NN as a more direct reflection of intrinsic representation quality.Source: §B.3 · p. 16

B.4 SPECTRAL ANALYSIS DETAILS

The appendix explains cumulative spectral energy as a measure of how broadly singular directions capture a weight matrix’s energy. Recurrently parameterized layers show slower spectral decay across attention and MLP components.

B.4 SPECTRAL ANALYSIS DETAILS

  • Cumulative spectral energy E(k) tracks energy captured by the top k singular directions; slower saturation indicates energy distributed across more directions.The analysis also reports the number of directions needed to capture 90% and 95% of total spectral energy.Source: §B.4 · p. 16
  • Across QKV, projection, FC1, and FC2 components, recurrently parameterized layers exhibit slower spectral decay than independently parameterized baselines, consistently across depth.Source: §B.4 · p. 16

B.5 QUALITATIVE TOKEN-NORM ANALYSIS

  • Figure 13 supplies additional qualitative examples of the token-norm dynamics discussed in the main text.Source: §B.5 · p. 16

B.6 TOKEN-NORM ANALYSIS DETAILS

The analysis tracks patch-token norms across depth and uses separate procedures to detect high-norm outliers and measure the attention they receive. It also compares layer-wise singular-value spectra under recurrent and independent parameterization.

B.6 TOKEN-NORM ANALYSIS DETAILS

  • Token-norm statistics pool nonzero patch-token norms across 200 fixed-seed ImageNet validation images at each block, excluding the CLS token as both query and key.Source: §B.6 · p. 16
  • High-norm token detection smooths a histogram of log10 token norms, locates its main mode, valley, and second mode, then returns a cutoff only when the separation and token fraction meet thresholds.The procedure uses 200 bins and a smoothing window of 5; it requires s ≥ 0.5 and a fraction between 0.1% and 20%.Source: §B.6 · p. 17
  • The statistic s measures the second mode’s height above the valley in orders of magnitude, while its threshold requires about a threefold separation and the fraction constraint filters smooth heavy tails.Source: §B.6 · p. 17
  • Procedural Core and default models begin with similar token-norm structure but diverge across depth as the core suppresses high-norm outliers in later blocks.The singular-value analysis also finds slower spectral decay in recurrent layers, indicating more distributed use of singular directions.Source: Figure 12 · p. 17, Figure 13 · p. 18
  • Attention mass on high-norm tokens is measured independently of the detection cutoff by marking tokens above three times each image’s median norm and summing attention received as keys.The analysis averages normalized post-softmax mass over heads, queries, and images at each block.Source: §B.6 · p. 18

B.7 DOWNSTREAM VISUAL EVALUATION

Across the downstream visual evaluation, Procedural Core ranks best overall and its gains vary across components and training recipes. Under DeiT-III, improvements persist on zero-shot segmentation and unsupervised localization, while ImageNet classification does not improve.

B.7 DOWNSTREAM VISUAL EVALUATION

  • The evaluation freezes backbones and uses distinct readouts: threshold-free attention-map mAP for IMAGENET-S, LOST corloc on VOC07, and a linear log-depth head for NYUV2.ADE20K uses a trained 1 × 1 convolution and reports mIoU; uncertainties are 95% bootstrap intervals unless otherwise stated.Source: §B.7 · p. 18
  • Shuffling V/proj reduces the benefit, falling below default on ADE20K (26.34 mIoU versus 26.56) and VOC07 (8.50 corloc versus 9.86); transplanting V/proj and LayerNorm recovers much of the gain on ADE20K, VOC07, and NYUV2.Shuffling Q/K largely preserves performance and is strongest on ADE20K and NYUV2, but is weaker on VOC07.Source: §B.7 · pp. 18–19
  • Across eight downstream metrics, Procedural Core achieves the best mean rank (2.00), significantly separated from default initialization (4.88) and the V/proj-shuffled ablation (5.12).The aggregate Friedman test rejects equal ranks across the six arms (χ2 = 18.14, p = 0.003); Figure 14 summarizes the downstream metrics.Source: Figure 14 · p. 19, §B.7 · p. 19
  • Under DeiT-III, Procedural Core adds +2.07 mAP on IMAGENET-S and +7.70 corloc on VOC07, but ImageNet classification is 82.4% versus 82.5% top-1.Figure 15 reports gains on the two attention-based tasks and broadly unchanged performance on remaining tasks.Source: Figure 15 · p. 19, §B.7 · p. 19

C EXTENDED DISCUSSION AND LIMITATIONS

The discussion reports that stronger DeiT-III training preserves gains on two attention-based tasks but not ImageNet classification. It frames standard ViT as a controlled setting for studying initialization and leaves adaptation to newer architectures and retuning training for future work.

C EXTENDED DISCUSSION AND LIMITATIONS

  • The paper focuses on standard ViT to isolate initialization effects and enable targeted mechanistic interventions in a simple, extensively studied architecture.Source: C EXTENDED DISCUSSION AND LIMITATIONS · p. 20
  • The study spans visual and language applications and identifies recurrence and attention value/output structure as important contributors to transfer; adapting the structure to variants such as XCiT remains future work.Source: C EXTENDED DISCUSSION AND LIMITATIONS · p. 20
  • With DeiT-III, gains persist on ImageNet-S zero-shot segmentation and VOC07 unsupervised localization, but not ImageNet-1K classification.The recipe changes optimization, augmentation, and regularization and is heavily optimized for image classification.Source: C EXTENDED DISCUSSION AND LIMITATIONS · p. 20
  • DeiT-III was not retuned for Procedural Core, so its outcomes do not bound the method’s potential and other optimization, augmentation, or regularization choices may better preserve or amplify its structure.Source: C EXTENDED DISCUSSION AND LIMITATIONS · p. 20
Loading 2609.37631v1…