PaperlayerView exact revision on arXiv

Source-linked AI summary

Pretraining and Test-Time Training Budgets Matter for ARC: A Controlled Study of VARC and TRM

Adrien Deliege, Marc Van Droogenbroeck

arXiv:2610.13660v1cs.LGcs.AI

TL;DR

ARC asks whether neural models can infer task-specific transformations from a few demonstrations, but differences in training resources complicate comparisons. We test VARC and TRM across controlled pretraining and TTT budgets on ARC-AGI-1. More TTT sample presentations help both models, while variant-count and pretraining effects differ by model.

Abstract

from arXiv · show

The Abstraction and Reasoning Corpus (ARC) is increasingly used to evaluate whether neural networks can infer task-specific transformations from few demonstrations. Recent ARC models differ in architecture, but also in broader training and evaluation protocols, including pretraining data, data augmentation, test-time setup, and optimization budgets, making conclusions about architecture benefits debatable. In this work, we study how the performance of the vision-inspired VARC and recursion-based TRM models is affected by pretraining and test-time training (TTT) budgets under a common inductive protocol on ARC-AGI-1. We cross five pretraining budgets with four TTT budgets for each test task. Each budget specifies a task-variant count and total sample presentations, defining the controlled training budgets. We observe that increasing TTT sample presentations improves the standard ARC pass@2 metric in all settings for both models. Increasing the TTT task-variant count consistently helps only TRM, whereas VARC often performs better with fewer variants. VARC also remains competitive without pretraining, while TRM benefits much more from it. These results show that pretraining and TTT budgets affect ARC evaluation and motivate reporting at comparable budgets.

1 Introduction

ARC tests whether a solver can infer a task-specific transformation from a few demonstrations and apply it to unseen inputs. This study examines how pretraining and test-time training budgets affect VARC and TRM under controlled conditions.

1 Introduction

  • ARC tasks provide demonstration input-output pairs and test inputs, requiring a solver to infer the transformation and predict the unseen outputs.The study focuses on learning new rules at test time without human intervention.Source: §1 · pp. 1–2
  • Because VARC and TRM differ in pretraining and TTT budgets, performance comparisons alone can obscure whether gaps reflect architecture or training resources.The study does not aim to determine whether either architecture is intrinsically better.Source: §1 · pp. 2–3
  • The study asks how VARC and TRM perform under equivalent pretraining and TTT budgets, varying task-variant count and sample presentations while leaving other protocol elements intact.A protocol includes the complete training and evaluation procedure; each budget specifies task variants and total demonstration presentations.Source: §1 · pp. 2–3
  • Increasing TTT sample presentations improves performance in every reported setting for both models, while more TTT task variants help TRM but do not reliably improve VARC’s top-ranked predictions.The findings concern budget sensitivity within the two model families, not an intrinsic architecture ranking.Source: §1 · p. 3
  • The authors encourage future ARC papers to report and compare models under fixed, explicitly stated pretraining and TTT budgets.They provide code for the study’s budget configurations.Source: §1 · p. 3

2 Background and Related Work

ARC evaluates whether solvers generalize to unseen tasks from a small number of demonstrations, and pass@2 is the primary performance measure described here. VARC and TRM use different neural approaches and test-time procedures, while this study focuses on their budget-sensitive, inductive per-task adaptation.

Background and Related Work

  • ARC has 400 training tasks and 400 test tasks; a task is solved at pass@2 when all test inputs are predicted correctly cell-wise among up to two candidates.The final performance is the proportion of test tasks solved at pass@2.Source: ARC evaluation · p. 3
  • VARC renders integer grids onto a fixed canvas and processes them with a vision-inspired architecture, then adapts independently on each test task’s augmented demonstrations.This study excludes VARC’s external RE-ARC training data to restrict the comparison to ARC training data.Source: Visual ARC models · p. 4
  • TRM recursively refines predictions over tokenized grids; prior TRM procedures encode test tasks concurrently, whereas this study uses inductive, independent per-task TTT.The TRM variant used has 7M parameters and unrolls into hundreds of forward layer passes.Source: Recursive ARC models · p. 4
  • ARC TTT adapts a model using evaluation-time demonstrations to infer a new transformation, making the adaptation setup and its budget part of the evaluation protocol.The study makes pretraining and TTT budgets explicit because TTT budgets have received less attention.Source: Test-time training (TTT) · p. 5

3 Budget Definitions and Experimental Setup

The study defines controlled budgets through task-variant counts and sample presentations, then crosses pretraining and test-time training configurations while retaining each model’s core protocol. Experiments use ARC-AGI-1 evaluation tasks and report pass@2 in the main text.

3.1 Budget Measure and Reference Configurations

  • The sample-presentation budget B is the total number of training pairs encountered, calculated as demonstrations × variants × presentations; for ARC-AGI-1 pretraining, D = 1302.For TTT, D is the evaluation task’s demonstration count; implementation-specific epoch counts are adjusted to match budget configurations across VARC and TRM.Source: Budget configuration: task variants… · p. 5

3.2 Controlled Budget Grid

  • The controlled grid crosses five pretraining configurations—including no pretraining—with four TTT configurations, yielding 20 budget configurations per model.With B fixed, increasing the task-variant count V necessarily reduces presentations per variant; TTT budgets are scaled for tasks with different demonstration counts.Source: TTT configurations · p. 6, Pretraining configurations · p. 6

3.3 Experimental Protocol

  • VARC and TRM retain their respective representations, optimizers, and decoding procedures, while the experiments vary pretraining and TTT budgets.The models have 18M and 7M parameters, respectively; VARC is trained without external RE-ARC data and TRM in an inductive setting, unlike their original papers.Source: TTT configurations · p. 6, Models · p. 7
  • Offline pretraining produces a shared checkpoint; independent per-task TTT adapts from it, generates ranked candidates, and resets before the next evaluation task.Predictions from transformed task variants are mapped back to the original setup and aggregated; the evaluation covers 400 ARC-AGI-1 tasks and reports pass@2 in the main text.Source: §3.3 · p. 7, TTT variants, aggregation and voting · p. 7, Dataset and metrics · p. 7

4 Results and Discussion

Across VARC and TRM, raising TTT sample presentations consistently improves pass@2, while task-variant and pretraining effects differ by model. The results cover ARC-AGI-1 and do not rank the architectures.

4.1 Results

  • Figure 3 reports pass@2 across all 40 pretraining and TTT budget configurations; pass@1 and oracle results are in Appendix A.Its rows vary pretraining variant count and sample-presentation budget, while columns vary those TTT budgets.Source: §4.1 · p. 8, Figure 3 · p. 8

4.2 Discussion

  • 6.13 points for VARC and 5.38 points for TRM are the median pass@2 gains when TTT sample presentations increase; this is the only uniformly beneficial factor for both models.The low-to-high comparison is 15.3 k to 300.3 k presentations, and gains occur in every comparison.Source: TTT sample-presentation budget is… · pp. 8–9, Figure 4 · p. 9
  • Moving from 51 to 1001 TTT variants improves pass@2 in nine of ten TRM comparisons and three of ten VARC comparisons.The median changes are +0.82 points for TRM and −0.37 points for VARC; VARC's oracle improves more reliably than its two-candidate score.Source: TTT task-variant count benefits mostly… · p. 9
  • 19.63% to 37.50% for TRM and 35.50% to 40.00% for VARC are the best pass@2 values without and after 51-variant/130M pretraining.ARC-task pretraining improves all 16 TRM cells and 13 of 16 VARC cells; VARC starts from the stronger no-pretraining baseline.Source: Pretraining helps especially TRM, but… · pp. 9–10
  • Increasing pretraining presentations from 40M to 130M improves VARC pass@2 in four of eight cases and TRM in all eight; more variants harm VARC in all eight comparisons.Increasing pretraining variants modestly improves five of eight TRM comparisons, so the appropriate budget depends on the model and TTT.Source: Pretraining helps especially TRM, but… · p. 10
  • High-budget TTT costs 190.75–198.37 GPU-hours for VARC and 148.67–152.62 for TRM, versus 12.46–18.27 and 9.78–13.66 at low budget.The consistent performance gain therefore comes with an order-of-magnitude increase in evaluation cost; pretraining can be reused, but TTT repeats for every evaluation task.Source: Note on compute time · p. 10
  • The study covers only ARC-AGI-1 and reports one run per result, so small performance differences warrant caution.Because VARC and TRM differ substantially in parameter count, the results measure budget sensitivity within each model family rather than ranking the architectures.Source: Limitations · p. 10

5 Conclusion

The study examines how pretraining and TTT budgets affect VARC and TRM under a controlled ARC-only inductive setup. TTT sample presentations help both models, while variant count and pretraining benefit them differently; the findings motivate comparable budget reporting, not an architecture ranking.

5 Conclusion

  • Under a controlled ARC-only inductive setup, the study finds that pretraining and TTT budgets influence performance within VARC and TRM.The original releases used external training data for VARC and a transductive setting for TRM.Source: §5 · p. 11
  • Increasing TTT sample presentations consistently helps both models, while more task variants mainly help TRM and pretraining benefits TRM more than VARC.VARC remains capable with TTT alone.Source: §5 · p. 11
  • The findings do not rank the architectures and support standardizing training budgets to assess whether performance gains may come from architecture alone.Source: §5 · p. 11

A Complete Performance Tables

Table 1 reports ARC-AGI-1 pass@1, pass@2, and oracle performance across 400 evaluation tasks, organized by pretraining and TTT budgets. Its oracle values require care because VARC and TRM produce different numbers of candidates per task variant.

A Complete Performance Tables

  • Table 1 reports pass@1, pass@2, and oracle percentages over 400 evaluation tasks.Its rows encode pretraining task-variant count and sample-presentation budgets; columns encode the corresponding TTT budgets.Source: Table 1 · p. 11, A Complete Performance Tables · p. 11
  • Pretrained rows use ARC training demonstrations without RE-ARC.Source: Table 1 · p. 11
  • VARC yields 10V candidates from V task variants, while TRM yields V; therefore oracle values are primarily comparable within each model family.Preliminary experiments with one-view VARC and ten-view TRM predictions did not seem to yield significantly different results.Source: A Complete Performance Tables · p. 12

B Compute Cost

The complete performance tables required approximately 4,904 A100 GPU-hours, with TTT accounting for most of the measured compute. Sample-presentation budgets and measured compute costs are related but distinct.

B Compute Cost

  • TTT is repeated for every evaluation task, whereas pretraining can be reused; sample-presentation budgets are therefore distinct from measured GPU-hour costs.GPU-hours also depend on the model, implementation, hardware, and inference procedure.Source: B Compute Cost · p. 12
  • Pretraining runtimes on eight-A100 nodes should be interpreted alongside model size and implementation efficiency, not as a hardware-normalized architecture comparison.Source: B Compute Cost · p. 12
  • Table 3 runtimes are means across complete 400-task TTT runs with different pretraining checkpoints, using one A100 GPU per task.Scratch pass@2 is included as a common performance reference, not averaged across runtime runs.Source: B Compute Cost · p. 12, Table 3 · p. 13
  • Approximately 4,904 A100 GPU-hours were used for successful runs; TTT required 3,722.9 GPU-hours, about 76% of measured compute.The high sample-presentation TTT configurations alone cost approximately 3,452 GPU-hours, or 93% of total TTT cost; failed jobs and development runs are excluded.Source: B Compute Cost · pp. 12–13
Loading 2610.13660v1…