PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta Models: Models that Model Learning

Peihang Li, Ziming Liu

arXiv:2610.37001v1cs.LG

TL;DR

The paper asks whether accumulated training records can support predictions of new runs before they are executed. It treats those records as numeric examples for a reusable TabPFN meta-model, reporting cross-task capabilities and lower future-loss error with trajectory context. It also explores KAN extensions and a prospective Marin Hero Run forecast.

  • Problem

    The paper asks whether accumulated training records can become a reusable model for predicting new experiments before they are executed.

    Source: §1 · p. 1

  • Method

    The authors use language-model training records as numeric examples, giving TabPFN historical context and new configurations as queries without updating its pretrained weights.

    Source: §1 · p. 2

  • Results

    Across tasks, TabPFN predicts training behavior and reuses its backbone; adding the first 20% of each target trajectory reduced run-averaged future-loss MAE by 56% across seven held-out Delphi runs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A predictive backbone can be reused across changing configurations, targets, and contextual observations while retaining task-specific representations and target processing.

    Source: §5 · p. 9

    Lower prediction error alone does not establish compute savings, and better experimental decisions still require prospective, equal-budget comparisons that count pilot runs, search, and adaptation.

    Source: §5 · p. 9

Abstract

from arXiv · show

Every neural-network training run produces not only a model, but also evidence about how models learn. Can these accumulated experiments enable us to predict and compare new training runs before executing them? We study meta-models: models that predict the learning behavior of other models. From this perspective, final-performance prediction, learning-curve prediction, and hyperparameter selection are different queries to the same training response function. We take a first, humble empirical step toward this goal: treating real language-model training records as native numeric data, we find that an off-the-shelf tabular in-context learner, TabPFN, can match or outperform a fine-tuned LLM meta-model in multiple evaluated settings. Whereas recent work fine-tunes LLMs on tokenized configurations at a cost of tens to hundreds of GPU-hours, TabPFN requires seconds of fit-and-predict time without task-specific weight updates. We argue that meta-modeling is better treated as its own numeric modality rather than as a text task for LLMs. Going one step further, augmenting TabPFN with Kolmogorov-Arnold Networks (KANs) further improves accuracy on selected prediction tasks. Finally, we pre-registered a forecast of the Marin Hero Run, a 535B-parameter public training run, whose ground truth is revealed point by point as training progresses, putting meta-model predictions to the test against training outcomes not yet observed at the time of forecasting.

1. Introduction

The introduction frames prediction of training runs as reusable modeling of learning behavior, with performance, trajectories, and experiment selection treated as queries to one function. It presents TabPFN as a numeric in-context approach, reports adaptation gains, and introduces KAN extensions and a prospective forecast.

1. Introduction

  • A meta-model predicts other models’ learning behavior; trajectory prediction and configuration selection are different queries to the same training response function.The function uses training configuration, progress, and historical observations; other measured outcomes can define further prediction tasks.Source: §1 · p. 1
  • Meta-modeling must combine accurate prediction, broad applicability, and efficient adaptation across configurations, prediction targets, and new observations.Prior scaling-law and normalized-curve work motivates this challenge, while NCPL represents a configuration-to-performance scaling law with a fine-tuned LLM (Zhang et al., 2026).Source: §1 · p. 1
  • The paper treats language-model training records as numeric input–output examples and uses TabPFN’s pretrained backbone with historical examples as context, without updating its weights.This contrasts with tokenized configuration descriptions and reuses the backbone across training questions.Source: §1 · p. 2
  • Adding the first 20% of each target trajectory reduced run-averaged future-loss MAE by 56% across seven held-out Delphi runs (Held and the Marin Community, 2026).This adaptation uses historical context without updating backbone weights.Source: §1 · p. 2
  • KANs learn inspectable configuration coordinates that improve StepLaw final-loss and curve prediction without updating TabPFN’s backbone.The introduction also describes a frozen forecast of the Marin Hero Run, evaluated against subsequently arriving observations; reliable extrapolation beyond observed conditions remains a challenge.Source: §1 · p. 2

2. TabPFN is already a strong meta-model

The section evaluates whether one frozen TabPFN backbone can reuse historical training records to predict training choices, trajectories, observables, and downstream outcomes. It also tests updating forecasts and selecting experiments as new observations arrive.

2. TabPFN is already a strong meta-model

  • TabPFN has lower final-loss MAE than paper-reported and reproduced NCPL on Marin and StepLaw ID, but beats reproduced NCPL only on Marin OOD.Paper-reported NCPL has lower OOD MAE on both corpora. The benchmark selects methods separately by ID MAE for final loss and curves; the joint-context benchmark answers 1,905 final-loss queries in 1.79 seconds on one H200, not as an end-to-end timing of Table 1 routes.Source: §2 · pp. 2–3, Table 1 · p. 3

2.1 Predicting differences between training configurations

  • Across 24 Delphi folds, mean Spearman correlation is 0.967 for TabPFN versus 0.875 for XGBoost; for five-domain mixtures, its MAE is about 20–40% below the best evaluated law at four sizes.TabPFN also has lower loss MAE in every Delphi group; parametric laws do better on two-domain mixtures and three of four RedPajama sizes, while the three-domain comparison is close.Source: §2.1 · pp. 3–4, Figure 2 · p. 4

2.2 Predicting the course of training

  • TabPFN captures time-varying learning-rate gaps and forecasts Marin-MoE curves, with OOD curve MAE about 70% below MoE-fine-tuned NCPL.The StepLaw time-varying OOD-gap error is 38% lower than a fixed-offset control; the MoE comparison uses systems as deployed, since the fine-tuned baseline lacks expert fields.Source: §2.2 · p. 4, Figure 3 · p. 5
  • For DataDecide recipes, smaller-model records support forecasts at held-out 750M–1B scales, but not for unseen recipes.Removing recipe identity increases OOD error nearly eightfold, and the evaluated recipes all occur in context.Source: §2.2 · p. 4

2.3 Predicting outcomes beyond loss

  • Ten-task OLMES macro-accuracy MAE is 1.62 percentage points for TabPFN versus 3.40 for XGBoost; it also predicts parameter- and gradient-norm trajectories, though peak timing and magnitude remain imperfect.The norm predictions capture parameter norms' rise and decline and early gradient-norm decay, but not individual gradient spikes.Source: §2.3 · p. 5, Figure 4 · p. 6

2.4 Adapting predictions and guiding experiments

  • Observing the first 20% of seven larger Delphi runs reduces average remaining-curve error by 56%, although the benefit varies across runs.Forecasts are revised by adding observed configuration–step–loss records to context, without updating backbone weights.Source: §2.4 · p. 6, Figure 5 · p. 7
  • Across 23 Delphi groups, mean excess loss is 0.0038 nats for TabPFN versus 0.0254 for XGBoost; learned StepLaw coordinates reduce larger-sweep selection regret by 80%.Selection hides candidate losses until after choosing the configuration with the lowest predicted final loss.Source: §2.4 · p. 6
  • After five StepLaw evaluations, OOD best-found regret is 0.0394% with continual context updates, versus 0.0473% with frozen context and 0.1398% with history-only predictions.All policies start from the same two random evaluations; the continual-update policy adds each selected candidate's measured loss before choosing the next.Source: §2.4 · p. 6, Figure 5 · p. 7

3. TabPFN could be better

This section tests whether learned configuration coordinates improve TabPFN's predictions and training decisions. It reports selected OOD gains and examines the fitted reference relationships.

3. TabPFN could be better

  • Two small KANs learn learning-rate and batch-size reference values from the best-performing StepLaw runs as functions of model size and token budget.Log-ratios to these references are appended to the original configuration fields; the TabPFN backbone stays frozen, and queried-run outcomes do not enter the KAN fit.Source: §3 · p. 7
  • Against default TabPFN, larger-model final-loss and curve errors decrease by 29% and 16%, while OOD selection regret falls by 80%; ID changes are inconclusive.OOD gains persist against a same-context baseline.Source: §3 · p. 7
  • The fitted learning-rate reference decreases with model size and increases with token budget, while batch size varies nonlinearly with token budget.These inspectable relationships are fitted, not universal training laws.Source: §3 · p. 7

4. A prospective forecast of the Marin Hero Run

The authors froze a TabPFN forecast of the Marin Hero Run using smaller-run ladder data and five observations from the target run, then evaluated it against later checkpoints. The forecast outperformed the official scaling law on those checkpoints, with the comparison limited by differences in information available to the predictors.

4. A prospective forecast of the Marin Hero Run

  • The forecast was frozen on 25 August 2026 after using four smaller ladder runs and five Hero Run observations through 3.84% of training.Compute, progress, and active parameter count defined future Paloma macro-loss queries; the target run has approximately 535B total and 23B active parameters.Source: §4 · p. 7
  • Figure 6 depicts centered KAN edge functions and compares StepLaw prediction and selection results with the default untuned TabPFN route.The caption identifies MAE in nats and regret in percent, and notes that TabPFN pools corpora while the KAN route uses StepLaw context.Source: Figure 6 · p. 8
  • V0’s MAE was 0.0216 on 14 later checkpoints, versus 0.0576 for the official law; using only ladder records, TabPFN’s MAE was 0.0788.The forecast also covered all 14 observations with its nominal 90% interval.Source: §4 · pp. 7–8, Figure 7 · p. 8
  • The official law did not use the target-run prefix, and scoring rules were finalized after the first two later observations arrived.Thus, the forecast was prospective, but the entire evaluation protocol was not frozen in advance; the comparison does not isolate the ladder records’ benefit beyond the prefix alone.Source: §4 · p. 8

5. Discussion and limitations

The discussion presents a shared predictive backbone as a way to address varied training questions, while retaining task-specific representations and target processing. It also identifies unresolved challenges in extrapolation and in demonstrating better experimental decisions.

5. Discussion and limitations

  • Configuration fields, prediction targets, and contextual observations can change without updating TabPFN weights, although representations and target processing remain task-specific.The authors frame this as extending meta-modeling beyond a fixed configuration-to-loss mapping.Source: §5 · p. 9
  • Three challenges remain: representing genuinely new regimes, recognizing unreliable extrapolation, and demonstrating better experimental decisions.The authors propose prospective, equal-budget comparisons that count pilot runs, search, and adaptation; lower prediction error alone does not establish compute savings.Source: §5 · p. 9

6. Related work

The related work spans structured scaling and training-dynamics laws, learned predictors, numerical in-context learning, and downstream prediction. The paper positions numerical response functions and reuse of a general tabular prior as complementary approaches.

6. Related work

  • Scaling laws cover mixtures, sparsity, and learning-rate schedules, while hyperparameter transfer and scale-dependent tuning qualify their predictions; numerical response functions are presented as complementary to structured laws.The cited work includes Ye et al. (2025), Abnar et al. (2025), Tissue et al. (2024), Luo et al. (2025), Yang et al. (2021), Wang and Aitchison (2024), and Porian et al. (2024).Source: §6 · p. 9
  • OmniPred and NCPL use language models as general regressors or training-outcome predictors; this paper asks whether a frozen numerical backbone can accommodate changing fields, targets, and contexts.The passage frames this as the paper’s question, not as a claim that text-based regression is incapable of the task.Source: §6 · p. 9
  • Bayesian surrogates and Deep Power Laws connect curve prediction to optimization, while PFN variants specialize numerical in-context learning to curves or hyperparameter search.The paper instead reuses TabPFN’s general tabular prior across training questions without pretraining a dedicated curve model.Source: §6 · p. 9
  • Downstream-prediction approaches include model ladders, low-dimensional capability spaces, and learning from accuracy trajectories and token-level losses; predictability remains setting-dependent.The cited examples are Bhagia et al. (2025), Ruan et al. (2024), Hu et al. (2026), and Lourie et al. (2025).Source: §6 · p. 9
  • KANs learn univariate edge functions, and this paper uses them to learn inspectable configuration coordinates for a frozen predictor rather than replace its backbone or establish causal training laws.The passage cites Liu et al. (2024).Source: §6 · p. 9

AI use statement

The statement describes AI assistance across research and manuscript tasks, with authors retaining responsibility for methodological choices, interpretations, review, and final content. The appendices describe the datasets, task splits, and baseline-fitting procedures used in the evaluations.

AI use statement

  • AI assistants supported hypothesis refinement, experimental design, coding, experiments, audits, analysis, figures, literature search, and manuscript preparation; authors made final methodological choices and interpretations and reviewed the work.The authors state that all experiments were approved and conducted under their supervision, and that they take responsibility for the final paper.Source: AI use statement · pp. 9–10

A.1 Datasets, prediction tasks, and splits

  • The dataset appendix describes task definitions and splits, including DataDecide’s 950 runs and 19,124 checkpoints, with 750M and 1B runs held out as OOD.DataDecide records are scored as log C4 perplexity; its remaining split is 8:2 by recipe–size group.Source: §A.1 · p. 14, Table 2 · p. 14

A.2 Baseline fitting and statistical evaluation

  • Baseline fitting uses parametric loss laws, optionally with a log learning-rate term, alongside ID-tuned XGBoost and a sparsity-aware law for MoE.The parametric form is L = E + A/N^α + B/D^β; the MoE law uses Ntot, D, and S.Source: §A.2 · p. 14

A.3 Evaluation and figure-construction protocols

The section specifies evaluation and figure-construction protocols across prediction, selection, and observables tasks, then reports results spanning held-out conditions, mixture prediction, recipe-aware forecasting, and observable trajectories. It also describes adaptation-cost accounting and main-table selection rules.

A.3 Evaluation and figure-construction protocols

  • For configuration prediction, mean Spearman correlation is 0.967/0.875/0.820/0.735 and MAE is 0.045/0.100/0.143/0.141 for TabPFN/XGBoost/quadratic ridge/kNN.The task predicts trials in 24 held-out cells; curve scoring retains every checkpoint even though displays omit progress at or below 5%.Source: §A.3 · pp. 14–15

A.4 Adaptation and inference cost

  • The adaptation-cost accounting distinguishes throughput-based and FLOPs-based NCPL training estimates, and reports a 9.9 GPU-hour TabPFN tuning search.The cost table concerns the joint-context benchmark, not the main-table final-loss routes.Source: §A.4 · pp. 15–16, Table 3 · p. 16

B.1 Delphi-HP: prediction and selection on held-out conditions

  • 0.0038 [0.0024, 0.0052] TabPFN regret versus 0.0254 for XGBoost on Delphi-HP; the equal-candidate-count check gives 0.00383 versus 0.01699.The sweep holds out one of 24 width–batch–token cells at a time; regret uses raw Paloma loss on 23 eligible cells.Source: §B.1 · p. 17

B.2 Mixture-law fitting and sensitivity

  • On five-domain mixtures, TabPFN MAE is 0.0084–0.0113 nats versus M1’s 0.0133–0.0147 and M4’s 0.0271–0.0286; M4 performs better on two-domain mixtures.The three-domain comparison is close, and M2 leads at three of four RedPajama sizes.Source: §B.2 · p. 17

B.2.1 Main-table selection and prediction settings

  • Table 1 selects each method’s lowest ID-MAE row separately for final loss and curves, without selecting on OOD scores or individual metric optima.TabPFN candidates are zero-shot variants; the selected StepLaw final-loss joint-mean and joint-median ID MAEs both round to 0.0063, but joint mean has the lower unrounded value.Source: §B.2.1 · p. 17

B.3 DataDecide: recipe information and context controls

  • Without recipe identity, DataDecide OOD final-loss MAE rises from 0.0178 to 0.1117, while recipe explains 99.2% of within-size variance.Released NCPL inputs are identical across recipes at a given size, so its recipe-free interface cannot distinguish their predictions.Source: §B.3 · p. 17

B.4 Predicting observables and their trajectories

  • The normalized parameter model improves curve error on 212 of 233 OOD runs relative to default feature typing.The section’s observable protocol predicts parameter and gradient norms from configuration and progress; illustrative runs are not claimed to be typical, and Table 4 scores all held-out runs.Source: §B.4 · p. 18, Table 4 · p. 19, Figure 9 · p. 19

B.5 DataDecide: downstream prediction from training histories

This section evaluates downstream prediction using historical training checkpoints, illustrates Marin loss-curve predictions, and tests forecasts updated with target-run observations. It also compares configuration-selection policies under a matched feedback protocol.

B.5 DataDecide: downstream prediction from training histories

  • 35% lower OOD macro error than the earlier endpoint protocol, but 1B predictions remain low by 2.44 percentage points on average.The added historical checkpoints improve all ten constituent targets, but the new context adds temporal evidence rather than larger training models; the results do not establish universally calibrated scale extrapolation.Source: §B.5 · p. 20, Table 5 · p. 20, Figure 10 · p. 20

B.6 Loss-curve examples on Marin

B.7 Updating forecasts with observations from the target run

  • The target run’s observations through cutoff p are appended to historical context, and later checkpoints are scored across seven Delphi ladder runs.Both TabPFN arms use the same frozen backbone and fixed Chinchilla offset; Table 7 reports run-averaged remaining-curve MAE across future windows.Source: §B.7 · p. 25, Table 7 · p. 25

B.8 Configuration selection and context updates

  • At budget five, conditioning on initial outcomes reduces mean regret by 0.09249 percentage points relative to history only (95% interval [0.01642, 0.17977]).The matched protocol compares history-only, initial-frozen, and updating policies; neither replay establishes prospective training-compute savings.Source: §B.8 · p. 25, Table 8 · p. 27

C.1 KAN-learned configuration coordinates

The section adds learned configuration coordinates to TabPFN and evaluates the resulting route on StepLaw prediction and selection tasks. It also reports matched-data comparisons and input and inference tuning without backbone updates.

C.1 KAN-learned configuration coordinates

  • OOD MAE differences against default TabPFN are −0.0066 for T1 and −0.0057 for T2, while OOD regret decreases by 0.343 percentage points; ID intervals include zero for all three tasks.Against the matched StepLaw-context baseline, OOD mean errors and regret also improve; the KAN route comparison with the default route changes both context and coordinates.Source: §C.1 · pp. 27–28, Table 9 · p. 28

C.2 Data efficiency and subset sensitivity

  • At 10% of training runs (323 runs), joint-context TabPFN has ID MAE 0.0197/0.0131 on Marin/StepLaw versus NCPL’s 0.0263/0.0227.With stratified subsets, TabPFN reaches 0.0121 at 30% Marin and 0.0098 at 10% StepLaw, close to full-data NCPL’s 0.0115/0.0096.Source: §C.2 · p. 28

C.3 Input and inference tuning without backbone updates

  • The 679-configuration, three-seed tuning search costs 9.9 H200 GPU-hours; the tuned joint recipe lowers Marin final-loss MAE from 0.0093/0.0173 to 0.0069/0.0146 (ID/OOD).A StepLaw-only recipe lowers MAE from 0.0063/0.0227 to 0.0051/0.0190; most of the gain comes from feature construction and selection.Source: §C.3 · p. 28, Figure 15 · p. 29

D.1 Marin Hero Run: frozen forecasts and scoring

The frozen V0 forecast was evaluated against subsequent Marin Hero Run outcomes and outperformed the official law and a ladder-only TabPFN forecast. Its completion forecast remains untested.

D.1 Marin Hero Run: frozen forecasts and scoring

  • V0 used 77 checkpoints from four smaller ladder runs and five Hero Run evaluations through 3.84% of training.Inputs were log compute, progress, and log active parameters; the target was log(L−1.5) for dropless Paloma macro loss.Source: §D.1 · p. 29
  • MAE was 0.0216 across 14 outcomes, versus 0.0576 for the official law and 0.0788 for TabPFN given only the ladder.The nominal 90% interval covered all 14 outcomes; the last scored evaluation was at step 56,999 (14.61%, loss 2.2525).Source: §D.1 · p. 29
  • V0’s completion forecast of 2.031, with interval [1.850, 2.455], remains untested.The forecast version was committed before the next evaluation; preregistration meant a version-controlled timestamp, not an independent registry.Source: §D.1 · pp. 29–30

E.1 Delphi: extrapolating beyond the observed compute range

Across five analyses, TabPFN extrapolates well on the Delphi compute ladder, while the paper also examines feature-order sensitivity, learned run embeddings, training-response sweeps, and a canonical record schema.

E.1 Delphi: extrapolating beyond the observed compute range

  • 0.0093 nats: residual-mean TabPFN’s MAE on the seven-run Delphi ladder, versus 0.0182 for the team’s prior forecast, 0.128 for XGBoost, 0.149 for Chinchilla, and 0.204 for released NCPL.Errors were 0.005/0.006/0.032 at 10^21/10^22/10^23 FLOPs; without residualization, ladder error rose to 0.053.Source: Figure 16 · p. 30, §E.1 · p. 30

E.2 Changing feature order without changing a training configuration

  • Across four splits, NCPL’s permutation-averaged MAE was 1.8–4.0× its canonical MAE and rose under every feature permutation.On Marin ID, MAE changed from 0.0115 to 0.0337.Source: §E.2 · p. 30

E.3 Row embeddings of the joint-context model

  • The joint-context embedding forms a horseshoe where the two corpora share an arc but remain locally separated, with same-source purity of 0.985.Final loss varies along the arc, and deviation from the fitted scaling law decreases along its geodesic (Spearman −0.50).Source: §E.3 · p. 30, Figure 17 · p. 31, Figure 18 · p. 32
  • Embedding distance to context tracks OOD error on Marin, StepLaw, and MoE, with Spearman correlations of 0.40, 0.39, and 0.45.The largest StepLaw models are both farthest from context and worst predicted.Source: §E.3 · p. 31, Figure 17 · p. 31

E.4 Training responses: recovering known relationships

  • TabPFN and NCPL reproduce learning-rate U-curves at all 31 anchors, the optimal-learning-rate exponent (−0.64; measured −0.643), and batch-size saturation beyond 256.The predicted batch-optimum exponent in token budget was 0.55 versus 0.62 measured; the probes were surrogate sweeps, not new training runs.Source: §E.4 · p. 32

E.5 A proposed canonical training record

  • The proposed canonical training-record schema has 133 columns: 67 values and 66 masks.It distinguishes recorded, inapplicable, and missing fields and fixes per-corpus units.Source: §E.5 · p. 32

Related papers

Loading 2610.37001v1…