PaperlayerView exact revision on arXiv

Source-linked AI summary

ArchitectureIQ: On the Measure of Training Intuition

Zirui Ren, Shaoyang Guo, Chencheng Tang, Jinxin Wang, Chengyu Xiong, Shanbin Yu, Peihang Li, Yidi Wu, Bangzhe Huang, Qingyu Qu, Leqian Yang, Ziming Liu

arXiv:2609.39714v1cs.AI

TL;DR

The paper asks whether language models can predict which training setting will work best before an experiment is run. It introduces ArchitectureIQ, a benchmark built from synthetic datasets and executable training recipes. Frontier models perform above chance, but their intuition is uneven across architectures, optimizers, and dataset changes.

  • Problem

    The paper asks whether language models can make the expert-like judgment of predicting which training setting will work best before an experiment is run.

    Source: §1 · p. 1

  • Method

    ArchitectureIQ presents synthetic datasets and candidate training recipes, executes the recipes to establish a ground-truth winner, and asks test-takers to predict it.

    Source: §1 · p. 1, §3 · p. 4, §3.1 · p. 4

  • Results

    Frontier models outperform chance and human experts overall, but optimizer comparisons are easier than architecture comparisons and predictions often fail to reverse when datasets change.

    Source: §6 · p. 9, §7 · p. 10

  • Takeaways & Limitations

    Executed outcomes can turn model-proposed principles into evidence-weighted propositions that transfer to weaker solvers without parameter updates, but do not raise the strongest solver beyond its existing ceiling.

    Source: §7 · p. 10

    Synthetic datasets, short training horizons, and a finite family registry do not capture the scale or data properties of modern large-model training runs.

    Source: §6 · p. 9

Abstract

from arXiv · show

Top researchers have good intuition, but do language models have as good intuition about model training as top AI researchers? To measure model intuition of LLMs and humans, we introduce the ArchitectureIQ benchmark. Each question presents a synthetic dataset and several training recipes, and the test-taker is asked to predict the recipe yielding the best test metric. Overall, we find that LLMs' model intuition is good but has four limitations: (1) The intuition is imperfect, or even sub-human in some cases. Frontier models achieve around 76% accuracy (random choice 33%) vs best human researcher (66.0%), yet remain far from perfect. For architecture-only questions, best human achieves 65% while GPT-6 Astra only has 38%. (2) The intuition is empirical, not structured, supported by the fact that more CoT compute does not lead to substantial improvement. Unlike math, we still lack a "Science of AI" language that enables structured reasoning on AI. (3) The intuition is not maximally condensed, and can be further compressed into a knoledge base. Our constructed knowledge base with only 20 items yields large gains for weak models: GPT-4o equipped with the accumulated knowledge almost matches the performance of Claude Opus 5. (4) The intuition is insensitive to dataset properties, but the best model should in general depend on data properties. This suggests that data is the real "dark matter" in AI -- LLMs (so do human researchers) understand too little about data, even less than model architectures.

1. Introduction

Expert researchers can often anticipate which training changes will work, and this judgment matters for efficient research. The paper asks whether language models can make similar predictions before running an experiment.

Benchmark design

  • ArchitectureIQ asks whether LLMs can predict the best training recipe from a synthetic dataset and candidate recipes without relying on prior knowledge of real tasks.Recipes are rendered as executable programs and run to establish ground truth; synthetic data is used to separate modeling capability from prior knowledge.Source: §1 · pp. 1–2, Figure 2 · p. 3

Benchmark results

  • Around 76.0% accuracy on the static benchmark for frontier models exceeds 33.3% for random choice and 66.0% for the best researcher on a 50-question subset, but remains imperfect.On architecture-only questions, GPT-6 Astra achieves 38% accuracy versus 65% for the best human.Source: Figure 1 · p. 2, §1 · p. 2

Findings

  • LLMs’ intuition is empirical rather than structured, can be compressed into a knowledge base, and often ignores dataset properties that should affect the best recipe.The paper reports that GPT-4o with a knowledge base containing only 20 items matches the raw performance of Claude Opus 5.Source: §1 · p. 2

Contributions

  • The paper introduces what it describes as the first benchmark for measuring model intuition in LLMs and humans, and proposes compressing reasoning traces into a transferable knowledge base.The knowledge base is described as more compressed and transferable than raw chain-of-thought.Source: §1 · pp. 2–3

2. Related Work

Prior work studies experiment selection through optimization, learned performance prediction, and language-model agents. ArchitectureIQ instead isolates whether a general-purpose language model can rank training recipes before observing their execution.

Training-performance prediction and neural architecture search

  • Bayesian optimization, multi-fidelity search, learning-curve extrapolation, and neural architecture search use executions or task-specific search data, whereas ArchitectureIQ tests recipe ranking before any run.The paper situates ArchitectureIQ alongside reproducible tabular and surrogate NAS benchmarks.Source: §2 · p. 3

Language agents for machine-learning research

  • General agent benchmarks and ML research agents assess broader workflows that may include tool use, execution, and feedback; ArchitectureIQ targets the earlier choice of a promising experiment.The paper frames this choice as a prerequisite to paying the experiment’s execution cost.Source: §2 · p. 3

Reasoning and experience accumulation

  • Unlike methods that accumulate textual feedback or executable skills, ArchitectureIQ scores natural-language propositions against executed training outcomes to measure knowledge-base transfer independently of parameter updates.The related work discusses chain-of-thought, self-training, environment interaction, reflection, and external skill libraries as approaches to reasoning or experience accumulation.Source: §2 · p. 4

3. ArchitectureIQ: A Generative and Verifiable Probe

ArchitectureIQ tests whether models and humans can predict training outcomes from synthetic datasets and candidate recipes. Models outperform chance and human researchers overall, but their judgments are much less reliable for architecture choices and when dataset changes alter the winner.

3. ArchitectureIQ: A Generative and Verifiable Probe

  • Each question pairs a synthetic dataset with candidate training recipes, and its winner is established by executing the generated code across 10 random seeds.The 500-question benchmark has equal portions of optimizer-only, architecture-only, and mixed questions; fully synthesized datasets prevent reliance on pretrained knowledge of specific datasets.Source: §3.1 · p. 4, §3 · p. 4

3.2 Evaluation Results

  • Leading models answer nearly one quarter of questions incorrectly, despite outperforming the 1/3 random baseline and human experts; the best human participant scores 66.0%.The mean human score is 51.0%; the best model's exact accuracy is not given in this passage.Source: §3.2 · pp. 4–5
  • Leading models get 85–87% of optimizer comparisons and 58–61% of architecture comparisons correct; GPT-6 Astra scores 38% on architecture questions.Optimizer differences offer a more reliable judgment signal when present, while architecture judgments require considering several design choices jointly.Source: §3.2 · p. 5
  • Accuracy is 81–95% when the same candidate wins across paired datasets, but 46–50% when changing the dataset changes the winner; models switch choices on only 6–23% of pairs.The results indicate that training-configuration knowledge dominates evidence about interaction with a particular dataset.Source: §3.2 · p. 5

3.3 Scaling with the Number of Choices

  • Figure 3 fits accuracy across k = 1, 2, 3, 5, and 10 choices with acc(k) = a + (1 − a)/k; R² is at least 0.94 for all four models.The paper interprets a, the fitted intercept, as judgment capability rather than accuracy on three-choice questions.Source: §3.3 · p. 5, Figure 3 · p. 6
  • Figure 4 reports no consistent accuracy gain beyond the initial reasoning budget as inference-time computation increases.The figure plots accuracy against total output tokens.Source: Figure 4 · p. 6

4. Analysis of LLMs’ intuition

The analysis finds that additional examples and longer reasoning do not reliably improve model judgments. Structured predictors capture some of the benchmark’s regularities, but do not match leading language models.

Analysis of LLMs’ intuition

  • The authors characterize model intuition as empirical rather than a structured explicit language, consistent with the lack of substantial gains from in-context learning or test-time scaling.They contrast this with a scientific language such as mathematics or chemistry that enables structured reasoning.Source: §4 · pp. 6–7

Ineffective In-context Learning

  • Few-shot prompting does not reliably improve performance, and human accuracy shows no significant rise as experts answer questions sequentially.The model comparison tests zero-shot and 10-shot performance on a subsample.Source: §4 · p. 6

Ineffective Test-time Scaling

  • Longer reasoning rarely changes models’ final preferences, and accuracy shows no consistent gain beyond the initial reasoning budget.The 50-question evaluation finds differences below roughly 7% indistinguishable from question-sampling noise; observed gains for GLM-5.2 concentrate in the first few hundred output tokens.Source: §4 · pp. 6–7

Rule-based and Structured Predictor Baselines

  • The best structured predictor, standardized logistic regression on 87 features, reaches 74.2%; a readable depth-four tree reaches 61.8%.The tree’s dominant decisions use learning-rate rank, effective-step rank, and optimizer identity; neither predictor matches leading language models.Source: §4 · p. 7, Figure 5 · p. 7

5. Online Knowledge Accumulation

The paper builds an online pipeline that collects and refines propositions from verifiable training questions. The resulting knowledge base improves a weaker solver on held-out questions, while a stronger solver remains near its original performance.

Learning Verifiable Propositions Online

  • The solver can use at most 20 propositions; Claude Opus 5 generates snapshots over eight epochs of 50 freshly generated questions each.A separate LLM curator merges duplicates and may replace broad rules with propositions that state their expected conditions.Source: §5 · pp. 7–8
  • Final knowledge-base entries are conditional, soft-quantitative statements that estimate a question-readable quantity and then predict which candidate should win.Figure 6 shows six entries from the final snapshot.Source: §5 · p. 8, Figure 6 · p. 8

Evaluation Across KB Snapshots

  • GPT-4o rises from 53.2% at KB0 to 67.6% at KB8 on the held-out evaluation.Figure 7 evaluates successive snapshots on a 50-question subset, injecting at most 20 highest-scoring propositions per snapshot.Source: Figure 7 · p. 9, §5 · p. 9
  • A small textual knowledge base improves weaker solvers on held-out questions without parameter updates, showing that some ArchitectureIQ competence transfers through propositions.The accumulated knowledge mostly externalizes capabilities Claude Opus 5 already possesses rather than establishing a new performance ceiling for that solver.Source: §5 · p. 9

Analytical Decision Tree

  • Separately from the knowledge-base experiment, an effective-learning-rate decision policy reaches 78.2% development-stage dataset-grouped out-of-fold top-1 accuracy.Source: §5 · p. 9

6. Discussion

The discussion frames ArchitectureIQ as a probe of capability between simple heuristics and exact execution, while noting that its synthetic setting is narrower than real-world experiment selection.

6. Discussion

  • ArchitectureIQ exposes a capability between simple heuristics and exact execution; performance is uneven, and predictions often remain fixed when dataset changes should reverse the answer.Structured baselines and knowledge-transfer results indicate compressibility, while few-shot examples and extra test-time computation do not remove the stated bottleneck in access to reliable training principles.Source: §6 · p. 9
  • The synthetic benchmark, short training horizons, and finite family registry do not capture the scale or data properties of modern large-model training runs.The authors note that executable ground truth and targeted interventions are advantages of this setting, and suggest extending the generator toward larger, more realistic regimes.Source: §6 · p. 9

7. Conclusion

ArchitectureIQ measures pre-execution training judgment against executed outcomes; frontier models exceed chance, but their predictions remain uneven and often fail to track dataset-dependent reversals.

7. Conclusion

  • Frontier models predict training outcomes far above chance, but their competence is uneven across optimizers, architectures, and dataset-dependent reversals.Their predictions behave more like strong but incomplete pretraining priors than faithful simulations of training.Source: §7 · p. 10
  • The online knowledge-accumulation experiment turns model-proposed principles into evidence-weighted propositions that transfer to weaker solvers without parameter updates.It does not raise the strongest solver beyond its existing ceiling.Source: §7 · p. 10
  • ArchitectureIQ provides a controlled test of pre-execution judgment and a mechanism for converting repeated experimental evidence into reusable knowledge.Source: §7 · p. 10

A. Benchmark Construction Details

  • The frozen v1.5 release contains 500 three-choice questions built from independently sampled datasets across six synthetic families and three question types.Its stated counts describe the realized release, not every configuration supported by the generator.Source: A. Benchmark Construction Details · p. 13

A.1 Dataset Families

The appendix specifies how ArchitectureIQ constructs synthetic datasets and controlled recipe comparisons, establishes ground-truth winners, and presents questions to test-takers. It also describes a proposition selector that balances empirical credibility with exploration.

A.1 Dataset Families

  • Six synthetic dataset families use fixed per-instance train/test splits shared across recipes and seeds; regression and classification are noiseless, while bigram targets are stochastic.The splits are sampled from each instance’s data-generating process.Source: §A.1 · p. 13
  • The families span Gaussian-feature tabular and XOR rules, spirals, symbolic regression, and bigram modeling, with irrelevant coordinates or varying function structure in several families.Spiral instances vary in number of turns; symbolic regression instances vary in oscillation, curvature, saturation, and local smoothness.Source: §A.1 · pp. 13–15

A.2 Question Taxonomy and Candidate Configurations

  • Each question compares three recipes on the same materialized dataset, with shared loss, batch size, training steps, and sample budget; questions vary optimizer, architecture, or both.The intended mixture is balanced across optimizer-only, architecture-only, and mixed questions.Source: §A.2 · pp. 15–16
  • Candidates vary architectures within MLPs or between Transformers and GRUs, and optimizer configurations; parameter counts limit gross size differences but do not equalize compute or runtime.Architecture-only questions hold the optimizer configuration fixed, while optimizer-only questions hold the model configuration fixed.Source: §A.2 · pp. 15–16

A.3 Ground-Truth Generation and Filtering

  • Ground truth executes each recipe for ten seeds; retained questions require one candidate’s worst loss to beat every rival’s best, a stronger filter than comparing means.This establishes empirical answer stability over the evaluated seeds, not all possible training randomness.Source: §A.3 · p. 17

A.4 Prompt Contents and Release Audit

  • Prompts provide dataset construction, split sizes, training budget, metric, recipes, and code; all 500 released questions passed checks of constraints, consistency, and metric leakage.Final scores, learning curves, and answer keys are withheld, and choice order is randomized.Source: §A.4 · p. 17

B. ArchitectureIQ Prompt Template

  • The benchmark template asks test-takers to choose the recipe with the best held-out metric after its stated training budget, without providing training results.Questions describe one dataset instance and candidate recipes.Source: B. ArchitectureIQ Prompt Template · p. 17

C.1 PUCT-Style Proposition Selection

  • The proposition selector combines Laplace-smoothed empirical credibility with a bounded exploration bonus, then injects up to 20 highest-scoring propositions into each prompt.Credibility favors propositions with stronger empirical records; exploration favors those with little credit mass.Source: §C.1 · p. 19

C.2 Representative Knowledge-Base Entries

The representative knowledge-base entries encode task- and budget-specific training heuristics, while the analytical decision policy reaches 78.2% dataset-grouped out-of-fold accuracy. The simplified flowchart itself was not independently tested as a frozen predictor.

C.2 Representative Knowledge-Base Entries

  • Across fixed short budgets, optimizer ranking tracks cumulative parameter displacement; Adagrad’s initial update advantage can give way to Adam or AdamW over longer runs.Adam and AdamW have approximate total displacement ηT, while Adagrad’s decaying steps bound displacement at approximately 2η.Source: §C.2 · p. 20
  • On continuous MSE regression, Adam can outperform unaccelerated SGD within roughly 500 steps, while Adam overtakes SGD after t* on under-trained tasks.Adam maintains effective updates near scale η after the output head aligns near the target mean; on under-trained tasks with T > t*, it tends to achieve greater displacement and lower final loss.Source: §C.2 · p. 20
  • For continuous targets with a large constant offset, momentum SGD can align the bias quickly and reduce MSE under budgets around 256 steps.Its effective learning rate is η/(1 −m), with a bias-adaptation time constant of approximately 1/(2η/(1 −m)) steps.Source: §C.2 · p. 20
  • On continuous MSE targets, momentum SGD tends to beat zero-initialized Adagrad under roughly 1,000 steps when its effective rate exceeds Adagrad’s by an order of magnitude.The comparison is between momentum SGD’s η/(1 −m) and zero-initialized Adagrad’s approximately 2ηT/T per-step rate.Source: §C.2 · p. 20
  • For short-budget XOR classification, candidate progress is largely ordered by stable effective step size because initial cross-entropy stays near chance until nonlinear interactions form.The stated scope is budgets of at most roughly 256 steps.Source: §C.2 · p. 20
  • Task structure favors deeper models for complex spiral boundaries, but a single-layer transformer for bigram lookup under short budgets.For spirals, deep normalized residual MLPs with at least three blocks improve curvature and cross-entropy reduction per step; for bigrams, a four-layer post-LayerNorm stack adds optimization overhead and backpropagation attenuation.Source: §C.2 · p. 20
  • For smooth tabular targets, a shallow wide MLP can outperform a narrow deep one; for oscillatory univariate targets, deep residual stacks fit faster than comparable shallow networks.The tabular comparison applies around 1,024 steps with AdamW near 3 × 10−3; the oscillatory-target result applies at most roughly 512 steps and adaptive rates around 3 × 10−4.Source: §C.2 · p. 21

C.3 Analytical Decision Tree

  • 78.2% dataset-grouped out-of-fold top-1 accuracy (391/500) is reached by the underlying analytical decision policy, but the simplified flowchart was not independently evaluated.The flowchart summarizes an update-regime rule set.Source: §C.3 · p. 21, Figure 8 · p. 21

D. Full Benchmark Results

The section presents complete results for the 500-question benchmark, including overall and question-type accuracies, with a human comparison drawn from the 50-question launch subset.

D. Full Benchmark Results

  • Table 4 reports overall accuracy and per-type accuracy across 170 architecture-only, 168 optimizer-only, and 162 mixed questions.The best-human row uses the 50-question launch subset: 22 architecture-only, 16 optimizer-only, and 12 mixed questions.Source: D. Full Benchmark Results · p. 21
  • Table 4 uses one shading scale across columns, from white at chance (33.3%) to darkest at 90%, so darker cells indicate stronger performance.Source: D. Full Benchmark Results · p. 21
  • Two participants tie for the best human score, and the per-type breakdown uses the participant with fully recorded item-level answers.Source: D. Full Benchmark Results · p. 21

E. Paired Label-Flipped Evaluation

The paired evaluation tests whether changing only the dataset can reverse the winning recipe while holding candidate configurations fixed. Its strict acceptance rule yields a limited set of valid pairs and directional imbalance in single-side accuracy.

E. Paired Label-Flipped Evaluation

  • Each pair uses two datasets from the same family with identical tensor shapes and a shared set of three candidates executed for ten seeds.The two sides are exchangeable; neither is designated original or counterfactual.Source: E. Paired Label-Flipped Evaluation · pp. 21–22
  • 52 pairs are retained across five task families only when each side has a fully separated 10/10 winner and the winning candidate differs across datasets.The filter is deliberately strict, and no XOR pair survives despite 13,169 sampled pairs and a subsequent nine-candidate search.Source: E. Paired Label-Flipped Evaluation · pp. 21–22
  • Single-side accuracies are directionally imbalanced because the finite accepted sample places the familiar candidate-level favorite more often on side A; paired accuracy and answer transitions are the appropriate estimands.Source: E. Paired Label-Flipped Evaluation · p. 22

F. Data-Flip Evaluation

The data-flip evaluation uses matched candidate sets across paired datasets and tests whether solvers adjust their choice when the dataset changes. The supplied passages describe the benchmark setup and visual encodings, not its outcome values.

F. Data-Flip Evaluation

  • Hard500 contains 250 data-flip pairs, while AIQ-Hard50 uses 25 pairs and 50 questions across five families.The other 225 pairs form the knowledge-base learning set and share no dataset or candidate with Hard50.Source: F. Data-Flip Evaluation · pp. 22–23
  • A solver that ignores the dataset can be correct on at most one side of each pair, capping accuracy at 50%.Each pair shares three candidates, while its two datasets have different winners.Source: F. Data-Flip Evaluation · p. 22
  • Table 6 reports AIQ-Hard50 results by overall and question-type accuracy; Table 7 summarizes paired outcomes such as same pick, both right, one right, or neither.Table 6 notes 36 of 50 valid answers for one entry and 47 of 50 for another; Table 7 notes 15 and 23 pairs with both sides answered.Source: Table 6 · p. 23, Table 7 · p. 24
  • Figure 10 pairs example questions with executed ground truth over ten seeds, while Figure 9 marks the main-benchmark-trained decision tree at 48.0%, chance at 33.3%, and the data-blind bound at 50%.Figure 10’s left panels show datasets and candidates; right panels show executed ground truth, with medians and winners emphasized.Source: Figure 9 · p. 24, Figure 10 · p. 26

Related papers

Loading 2609.39714v1…