PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Discovery of Neural Architectures: AIRA-Compose and AIRA-Design

Alberto Pepe, Chien-Yu Lin, Despoina Magka, Bilge Acun, Yannan Nellie Wu, Anton Protopopov, Carole-Jean Wu, Yoram Bachrach

arXiv:2605.15871v1cs.AI

TL;DR

Designing neural architectures beyond Transformers is difficult because the hybrid design space is vast and combinatorial. This paper introduces AIRA-Compose and AIRA-Design for agentic architecture search and mechanistic implementation, with agents producing models that approach human state-of-the-art on LRA and achieve 0.968 validation BPB on Autoresearch.

  • Problem

    The vast, combinatorial space of hybrid architectures makes manual exploration liable to overlook high-performing configurations and new computational primitives.

  • Method

    AIRA-Compose searches arrangements of predefined primitives, while AIRA-Design uses open-ended code generation to implement novel models and training loops.

  • Results

    Agent-designed systems approached human state-of-the-art on Long Range Arena tasks and achieved 0.968 validation BPB on Autoresearch.

  • Takeaways & Limitations

    The results support agent-based discovery as a viable approach for engineering high-performing architectures and efficient language-model training loops.

  • Takeaways & Limitations

    AIRA-Compose relies on small-scale proxy evaluations, non-agentic aggregation and extrapolation, and a single dataset per task, limiting scale and validation robustness.

Abstract

from arXiv · show

Toward recursive self-improvement, we investigate LLM agents autonomously designing foundation models beyond standard Transformers. We introduce a dual-framework approach: AIRA-Compose for high-level architecture search, and AIRA-Design for low-level mechanistic implementation. AIRA-Compose uses 11 agents to explore fundamental computational primitives under a 24-hour budget. Agents evaluate million-parameter candidates, extrapolating top designs to 350M, 1B, and 3B scales. This yields 14 architectures across two families: AIRAformers (Transformer-based) and AIRAhybrids (Transformer-Mamba). Pre-trained at 1B scale, these consistently outperform Llama 3.2 and Composer-found baselines. On downstream tasks, AIRAformer-D and AIRAhybrid-D improve accuracy by 2.4% and 3.8% over Llama 3.2. Furthermore, AIRA-Compose finds models with highly efficient scaling frontiers: AIRAformer-C scales 54% and 71% faster than Llama 3.2 and Composer's best Transformer, while AIRAhybrid-C outscales Nemotron-2 by 23% and Composer's best hybrid by 37%. AIRA-Design tasks 20 agents with writing novel attention mechanisms for long-range dependencies and high-performing training scripts. On the Long Range Arena benchmark, agent-designed architectures reach within 2.3% and 2.6% of human state-of-the-art on document matching and text classification. On the Autoresearch benchmark, Greedy Opus 4.5 achieves 0.968 validation bits-per-byte under a fixed time budget, surpassing the published minimum. Together, these frameworks show AI agents can autonomously discover architectures and algorithmic optimizations matching or surpassing hand-designed baselines. This establishes a powerful paradigm for discovering next-generation foundation models, marking a clear step toward recursive self-improvement.

1 Introduction

The paper proposes AIRA-Compose and AIRA-Design as complementary agentic frameworks for Recursive Self-Improvement through neural architecture search and open-ended model implementation. Together, they show agents can discover performant hybrid architectures, engineer attention mechanisms, and improve language-model training efficiency.

  • Frameworks: AIRA-Compose searches arrangements of predefined computational primitives at small scale, then scales only top-performing architectures.AIRA-Design instead requires agents to implement novel computational primitives and train them efficiently for Long Range Arena and Autoresearch.
  • Frameworks: AIRA-Compose produces AIRAformers and AIRAhybrids with original interleavings of attention, MLP, and Mamba2 SSM blocks across 350M, 1B, and 3B parameter scales.AIRAformers use multi-head attention and MLP blocks, whereas AIRAhybrids additionally include Mamba2 State Space Model blocks.
  • Results: 2.3pp and 2.6pp: agent-designed attention mechanisms approach human SOTA on document matching and text classification, respectively.Peak accuracies were 82% on document matching and 91% on text classification; four agents exceeded an average normalized score of 0.3 across three tasks.
  • Results: 0.968 validation BPB: the best Autoresearch agent surpassed the published reference minimum under the open-ended training-loop optimization task.The agents iteratively improved small-language-model training loops, and 20% of experiments reproduced a baseline delta larger than reported in Karpathy (2026).
  • Results: Agent-discovered AIRAformer and AIRAhybrid models consistently outperform Llama 3.2, approximated Nemotron-2, and Composer-found alternatives across the reported downstream metrics.AIRA-Design agents also write functional code from scratch, including novel sub-quadratic attention mechanisms that reach within 2–3 percentage points of human SOTA across all three Long Range Arena tasks.

2 Methodology

The methodology formulates architecture search and mechanistic design as standardized AIRS-Bench tasks, executed through AIRA-dojo with configurable search scaffolds and operators. Tasks define a problem, dataset, and metric while evaluating agent-generated architectures or training scripts through modular artifacts and isolated scoring.

  • Search harness: AIRA-dojo evolves Python solutions through tree-based exploration using greedy or Monte Carlo Tree Search policies.The harness manages agent execution and tool access while guiding candidate-solution exploration.
  • Search operators: Four operators support search: Draft generates candidates, Debug fixes errors, Improve optimizes validation fitness, and Analyze evaluates each step.These operators act on candidate Python solutions during the search process.
  • Search scaffolds: One-shot scaffolds produce one solution, whereas greedy scaffolds begin with 5 drafts and iteratively improve the highest-fitness solution.Greedy runs explore tens to hundreds of steps, applying Debug when Analyze identifies bugs such as incorrect primitive counts or OOM errors.
  • Task formulation: AIRS-Bench tasks specify a problem, dataset, and metric triplet for quantifying agent fitness.The problem defines the challenge, the dataset supplies the data, and the metric measures performance.
  • Task infrastructure: The modular task structure includes a research prompt, setup scripts, an isolated evaluate.py scorer, and metadata.yaml constraints.For these tasks, evaluate.py trains the agent-generated architecture or launches its produced training script.

3 The AIRA-Compose Pipeline

AIRA-Compose recasts Composer’s search and evaluation stages as agentic AIRS-Bench tasks, allowing agents to propose, evaluate, and iteratively refine 16-layer architectures. The pipeline aggregates promising designs, then extrapolates them to larger model scales using proxy evaluations intended to predict at-scale performance.

  • Aggregation and scaling: The pipeline aggregates top candidates with layer-wise clustering, selecting frequent primitives to reduce proxy-training noise and overfitting before extrapolating architectures to 350M, 1B, and 3B parameters.Composer’s broader process combines search, fast-proxy evaluation, aggregation, and extrapolation.
  • Agentic search: AIRA-Compose recasts Composer’s search and evaluation stages as AIRS-Bench tasks, replacing rigid Bayesian and incremental search with agent-formulated structural hypotheses and iterative refinement.The search focuses on 16-layer small-scale proxies because they correlate with equivalent large-scale models.
  • Agentic search: Agents propose candidate architectures, write evaluation scripts, and validate designs during greedy exploration of the architecture space.The highest-scoring node is selected for further exploration, with debug and improve operations guiding the search.
  • Search spaces: The search spaces use MLPs (M), multi-head Attention (mA), and Mamba SSM (Mb), yielding 65,536 two-primitive and ≈43M three-primitive 16-layer arrangements.Configurations at small scale and at 350M, 1B, and 3B scales are provided in Appendix D.
  • Proxy evaluation: Architectures are evaluated on MAD, BabiStories, and a fixed DCLM subset using average accuracy or cross-entropy loss as proxy metrics.MAD uses 800 training and 1,280 test samples; BabiStories and DCLM each use 9,275 evaluation samples, with BabiStories training on 927,158 samples and DCLM on 10,000.

4 AIRA-Design: low-level mechanistic design

AIRA-Design recasts LRA and Autoresearch as agent-driven AIRS-Bench tasks for designing computational primitives, complete model implementations, and training loops. Up to 20 agents use one-shot and greedy scaffolds with 12 LLMs to explore memory-efficient sequence modeling and training optimization.

  • Task framework: Up to 20 agents use 12 LLMs to write novel computational primitives or complete training loops into model.py and train.py files.Tasks run with the one-shot and greedy scaffolds of the AIRA-dojo harness.
  • Evaluation tasks: The LRA benchmark evaluates sequence models on long-range dependencies, while Autoresearch evaluates autonomous exploration, implementation, and validation of ideas improving training efficiency.Unlike high-level architecture search, AIRA-Design targets low-level mechanistic design and optimized training loops.
  • Long Range Arena: LRA tasks require memory-efficient, sub-quadratic attention mechanisms that model long-range interactions without materializing full O(n^2) matrices for sequences exceeding 2000–4000 tokens.Agents implement executable JAX/Flax model.py files defining CustomEncoder or CustomDualEncoder architectures.
  • Autoresearch: Autoresearch agents improve a GPT training script toward the lowest validation bits per byte under a fixed 5-minute wall-clock budget on one GPU.The baseline train.py contains a nanochat model with multi-head causal attention and rotary embeddings.

5 Experiments

The experiments follow AIRS-Bench protocols, using bounded agent runs and task-specific evaluation metrics. AIRA-Compose emphasizes raw and scaled-up model performance, while AIRA-Design additionally evaluates submission validity and normalized progress.

  • Experimental setup: 20 seeds for one-shot agents and 10 seeds for greedy agents are used unless otherwise specified, with each run capped at 24 hours or 500 steps.Each agent receives one H200 GPU for drafting and validation.
  • AIRA-Compose metrics: AIRA-Compose ranks small models by raw accuracy or cross-entropy loss and evaluates scaled models using validation loss, downstream accuracy, and DCLM Core score.Downstream evaluation covers six 0-shot tasks, while DCLM Core averages performance across 14 tasks under 0-, 3-, and 10-shot settings.
  • AIRA-Design metrics: AIRA-Design reports raw score, valid submission rate, and normalized score to measure performance, consistency in producing working submissions, and progress toward SOTA.Valid submission rate is based on successfully submitted runs divided by total runs.
  • AIRA-Design metrics: Normalized score ranges from 0 at the worst observed performance to 1 at SOTA, exceeds 1 above SOTA, and assigns failed or invalid submissions a score of 0.The overall possible optimal task score is 1.0.
  • Task-specific reporting: Because AIRA-Compose outputs predefined-block strings with virtually 100% VSR, it reports raw and scaled-up metrics, whereas Autoresearch focuses on validation BPB.LRA tasks require all three AIRA-Design metrics because agents must write novel, functional code from scratch.

6 Results

The results show that agent-discovered Transformer and hybrid architectures achieve strong validation, downstream, and scaling performance, while agent-designed mechanisms produce functional but variable solutions across long-range reasoning tasks. Performance depends on architecture composition, compute regime, configurability, and agent behavior.

  • AIRAformer results: AIRAformer-D Stretched achieves validation loss 2.734, average 0-shot accuracy 59.7% (60.8%), and DCLM Core score 48.9%, outperforming Llama 3.2 and Composer-found models.Low variance across seeds indicates statistically robust improvements.
  • AIRAhybrid results: ∆q = −0.14 gives attention-heavy AIRAhybrid-C Stretched the steepest scaling slope, although balanced Mamba-to-attention ratios achieve the lowest validation loss across model sizes and FLOP budgets.AIRAhybrid-D variants advance Composer’s latency–validation-loss Pareto frontier under isoToken analysis.
  • AIRA-Design results: Greedy Opus 4.6 always successfully submits a solution, whereas one-shot agents produce no valid submissions and weaker models fall below a 10% Valid Submission Rate.Configurability generally lowers validity, indicating difficulty navigating the expanded hyperparameter search space.
  • Autoresearch and agent behavior: 0.036 is the single largest improvement, achieved by Opus 4.5 +Literature after replacing cross-entropy with focal loss around step 15.Agents also produced functional solutions using linear attention, hierarchical pooling, and blockwise local attention with recurrent mechanisms.

7 Conclusion · Appendix · A Related Work

AIRA-Compose and AIRA-Design demonstrate that agents can autonomously search architectures and design training methods, producing competitive foundation-model designs. The paper also identifies proxy-scaling, mechanistic-innovation, and iterative-refinement limitations while situating the work within hybrid modeling, NAS, and recursive self-improvement research.

  • 7 Conclusion: AIRA-Compose and AIRA-Design assessed agents’ ability to design neural architectures and training methods across 12 tasks from Composer, LRA, and Autoresearch.The approaches are complementary and span three distinct frameworks.
  • 7 Conclusion: 11 agents across 340 24-hour and 300 60-hour runs autonomously formulated computational-primitive hypotheses and constructed original architectures.This search navigated combinatorial spaces without rigid traditional optimization objectives.
  • 7 Conclusion: 14 novel AIRAformers and AIRAhybrids showed favorable downstream performance, robust scaling, and promising loss–efficiency trade-offs.The architectures include Transformer-based and hybrid design families described in the paper context.
  • 7 Conclusion: 20 agents on Long Range Arena produced designs within 3% of current human state-of-the-art, while strongest agents improved designs through efficient hyperparameter tuning.Greedy Gemini 3 Pro and Greedy Opus 4.6 achieved the reported peak accuracy.
  • 7 Conclusion: Agent-driven architecture search and design were feasible and yielded highly competitive designs for next-generation foundation models and agents, supporting the prospect of recursive self-improvement.This is the paper’s stated overall conclusion rather than an additional interpretation.
  • Limitations and Future Work: AIRA-Compose’s small-scale proxy evaluation does not always faithfully predict large-scale performance, and translating configurations across scales requires careful tuning.Future work proposes advanced harnesses such as AIRA2 and more agentic aggregation and extrapolation.
  • Limitations and Future Work: AIRA-Design architectures largely recombined prior ideas rather than introducing paradigm-shifting theoretical innovation, indicating stronger engineering synthesis than genuine algorithmic innovation.One-shot agents produced no valid submissions, underscoring the importance of iterative refinement and debugging for low-level code generation.
  • A Related Work: Hybrid language models address Transformers’ quadratic self-attention cost by interleaving computational primitives such as Attention and SSMs to balance efficiency and expressiveness.This related-work context motivates architectures that combine diverse primitives for long-context processing.

B Small Scale Ranking

The section ranks agent-discovered small-scale architectures from greedy search by test performance across two- and three-primitive MAD settings. Each ranking shows primitive sequences, accuracies, and the discovering agent through bar colors.

  • B Small Scale Ranking: Top 120 small-scale architectures in the MAD 2-Primitives setting are ranked by test performance.The primitives are M and mA, and each row pairs its sequence with accuracy; bar colors identify the discovering agent.
  • B Small Scale Ranking: The rankings compare architecture sequences and accuracies while indicating which agent discovered each candidate.The two-primitive figure uses M and mA, whereas the three-primitive figure adds Mb.
  • B Small Scale Ranking: Top 120 small-scale architectures in the MAD 3-Primitives setting are ranked by test performance.The primitives are M, mA, and Mb, with each row showing its sequence and accuracy; bar colors identify the discovering agent.

C Additional Details · D Aggregation and Scaling Patterns

The additional analyses expose both task-specific strengths and systematic scaling patterns in agent-discovered architectures. Across downstream evaluation and isoFLOP comparisons, balanced allocation of computation generally improves performance, while BoolQ remains near chance across architectures.

  • C Additional Details: The 2-primitive exploration trajectories compare mean validation and test accuracy across 10 seeds, with their gap representing proxy-selection generalization difficulty.Validation trends and ±1 standard deviation shading are reported for greedy search on MAD.
  • C Additional Details: The 3-primitive exploration trajectories likewise track mean validation and test accuracy across 10 seeds, highlighting generalization gaps during greedy search.The figure reports validation trend lines and ±1 standard deviation shading on the MAD benchmark.
  • C Additional Details: ∼37.9%: BoolQ remains essentially at chance across all architectures under 10-shot evaluation.The result indicates uniform failure at 1B scale with the stated training setup, regardless of architecture.
  • C Additional Details: Balanced 2-primitive architectures consistently achieve lower validation loss than attention-heavy variants across model scales and FLOP budgets.AIRAformer-A Stretched, AIRAformer-B Stacked, and Composite baselines outperform AIRAformer-C and AIRAformer-D, whose A/M ratios exceed 2.
  • C Additional Details: 2.7415 validation loss: AIRAformer-A Stretched versus 2.7647–2.7732 for attention-heavy variants at 1B FLOPs.This example illustrates the advantage of balanced compute allocation in the 2-primitive setting.
  • C Additional Details: Evenly distributing computation across Mamba, MLP, and attention yields the lowest validation losses in the 3-primitive hybrid setting.AIRAhybrid-C Stretched and Composer (2Mb-M-3A) perform best, whereas Mamba-only AIRAhybrid-A and attention-dominated hybrids underperform.
  • D Aggregation and Scaling Patterns: IsoFLOP evaluations compare validation loss for 2-primitive and 3-primitive architecture families across 350M, 1B, and 3B models and five FLOP budgets.The full scaling analysis also compares optimal frontiers against Nemotron-2 and Nemotron-H.

D.1 Aggregation methods

Aggregation methods produced four AIRAformer variants in the two-primitive space and five AIRAhybrid variants in the three-primitive space. The variants combine rank-weighted dataset aggregation, N1/N2 aggregation, and k-means clustering over agent-discovered architectures.

  • Two-primitive space: AIRAformer-A uses top-20 architectures from each of three datasets, with strong rank-based exponential weighting within datasets and equal weighting across datasets.Its base pattern is (A + M) + (2A + 2M) + 4 × (A + M) + 2M.
  • Two-primitive space: AIRAformer-B applies N1 aggregation to the top-400 architectures found by all five agents on MAD.Its base pattern is 2A + 5 × (A + M) + 4M.
  • Two-primitive space: AIRAformer-C and AIRAformer-D use N1 and N2 aggregation, respectively, on the top k-means cluster from 20-seed greedy GPT-5 searches on MAD.Their base patterns are (2A + M) + 3 × (A + M) + (2A + M) + 4A and 5 × (2A + M) + A, respectively.
  • Three-primitive space: AIRAhybrid-A, AIRAhybrid-B, and AIRAhybrid-C use N0, N1, and N2 aggregation, respectively, on top clusters from three-cluster k-means over greedy GPT-5 architectures.Their base patterns are 2Mb + M + 11Mb + 2M; 3 × (2Mb + M + A) + 2Mb + 2M; and 2Mb + A + 2 × (Mb + A) + M + 2 × (A + Mb + A + M).
  • Three-primitive space: AIRAhybrid-D and AIRAhybrid-E use N1 and N2 aggregation, respectively, on top clusters from three-cluster k-means over architectures found by all six agents.Their base patterns are 2Mb + M + 2 × (Mb + M) + A + M + Mb + M + A + M + Mb + M + A + M and 5 × (Mb + M + A) + M.

D.2 Scaling patterns

The architecture primitives are specified at a 16-layer proxy and extrapolated to 350M, 1B, and 3B scales. Small-scale components use fixed dimensions, while large-scale configurations adjust dimensions and employ more scalable variants.

  • Scaling setup: 16-layer proxy models use d = 128, while 350M, 1B, and 3B models configure d, attention heads, head dimensions, and hidden dimensions per scale.These configurations follow the IsoFLOP methodology.
  • MLP: MLP blocks change from a two-layer ReLU network with hmlp = 258 to large-scale SwiGLU blocks with gated projections and scale-dependent hidden dimensions.The large-scale design uses gate and up projections d → hmlp and a down projection hmlp → d.
  • Attention: Attention changes from 16 query and 16 key-value heads with dh = 8 to GQA using nkv = 8 shared key-value heads at large scale.Large-scale grouped-query attention yields total KV dimension dkv = nkv × dh.
  • Mamba: Mamba uses a Mamba-2 selective SSM with linear-complexity selective scanning, while large-scale settings increase state capacity and set dssm via a dimension-dependent rule.At small scale, essm = 1.25 gives dssm = 160, with ns = 4, k = 4, and ng = 1.

D.3 IsoFLOP Budget Methodology

The methodology equalizes training compute by estimating per-token, per-layer FLOPs for each primitive and converting architecture-specific step costs into training steps under fixed FLOP budgets. It uses shared sequence and batch settings, with worked examples for Transformer and hybrid architectures.

  • FLOP accounting: 6 · d_in · d_out FLOPs per projection per token follows from 2MKN forward-pass cost multiplied by 3 for training.The factor accounts for one forward and two backward passes.
  • Shared configurations: All three model scales use sequence length s = 8,192, n_kv = 8 key-value heads, and batch size 524,288 tokens per step.At the 1B scale, the batch is illustrated as 16 × 4 × 1 × 8,192.
  • Per-primitive FLOPs: Primitive costs include attention projections and quadratic attention, SwiGLU’s three projections, and Mamba2’s projection, convolution, selective scan, and output projection components.The Mamba2 input projection uses d_in_proj = 2d_ssm + 2n_g n_s + n_ssm, while the selective scan costs 2 · n_ssm · d_ssm h · n_s per token.

E AIRA-Design: LRA - Task Details

AIRA-Design’s configurable LRA experiments involved extensive hyperparameter tuning across ListOps, Text, and Retrieval, while tables document explored settings and best solutions across configurable and non-configurable tasks.

  • Hyperparameter Tuning: 7,537 valid evaluation steps were generated across the three configurable LRA tasks by all greedy agents.The steps comprised 3,189 for ListOps, 2,642 for Text, and 1,706 for Retrieval.
  • Hyperparameter Tuning: 3,189 evaluation steps were conducted for ListOps, 2,642 for Text, and 1,706 for Retrieval.These counts cover the configurable versions of the tasks.
  • Hyperparameter Tuning: Table 13 reports the explored value ranges and unique-value counts for each configuration parameter across the three configurable LRA tasks.The table summarizes the agents’ hyperparameter exploration.
  • Best Generated Solutions: Table 14 details the generating agent, achieved score, and architecture summary for the six LRA AIRA-Design tasks in configurable and non-configurable setups.For non-configurable tasks, it identifies configurations that deviate from defaults.

F AIRA-Design: Autoresearch - Task Details

The Autoresearch task augments the base setup with structured literature context from 41 curated papers. Its review spans architecture improvements, training strategies, and optimizers.

  • Task setup: 41 curated research papers are provided through an additional pwc/ folder containing task_info.jsonl with structured task, pipeline, dataset, and architecture information.The literature context is generated using the pipeline of Seo et al. (2025).
  • Literature review: 20 papers cover architecture improvements in the Autoresearch literature review.The review includes mechanisms such as attention variants, sparse computation, memory, and scaling-related changes.
  • Literature review: 16 papers cover training strategies in the Autoresearch literature review.These papers address feature learning, fixed-budget training, kernels, schedules, optimizers, batch size, and scaling laws.
  • Literature review: 5 papers cover optimizers in the Autoresearch literature review.This category is separate from the architecture-improvement and training-strategy reviews.

G AIRA-Design: Autoresearch - Additional Results · H Best Autoresearch Solution (Greedy Opus 4.5 + Literature access)

AIRA-Design’s Autoresearch analysis shows agents improve validation bits-per-byte through frequent architectural and hyperparameter changes, while the best solution combines a GPT implementation with configurable windows, value embeddings, and parameter-scaling components. Across 156 improvements, attention patterns and selected boolean additions produce especially strong gains, alongside a reported RADv1 comparison baseline.

  • G AIRA-Design: Autoresearch - Additional Results: The Autoresearch trajectories compare 10 agents across 10 seeds against the RADv1 baseline at BPB = 1.0121.Green-shaded cells identify seeds where agents improve over the baseline by at least 0.0205 BPB, the improvement obtained in Karpathy (2026).
  • G AIRA-Design: Autoresearch - Additional Results: 156 improvement steps show depth and width changes in 46% of steps, while attention pattern changes occur in 37% and have the highest median ∆ of 0.009.Depth and width changes have a median ∆ of 0.007.
  • G AIRA-Design: Autoresearch - Additional Results: 49% of improvements include learning-rate modifications, indicating that agents frequently adjust hyperparameters alongside architectural changes.The passage describes this co-occurrence as nearly half of all improvements.
  • G AIRA-Design: Autoresearch - Additional Results: ReLU2 → SwiGLU transitions yield a median ∆ of 0.007, exceeding the 0.003 median ∆ for SwiGLU → ReLU2 transitions.The most frequent transitions occur 14 times from ReLU2 to SwiGLU and 13 times in the reverse direction.
  • G AIRA-Design: Autoresearch - Additional Results: Value embeddings and EMA additions have median ∆ values of 0.014 and 0.015, while removing z-loss and gradient clipping has median ∆ values of 0.013 and 0.004.Only 7 of 27 features were ever the sole change in an improvement step, showing that compound modifications dominate.
  • H Best Autoresearch Solution (Greedy Opus 4.5 + Literature access): The best-solution implementation defines a GPT model with configurable window sizes, token embeddings, stacked blocks, and a bias-free language-model head.The implementation constructs one block per configured layer and computes window sizes from the model configuration.
  • H Best Autoresearch Solution (Greedy Opus 4.5 + Literature access): The implementation separately tracks matrix, embedding, language-head, value-embedding, residual, and x0 parameters, with dmodel learning-rate scaling proportional to (model_dim / 768)^-0.5.The code asserts that these parameter groups account for all model parameters.
  • H Best Autoresearch Solution (Greedy Opus 4.5 + Literature access): The model configuration derives model dimension and head count from depth, an aspect ratio, and a fixed head dimension before instantiating the GPT model.The configuration sets sequence length, vocabulary size, layer count, head counts, embedding dimension, and window pattern.
Loading 2605.15871v1…