Agentic Neural Architecture Search

• The LLM seed already surpasses published baselines on most tasks, while NAS adds gains in most cases through combinatorial recombination across slots.Independent LLM samples do not systematically explore interactions among multiple design choices.Source: §1 · pp. 1–2
• An LLM decomposes a high-quality seed architecture into named, interchangeable slots, automatically creating a bounded task-specific NAS space without manual engineering.The mechanism preserves LLM-driven design while making the resulting space suitable for conventional NAS.Source: §1 · pp. 1–2
• The study separates LLM-driven design from NAS-driven search to measure how each contributes to architecture discovery.This directly addresses whether an LLM can replace NAS and how their labor should be divided.Source: §1 · pp. 1–2
• AgentNAS establishes state-of-the-art results on 11 of 17 diverse tasks, including classification, dense regression, segmentation, and multi-label tagging.The tasks span NAS-Bench-360 and Unseen NAS, including a blind benchmark setting.Source: §1 · p. 2

2.1 Neural Architecture Search

• Conventional NAS methods operate within fixed, human-designed spaces whose construction remains labor-intensive and domain-dependent.The expressiveness of the search space can matter at least as much as the search algorithm, while grammar-based alternatives still shift manual design to grammar engineering.Source: §2.1 · p. 2

2.2 Large Language Models for Neural Architecture Search

• LLM-based NAS methods generate, mutate, optimize, rerank, or tune architectures in open-ended code-level spaces, but typically use the LLM inside the search loop.Prior work does not use an LLM to construct a bounded, task-specific space that is then handed to conventional NAS.Source: §2.2 · pp. 2–3
• The unresolved research question is how to divide labor between LLM-driven design and NAS-driven search, including when each component adds value.Existing approaches do not empirically study this handoff between space construction and conventional exploration.Source: §2.2 · pp. 2–3

2.3 Automated Scientific Discovery

• Related automated-discovery systems combine LLMs, multi-agent systems, evaluators, and evolutionary search to discover algorithms or conduct scientific research.This broader paradigm includes scientific machine learning applications such as PINN configurations and PDE surrogates, while AgentNAS targets general-purpose NAS that can extend to such domains.Source: §2.3 · p. 3

3 Method

• AgentNAS constructs a task-specific search space from LLM architectural priors and explores it with conventional NAS.The pipeline uses the LLM to produce both a strong seed architecture and the surrounding search space.Source: §3 · p. 3

3.1 Phase 1: LLM-Driven Seed Search

• Phase 1 iteratively refines candidate architectures through a Planner–Code Generator–Explorer loop, returning a seed architecture and training recipe.The process uses up to 160 evaluations, 30%-epoch proxy training, and full training of the top 8 architectures after termination.Source: §3.1 · p. 3

3.2 Phase 2: Slotted Architecture Synthesis

• Phase 2 preserves the seed’s macro-structure while exposing module-level combinatorial choices through named, interchangeable slots.The design avoids excessively large spaces, which reached approximately 10^90 configurations when every operation was made a slot.Source: §3.2 · p. 3, Figure 1 · p. 4
• A Modularizer groups primitive operations into coherent units before decomposition when needed to prevent unstable, overly fine-grained search.Module-level slots replace coherent blocks, while additive glue slots default to identity and can add operations, activations, or skip connections.Source: §3.2 · p. 3
• The search space includes a virtual learning-rate slot with multipliers µ∈{0.25, 0.5, 1.0, 2.0, 4.0} to account for changed parameter sizes after module replacement.The learning rate is optimized alongside architectural alternatives because the seed recipe may no longer be optimal.Source: §3.2 · p. 4

3.3 Phase 3: Slotted Neural Architecture Search

• Phase 3 searches the discrete slotted space with regularized evolution by default, while also evaluating random search and GDAS under the same proxy protocol.The top 8 candidates are fully trained and evaluated after search.Source: §3.3 · p. 4

4.1 Benchmarks

• The evaluation covers 17 tasks from two diverse-task benchmark suites, with NAS-Bench-360 contributing 10 tasks spanning classification, multi-label audio tagging, dense regression, and segmentation.Source: §4.1 · p. 4

4.2 Baselines

• The experiments compare AgentNAS with fixed architectures, conventional NAS methods, task-specific expert designs, and einspace baselines across both benchmark suites.NAS-Bench-360 uses WideResNet-16-4, GAEA DARTS, expert architectures, and einspace; Unseen NAS uses ResNet-18, conventional-NAS best results, and einspace variants.Source: §4.2 · pp. 4–5

4.4 Results

• AgentNAS achieves the best result on 8 of 10 NAS-Bench-360 tasks and state-of-the-art performance on 11 of 17 tasks overall.On NAS-Bench-360, its average rank is 1.5 versus 3.60 for the next-best Expert baseline.Source: §4.4 · p. 5, Table 1 · p. 6
• Table 2 reports Unseen NAS test errors, where lower values are better, and defines NAS Best as the per-task best among six conventional NAS methods.Source: Table 2 · p. 6
• The LLM-only seed matches or exceeds published baselines on most tasks, while Phase 3 NAS adds gains on most tasks through search over the constructed space.Reported improvements include Spherical from 45.08 to 38.71, CIFARTile from 11.20 to 8.48, and Chesseract from 33.84 to 31.39.Source: §4.4 · p. 6
• AgentNAS underperforms the per-task-best baseline on six tasks, showing that LLM-generated search spaces do not universally dominate hand-designed alternatives.The affected tasks are NinaPro, PSICOV, AddNIST, MultNIST, CIFARTile, and Chesseract.Source: §4.4 · p. 6

5.1 LLM Seed Search vs. Fixed-Seed NAS

• AgentNAS outperforms fixed-seed NAS on every completed task, showing that LLM-designed, task-adapted seeds provide a stronger starting point than generic architectures.The fixed-seed comparison replaces Phase 1 with WideResNet-16-4 or ResNet-18.Source: §5.1 · p. 7
• With a fixed seed, 160-evaluation Phase 2–3 outperforms einspace with 500 evaluations on Spherical and CIFARTile, indicating a strong LLM-constructed search space.The result supports contributions from both seed quality and search-space construction.Source: §5.1 · p. 7

5.2 NAS Complements LLM Sampling

• On most tasks, LLM sampling saturates within 480 evaluations across three LLMs, whereas NAS continues improving beyond saturation through combinatorial module recombination.Spherical and Chesseract show continued NAS gains; on Language, LLM-320+NAS slightly outperforms LLM-480.Source: §5.2 · p. 7
• NAS’s complementarity is broadly consistent across LLM capability levels, although Darcy Flow depends more on search-space quality than seed quality.Haiku generally starts worse, but NAS adds value after saturation in most configurations.Source: §5.2 · p. 7
• Search-algorithm differences are modest, but their hyperparameters were not optimized; AgentNAS also shows moderate seed variance, with standard deviations from 0.38 to 2.10 percentage points.All tested random seeds remained competitive with published baselines.Source: §5.2 · p. 7

5.3 What Does NAS Change?

• CIFARTile consistently gains channel-attention mechanisms, while Chesseract gains missing gating or spatial-mixing capabilities and prunes redundant components.NAS adds or diversifies channel recalibration on CIFARTile and compensates for absent functional components on Chesseract.Source: §5.3 · pp. 7–8
• On Darcy Flow, Sonnet’s nearly invariant spectral slots prevent improvement, whereas Opus’s diverse elementwise-filtering alternative enables NAS to select a simpler, parameter-efficient design.The contrast shows that the LLM contributes search-space quality in addition to seed quality.Source: §5.3 · p. 8

• AgentNAS automatically converts an LLM seed into a slotted, task-specific search space that conventional NAS can explore without manual engineering.Source: §6 · p. 8
• The evaluation is constrained by hardware and time limits, variable search budgets, validation–test misalignment, and occasional failures by weaker LLMs to construct viable slotted architectures.Source: §6 · p. 9
• Future work should improve understanding of LLM-constructed search spaces, optimize NAS algorithms and hyperparameters, and test generative tasks under relaxed resource constraints.Source: §6 · p. 9

• AgentNAS is more expensive than supernet-based GAEA but has search costs similar to einspace on Unseen NAS, with GPU cost driven mainly by per-candidate training time.Source: §A.1 · p. 17
• AgentNAS models are generally smaller than einspace on Unseen NAS but larger than GAEA and expert designs on NAS-Bench-360.On NAS-Bench-360, examples include CIFAR-100 at 23.4M versus GAEA’s 4.9M and Satellite at 67.3M versus 3.4M.Source: §A.1 · p. 17
• Phase 3 can reduce parameter count relative to the Phase 1 seed, including Spherical at 15.4M versus 19.7M and Satellite at 54.7M versus 67.3M.Source: §A.1 · p. 17
• AgentNAS uses fewer FLOPs than GAEA on 6 of 10 NAS-Bench-360 tasks despite often having larger parameter counts.The search space may favor parameter-heavy but compute-efficient operations such as depthwise separable convolutions and mobile inverted bottlenecks.Source: §A.1 · p. 17
• Efficiency comparisons use different hardware and incomplete reference coverage, including missing wall-clock GAEA data and unavailable FLOPs for expert models with non-standard modules.Source: Table 6 · p. 18, Table 7 · p. 18, §A.1 · p. 18

• AgentNAS uses the same training recipes for NAS as in Phase 1, except for the learning rate, and compares them with einspace’s recipes.Source: §A.2 · p. 19
• Unlike einspace’s single fixed recipe per benchmark suite, AgentNAS dynamically searches hyperparameters and tailors them to each dataset.Learning-rate changes selected by NAS are marked with an arrow in the comparison table.Source: Table 8 · p. 19

A.3 Search Algorithm Ablation

• RE and RS perform comparably overall, with neither consistently dominating across tasks.RE is best on Spherical, CIFARTile, and Chesseract, whereas RS wins on NinaPro, Darcy Flow, and Language.Source: §A.3 · pp. 19–20
• GDAS performs substantially worse on Darcy Flow and CIFARTile than the best competing method.The reported comparisons are 0.0341 versus 0.0056 for Darcy Flow and 75.57 versus 8.48 for CIFARTile; GDAS hyperparameters were not optimized for the slotted setting.Source: Table 9 · p. 19, §A.3 · p. 20
• The slotted search space supports diverse NAS methods, and uniform sampling often finds competitive architectures.The modest differences between RE and RS suggest that the LLM-defined space contains a high density of strong candidates.Source: §A.3 · pp. 19–20

A.4 Multiple Random Seed Ablation

• Table 10 reports AgentNAS-Full’s per-task variability across three random seeds, {42, 43, 44}, using mean ± standard deviation computed from each seed’s ever-best val-final architecture test metric.Source: Table 10 · p. 20
• The NinaPro seed-43 phase-3 FLOPs value is excluded because of a recording bug.This exclusion applies specifically to the phase-3 FLOPs aggregate.Source: §A.4 · p. 20

A.5 Full Baseline Results

• Table 11 reports full per-method test errors on Unseen NAS, with lower values indicating better performance.The table transcribes results from the original Unseen NAS dataset and einspace, marking the best and second-best performance per dataset.Source: Table 11 · p. 20

A.6.1 NAS-BENCH-360 DATASET DESCRIPTIONS

• NAS-Bench-360 comprises ten tasks spanning image classification, signal processing, scientific computing, and sequence analysis, with task-specific modalities and metrics.Examples include RGB images, EMG signals, audio clips, 2D scientific grids, protein features, astronomical images, ECG recordings, satellite time series, and genomic sequences.Source: §A.6.1 · pp. 20–21

A.6.2 UNSEEN NAS DATASET DESCRIPTIONS

• Unseen NAS comprises eight unconventional-modalities tasks categorized as Type-1, solvable by humans, or Type-2, requiring non-trivial pattern recognition beyond human intuition.The suite includes arithmetic digit tasks, encoded language and authorship, tiled image comparison, music-era classification, geolocation, and chess outcomes.Source: §A.6.2 · p. 21
• Despite blindness, the LLM correctly identifies Chesseract and Language, defaults to generic image-classification hypotheses on five other tasks, yet AgentNAS reaches state-of-the-art results on Language and GeoClassing.These findings suggest the pipeline can remain effective despite incorrect or incomplete domain recognition.Source: §A.6.2 · p. 22, Table 12 · p. 22
• Removing blindness substantially improves AddNIST and MultNIST, has negligible effect on Gutenberg, and makes Chesseract nearly 4 percentage points worse.The blind-versus-non-blind comparison evaluates whether domain knowledge benefits performance across four tasks.Source: §A.6.2 · pp. 22–23, Table 13 · p. 22

A.7.1 SLOT GENERATION

• The slot generator caps each slotted architecture at 20 slots with up to 8 alternatives per slot, enforcing these limits after parsing rather than by re-prompting.The planner is a single-call agent, so exceeding either cap triggers truncation instead of a full Phase 2 retry.Source: §A.7.1 · p. 23

A.7.2 SEARCH PROCESS

• AgentNAS uses independent 160-evaluation budgets and 64-successful-evaluation patience counters for Phases 1 and 3, with evolution capped at 10 generations and population size 16.Proxy training uses 0.3 of the full epoch schedule, a 3,600-second GPU-time cap, and an 11,264 MB VRAM cap; final training evaluates the top 8 architectures in two waves.Source: §A.7.2 · p. 23

A.7.3 MODULARIZATION ABLATION

• NAS uses modularization to avoid early patience-based termination, enabling the full 160-evaluation budget and substantial improvements on both block-stack tasks.Without modularization, NAS terminates after 80 evaluations and finds weaker architectures; modularization groups tightly coupled primitives into coherent modules without altering the seed.Source: §A.7.3 · p. 24, Table 14 · p. 24

A.8 Context Management and Prompts for Agents

• The pipeline uses five agents across three phases: Planner, Data Explorer, Code Executor, Modularizer, and Slotted Architecture Planner.The Planner controls Phase 1, the Data Explorer analyzes data through code execution, the Code Executor translates descriptions into PyTorch, the Modularizer refactors seeds, and the Slotted Architecture Planner designs the Phase 2 search space.Source: §A.8 · p. 24

A.8.2 DATA EXPLORER

• The Data Explorer operates under a maximum-step budget and must report on the final step.Its investigation purpose is supplied by the Planner via <EXPLORE>.Source: §A.8.2 · p. 26

A.8.3 CODE EXECUTOR

• The Code Executor converts architecture descriptions into self-contained PyTorch models and training configurations under VRAM, proxy-runtime, loss, and recipe constraints.It supports validation retries that fix implementation bugs without changing the architecture’s functional design.Source: §A.8.3 · p. 26

A.8.4 MODULARIZER

• The Modularizer refactors repeated stages into self-contained modules while requiring bit-equivalent forward behavior, identical parameters and operation order, and unchanged construction and augmentation logic.Validation checks can detect parameter mismatches or forward-output differences before retrying the refactoring.Source: §A.8.4 · pp. 28–29

A.8.5 SLOTTED ARCHITECTURE PLANNER

• The Slotted Architecture Planner creates a 10–20-slot, domain-aware search space with exact interfaces, seed-first alternatives, novel and standard operations, identity and zero choices, and no pre-trained weights.For self-contained seed stages, it uses one whole-stage operation slot plus outer slots whose seed alternatives are explicitly Identity, avoiding duplicated internal functionality.Source: §A.8.5 · pp. 30–31