LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimization

• NAS automates architecture design by searching for architectures that maximize an objective, but each candidate must be trained and many thousands may be discarded.The objective can, for example, be classification accuracy subject to parameter and training-cycle budgets.Source: §1 · p. 1
• Reinforcement-learning NAS trains a controller to output architectures, whereas evolutionary NAS searches architecture populations directly using performance as fitness.In NAS, evolutionary search concerns architecture rather than network parameters.Source: §1 · p. 1
• LLMs cannot generally test architectures and learn from experiments alone, so the paper pairs their code-generating knowledge with robust search and quality-diversity.Quality-diversity can return architectures spanning dimensions such as parameter count or depth, supporting trade-off analysis and choices for differing RAM budgets.Source: §1 · pp. 1–2
• LLMatic combines an LLM with two quality-diversity archives to search for competitive networks using just 2,000 evaluations.The authors report testing it on CIFAR-10 and NAS-bench-201, where it seeks networks with performance similar to state-of-the-art results.Source: §1 · p. 2

• NAS approaches including Bayesian, reinforcement-learning, evolutionary, and Monte Carlo tree search require designers to define atomic network components or edit actions.The passage notes Bayesian methods' low sample complexity and the computational expense of training each architecture.Source: §2 · p. 2
• LLM-based code evolution includes ELM, which uses LLMs as MAP-Elites evolutionary operators for robot morphology, and EvoPrompting, which uses code LLMs for NAS.EvoPrompting is tested on MNIST-1D and the CLRS algorithmic reasoning benchmark; LLMatic instead treats FLOPS as a diversity metric.Source: §2 · p. 2
• Quality-diversity methods optimize fitness while partitioning archives by behavioral descriptors, retaining diverse high-quality individuals that may avoid local optima or provide stepping stones.The cited motivations include maintaining alternative robot-control trajectories and preserving diverse fit solutions in deceptive domains.Source: §2 · p. 2

• The network archive stores networks by width-to-depth ratio and FLOPS, with post-training test accuracy as fitness; the prompt archive stores prompt and temperature as descriptors.FLOPS is chosen over parameter count because it correlates better with actual training time; prompt selection depends on curiosity, which reflects whether generated networks enter the network archive.Source: §3 · pp. 2–3
• The search begins with a basic network and random prompt, then evaluates network and prompt individuals into separate archives.During the evolutionary loop, selected prompts and networks undergo evolutionary operations; crossover keeps the prompt fixed.Source: §3 · p. 2, Figure 1 · p. 3
• Temperature rises when network fitness improves and falls when it declines, shifting LLM generation toward exploration or exploitation, respectively.The prompt individual's fitness score is accumulated to determine its overall fitness in the prompt archive.Source: §3 · p. 3
• Mutation selects a curious prompt and a random network, while crossover selects random networks and their closest archive neighbors; crossover adds no prompt individual.Mutation also samples random prompts for exploration, and resulting networks are trained before archive insertion is attempted.Source: §3 · p. 3

4 EVALUATING LLMATIC

• CIFAR-10 is used for evaluation and ablations; NAS-bench-201 provides precomputed architecture accuracies for later search without candidate retraining.The benchmark enumerates architectures in a fixed search space and their post-training test accuracies.Source: §4 · pp. 3–4

4.1 Setting up LLMatic

• Search starts from a one-convolutional-layer network and trains generated networks for 50 epochs using stochastic gradient descent.The initial network connects a 1×1 convolution to a 1024-unit dense layer and then 10 output units; generated networks use learning rate 0.001 and momentum 0.9.Source: §4.1 · p. 4
• The two archives use two-dimensional descriptors: network width-to-depth ratio and FLOPS, and prompt identity and temperature.Prompt selection favors the most curious individual; curiosity rises when its generated network enters the network archive and falls when generation fails or is untrainable.Source: §4.1 · pp. 4–5

4.2 Ablation Study

• 2,000 evaluations produce a competitive network, compared with 8,000 in the original EfficientNet study.EfficientNet-B0 was pretrained on ImageNet and then trained on CIFAR-10 via transfer learning.Source: §4.2.1 · p. 5
• Mutation-Only-LLMatic and Network-Archive-LLMatic are closest to LLMatic, while Crossover-Only-LLMatic performs worst; both operators together yield competitive results.Figure 2 plots best accuracy per generation across 30 seeds, with the mean as a solid line and standard deviation shaded.Source: §4.2.1 · p. 5, Figure 2 · p. 6
• The network archive contains more than 20 competitive networks spanning FLOPS and width-to-depth ratio, while high-performing prompt cells favor early prompts and higher temperatures.Some high-performing prompt individuals have lower temperatures as well.Source: §4.2.1 · pp. 5–6, Figure 3 · p. 6
• Uniform operator probabilities remain competitive by trainable-network count, but increasing crossover lowers that count, supporting mutation-heavy selection.The study uses trainable networks per generation as a proxy because more functional individuals offer a greater chance of finding high-performing ones.Source: §4.2.1 · p. 6

5 EXPERIMENTS ON NAS-BENCH-201

• NAS-Bench-201 uses a fixed network skeleton whose cell block searches a constant neural-network structure, restricts cells to five possible operations, and evaluates on CIFAR10, CIFAR100, and ImageNet16-120.Search runs for 20 generations of 100 cells each; minimum and maximum FLOPS are the network-archive behaviour-descriptor bounds.Source: §5 · pp. 6–7, §5.1 · p. 7

5.1 Results

• LLMatic outperforms GPT-4-based GENIUS and produces near-state-of-the-art NAS-bench-201 test accuracy, although its results show more variation than competing methods.Table 1 compares test accuracy; LLMatic results are averaged over 10 runs.Source: §5.1 · p. 7, Table 1 · p. 7
• Figure 5 shows test accuracies for all networks across datasets and for generation-wise best networks; the discovered networks span the search space.Source: Figure 5 · p. 7, §5.1 · p. 7
• Table 2 reports LLMatic’s maximum attained rank on each NAS-bench-201 dataset to illustrate the authors’ near-optimal-network claim.Source: §5.1 · p. 8, Table 2 · p. 8

6 CONCLUSION AND FUTURE WORK

• With 2,000 evaluations, LLMatic finds more than 20 competitive CIFAR-10 networks and near-optimal NAS-bench-201 networks, while producing diverse architectures.The method combines large language models with Quality-Diversity optimization.Source: §6 · p. 8
• LLMatic uses CodeGen-6.1B, described as smaller than existing LLMs; due to computational resources, its searches were kept to 2,000, yet it still found competitive networks.Source: §6 · p. 8
• Future work should compare LLMatic with other NAS methods on computer-vision and NLP tasks, and with reinforcement-learning benchmarks.The authors also suggest exploring LLMatic for open-ended learning.Source: §6 · p. 8