PaperlayerView exact revision on arXiv

Source-linked AI summary

Resource-Efficient Iterative LLM-Based NAS with Feedback Memory

Xiaojie Gu, Dmitry Ignatov, Radu Timofte

arXiv:2603.12091v1cs.LGcs.AI

TL;DR

Conventional NAS is computationally expensive, while single-shot LLM architecture generation discards evaluation feedback. This paper proposes iterative LLM-driven NAS with bounded historical feedback and reports substantial accuracy gains across models and datasets using a low-budget single-GPU setup.

  • Problem

    Conventional NAS requires substantial computation, and single-shot LLM generation discards evaluation signals instead of iteratively learning from architecture attempts.

  • Method

    A closed-loop pipeline generates, evaluates, and refines architectures using a K=5 sliding memory of structured diagnostics containing problems, suggested improvements, and outcomes.

  • Results

    On CIFAR-10, accuracy improved from 28.2% to 69.2% for DeepSeek-Coder-6.7B, from 50.0% to 71.5% for Qwen2.5-7B, and from 43.2% to 62.0% for GLM-5.

  • Takeaways & Limitations

    A full 2000-iteration search required approximately 18 GPU hours on one RTX 4090, supporting low-budget iterative NAS with frozen LLMs of at most 7B parameters.

  • Takeaways & Limitations

    Code-generation success rates vary substantially, and larger-scale benchmarks and more diverse tasks are needed to better demonstrate versatility.

Abstract

from arXiv · show

Neural Architecture Search (NAS) automates network design, but conventional methods demand substantial computational resources. We propose a closed-loop pipeline leveraging large language models (LLMs) to iteratively generate, evaluate, and refine convolutional neural network architectures for image classification on a single consumer-grade GPU without LLM fine-tuning. Central to our approach is a historical feedback memory inspired by Markov chains: a sliding window of $K{=}5$ recent improvement attempts keeps context size constant while providing sufficient signal for iterative learning. Unlike prior LLM optimizers that discard failure trajectories, each history entry is a structured diagnostic triple -- recording the identified problem, suggested modification, and resulting outcome -- treating code execution failures as first-class learning signals. A dual-LLM specialization reduces per-call cognitive load: a Code Generator produces executable PyTorch architectures while a Prompt Improver handles diagnostic reasoning. Since both the LLM and architecture training share limited VRAM, the search implicitly favors compact, hardware-efficient models suited to edge deployment. We evaluate three frozen instruction-tuned LLMs (${\leq}7$B parameters) across up to 2000 iterations in an unconstrained open code space, using one-epoch proxy accuracy on CIFAR-10, CIFAR-100, and ImageNette as a fast ranking signal. On CIFAR-10, DeepSeek-Coder-6.7B improves from 28.2% to 69.2%, Qwen2.5-7B from 50.0% to 71.5%, and GLM-5 from 43.2% to 62.0%. A full 2000-iteration search completes in ${\approx}18$ GPU hours on a single RTX~4090, establishing a low-budget, reproducible, and hardware-aware paradigm for LLM-driven NAS without cloud infrastructure.

1. Introduction

Conventional NAS can be highly resource-intensive, while many LLM-based architecture generators operate without iterative self-improvement. This work proposes a closed-loop pipeline that uses evaluation feedback and bounded history to iteratively refine executable architectures.

  • Motivation: Early NAS methods required up to 22,400 GPU-days or thousands of GPU-days, while proxy-based methods remained constrained by predefined search spaces.These costs motivate more data-efficient and training-free approaches.
  • Motivation: Single-shot LLM architecture generation discards evaluation signals instead of learning from failures and accumulated design experience.The paper frames iterative learning from structured attempt histories as its central question.
  • Approach: The proposed pipeline combines a Code Generator, one-epoch Evaluator, and Prompt Improver that uses recent iteration history to suggest targeted refinements.The pipeline generates executable PyTorch models, evaluates them on image-classification tasks, and feeds diagnostic results into the next iteration.
  • Approach: The method uses a bounded history window so improvement decisions depend on the current best architecture and recent transitions rather than the full trajectory.This Markov-inspired design keeps context bounded while retaining recent feedback.

2. Related Work

Prior work established LLM-based architecture generation and iterative optimization, but commonly used single-step generation, constrained search spaces, or histories that discarded failure trajectories. This paper positions its method as iterative NAS in open executable-code space with structured failure feedback.

  • Neural Architecture Search: Traditional NAS spans reinforcement learning, evolutionary, parameter-sharing, differentiable, and training-free approaches with differing search costs and evaluation strategies.Early reinforcement learning and evolutionary methods required thousands of GPU hours, whereas later methods amortized or bypassed training.
  • LLMs for Code Generation and AutoML: LLM-based AutoML work has generated functional vision models and explored architecture creativity, prompting, hyperparameter optimization, and related automation tasks.Most existing architecture-generation approaches nevertheless treat generation as a single-step process.
  • Iterative LLM Optimization: Iterative optimizers such as OPRO and FunSearch condition on historical attempts, while LLMO applies optimization within a predefined NAS cell space.These methods motivate iterative feedback but differ in search representation and history design.
  • Iterative LLM Optimization: Prior methods may discard failure trajectories, maintain evolutionary populations, or operate in constrained discrete spaces rather than open executable-code spaces.The paper identifies these limitations as the context for bounded Markovian memory and explicit failure diagnosis.
  • Positioning: The proposed method is reported as uniquely combining NAS, unconstrained code search, structured failure diagnosis, single-consumer-GPU operation with ≤7B LLMs, and no LLM fine-tuning.Table 1 compares these five properties across closely related methods.

3. Methodology

The methodology forms a closed loop in which an instruction-tuned LLM generates executable PyTorch code, an evaluator validates and proxy-trains it, and a Prompt Improver uses bounded structured history to guide the next iteration.

  • Pipeline Overview: Each iteration generates a candidate architecture, evaluates its proxy outcome, and produces improvement suggestions conditioned on the current best architecture and recent history.The best-performing architecture is retained while suggestions guide subsequent generation.
  • Code Generation: The Code Generator receives a fixed prompt containing its role, dataset task, current best implementation, and previous improvement suggestions.An extended prompt optionally supplies top-performing architectures and recent executable outcomes when context permits.
  • Evaluation: Generated code undergoes quick instantiation and forward-pass validation before models passing validation receive one epoch of proxy training.Validation failures are discarded and their error messages are forwarded to the Prompt Improver.
  • Evaluation: One-epoch training uses Top-1 test accuracy as a fast ranking signal across CIFAR-10, CIFAR-100, and ImageNette.The evaluator uses SGD with momentum, weight decay, cosine annealing, batch size 128, and standard augmentation.
  • Prompt Improver: The Prompt Improver receives the best code and accuracy, the current code and outcome, and a history containing the last K=5 improvement attempts.Its suggestions diagnose results and close the loop by informing the next Code Generator call.
  • Historical Feedback Memory: Each history entry records a problem, suggested modification, and outcome, including accuracy gains or execution errors.The bounded window preserves recent failure patterns while preventing context overflow.

4. Experiments

The experiments evaluate three frozen instruction-tuned LLMs across three image-classification datasets using one-epoch accuracy and reproducible single-GPU training. All models show statistically significant upward architecture-quality trends, with distinct trade-offs between peak accuracy, reliability, and iteration scale.

  • Experimental Setup: The evaluation covers CIFAR-10, CIFAR-100, and ImageNette using three LLMs with standardized generation settings and reproducible RTX 4090 training.Each candidate is trained for one epoch as the evaluation protocol, with deterministic CUDA operations and isolated subprocesses.
  • Evaluation Metrics: One-epoch validation accuracy is the primary architecture-quality metric, while Spearman ρ and Kendall τ quantify improvement across iterations.These measures assess whether later iterations tend to produce higher-quality architectures.
  • Overall Results: Statistically significant upward trends occur across all three datasets, with Spearman ρ reaching 0.75 and p ≈0.The results compare iterative search against the first successfully generated architecture as the single-shot baseline.
  • Model Results: DeepSeek-Coder improved CIFAR-10 one-epoch accuracy from 28.2% to 69.2%, the largest absolute gain among the model–dataset combinations.It completed 1519 of 2000 CIFAR-10 iterations successfully and reached 29.2% from 5.0% on CIFAR-100.
  • Model Results: Qwen2.5 reached the highest CIFAR-10 peak accuracy, 71.5%, despite an 18.8% success rate across 2000 iterations.Its ImageNette accuracy also rose from 23.0% to 54.6%, with ρ = 0.663.
  • Model Results: GLM-5 improved CIFAR-10 accuracy from 43.2% to 62.0% and achieved a 91.0% success rate in only 100 iterations.It also improved on CIFAR-100 and ImageNette, reaching 24.8% and 58.3%, respectively.
  • Cross-Model Comparison: Accuracy trajectories rise across datasets, with DeepSeek-Coder showing the largest CIFAR-10 gain, Qwen2.5 the highest peak, and GLM-5 robust trends across tasks.The trajectories support generalization of the iterative pipeline across datasets of varying difficulty.

5. Ablation Study

The ablation study shows that the complete iterative loop depends on historical feedback and reference architectures to sustain improvement. Removing either component causes stagnation or degradation relative to single-shot generation.

  • Ablation Results: Removing historical feedback memory or the reference architecture makes the search stagnate or degrade and prevents it from surpassing the single-shot baseline.The ablations use DeepSeek-Coder-6.7B on CIFAR-10, CIFAR-100, and ImageNette.
  • Ablation Results: A transient ∼66% CIFAR-10 spike without feedback is quickly lost when history is not retained.This behavior supports retaining past execution outcomes during iterative refinement.

6. Discussion

The discussion attributes systematic improvement to short-window historical feedback while highlighting distinct model trade-offs and the practical value of low-budget, hardware-aware search. It also identifies failure rates and benchmark scope as important boundaries.

  • Iterative Refinement: The last K=5 improvement attempts provide context for identifying recurring failures and avoiding unsuccessful strategies.Accuracy curves typically rise rapidly before flattening, consistent with meaningful optimization dynamics.
  • Model Specialization and Scale: DeepSeek-Coder delivers the largest CIFAR-10 gain, Qwen2.5 the highest peak accuracy, and GLM-5 the highest reliability within 100 iterations.The models therefore differ in the balance between improvement magnitude, exploration success, and iteration scale.
  • Structured Failure Modeling: Structured diagnostic triples preserve code-execution failures as learning signals instead of discarding low-performing solutions.Observed failure rates of 5–99% make these failures a substantial part of the search process.
  • Hardware-Aware Search: The full 2000-iteration pipeline completes in approximately 18 GPU hours on one consumer-grade RTX 4090 using frozen LLMs of at most 7B parameters.Shared VRAM between language-model inference and architecture training implicitly favors compact models.
  • Evaluation Trade-off: One-epoch accuracy offers a low-cost ranking signal, although its absolute values remain below fully trained accuracy.The proxy is used to distinguish architectures efficiently rather than estimate converged performance.
  • Limitations: Success rates range from 0.7% to 95.1%, and the evaluation covers only three datasets, leaving larger benchmarks and diverse tasks for future work.The authors specifically identify model-specific failures and broader evaluation as unresolved issues.

7. Conclusion

The pipeline combines open-code iterative NAS with bounded historical feedback, structured failure diagnostics, and dual-LLM specialization. Across datasets and model origins, it improves proxy accuracy while completing 2000 iterations on a single consumer-grade GPU without LLM fine-tuning.

  • The pipeline iteratively generates and improves architectures in an unconstrained executable PyTorch code space.
  • CIFAR-10 accuracy improved from 28.2% to 69.2% for DeepSeek-Coder-6.7B, from 50.0% to 71.5% for Qwen2.5-7B, and from 43.2% to 62.0% for GLM-5.
  • ImageNette results included Qwen2.5-7B improving from 23.0% to 54.6%, while CIFAR-100 DeepSeek-Coder improved from 5.0% to 29.2%.
  • The approach uses K=5 bounded memory, structured diagnostic triples, and dual-LLM specialization to retain failure patterns and reduce per-call cognitive load.The shared VRAM setting also biases search toward compact, hardware-efficient models.
  • A full 2000-iteration search completes in ≈18 GPU hours on one RTX 4090 without LLM fine-tuning.The authors characterize the resulting paradigm as lightweight, data-efficient, low-budget, reproducible, and accessible without cloud infrastructure.
Loading 2603.12091v1…