PaperlayerView exact revision on arXiv

Source-linked AI summary

From Memorization to Creativity: LLM as a Designer of Novel Neural Architectures

Waleed Khalid, Dmitry Ignatov, Radu Timofte

arXiv:2601.02997v2cs.LGcs.CV

TL;DR

The paper asks whether iterative refinement can make an LLM a more reliable, capable, and structurally diverse neural-architecture generator. It closes the loop by evaluating generated PyTorch models, filtering for validity, early performance, and novelty, and LoRA-fine-tuning on selected outputs; across 22 cycles, the resulting distributions shift toward valid and high-performing architectures across three image benchmarks.

  • Problem

    Existing LLM architecture-synthesis studies offer limited evidence about how syntactic validity, structural novelty, and diversity evolve as the generator specializes.

  • Method

    A code-oriented LLM generates PyTorch architectures over 22 cycles; valid, high-performing, non-redundant candidates are added to the corpus and used for LoRA fine-tuning.

  • Results

    Across CIFAR-10, CIFAR-100, and SVHN, iterative refinement shifts outputs toward higher validity and early performance while sustaining structural novelty; on CIFAR-10, mean first-epoch accuracy rises from 28.06% to 49.48%.

  • Takeaways & Limitations

    The results characterize iterative self-refinement as a way to reshape an LLM into a task-specialized architectural prior while retaining non-trivial structural diversity.

  • Takeaways & Limitations

    The study evaluates only low-resolution 32×32 image-classification benchmarks, leaving transfer to higher-resolution, non-visual, segmentation, and detection tasks unclear.

Abstract

from arXiv · show

Large language models (LLMs) excel in program synthesis, yet their capacity for neural architecture design -- balancing syntactic reliability, performance, and structural novelty -- remains underexplored. We present a closed-loop architecture synthesis pipeline within the NNGPT framework, in which a code-oriented LLM evolves over 22 supervised fine-tuning cycles. At each cycle, the LLM synthesizes PyTorch convolutional networks, validated via low-fidelity performance signals and filtered via a MinHash--Jaccard criterion to prevent structural redundancy before being incorporated into the LEMUR dataset. High-performing candidates with novel architectures are converted into prompt--code pairs for parameter-efficient LoRA fine-tuning. This feedback loop drives a measurable distributional shift, progressively internalizing empirical architectural priors such that valid and high-performing outputs evolve from scarce to dominant across cycles. On CIFAR-10, the valid generation rate stabilizes at 50.6% (peaking at 74.5%), mean first-epoch accuracy rises from 28.1% to 51.0%, and candidates exceeding 40% accuracy grow from 2.0% to 96.8%. Cross-dataset transfer to CIFAR-100 and SVHN confirms that improved validity, shifted accuracy distributions, and sustained novelty generalize across benchmarks of varying difficulty and visual domain. Across 22 cycles, 455 unique architectures absent from the original corpus are admitted under the novelty filter. By grounding synthesis in execution feedback and novelty filtering, we demonstrate that iterative self-supervised fine-tuning reshapes an LLM into a task-specialized architectural prior -- improving generation reliability, proxy performance, and structural diversity -- offering a reproducible, annotation-free alternative to hand-crafted search spaces.

1. Introduction

The paper studies whether iterative fine-tuning on an LLM’s own successful architecture generations improves validity, early performance, and structural novelty. Across 22 cycles, the feedback loop shifts outputs toward valid, high-performing architectures while retaining diversity, but the study characterizes generator dynamics rather than a complete NAS method.

  • Existing LLM architecture-generation studies provide limited insight into how syntactic validity, structural novelty, and diversity evolve under iterative refinement.
  • The central question is whether repeatedly fine-tuning an LLM on successful generations improves valid, high-quality, and structurally novel architecture production.The study uses single-epoch CIFAR-10 accuracy as a low-cost performance proxy and treats structural uniqueness as a first-class objective.
  • Across 22 cycles, candidates are filtered by compilation validity, single-epoch training performance, and MinHash–Jaccard novelty before fine-tuning.The loop uses low-fidelity signals to guide iterative generate–evaluate–select–fine-tune updates.
  • The iterative process produces an upward shift in first-epoch accuracy while maintaining significant structural diversity and raising valid generation above early-cycle levels.The valid generation rate is not monotonic but stabilizes substantially above early-cycle baselines for much of the run.
  • The study characterizes changes in the generator’s architecture distribution rather than competing with established NAS methods on fully trained final accuracy.It does not claim generated architectures surpass hand-designed baselines after full training.
  • The framework contributes a trainable architectural prior, a MinHash–Jaccard novelty filter, a 22-cycle longitudinal analysis, and cross-benchmark validation.The reported benchmarks are CIFAR-10, CIFAR-100, and SVHN.

2. Related Work

Related work combines traditional NAS, low-fidelity evaluation, and code-capable LLMs, but this paper uses those ingredients to study how iterative feedback changes a generator’s behavioral priors. Structural diversity and executable validity are treated as distinct requirements for useful PyTorch architecture synthesis.

  • Traditional NAS automates network design with reinforcement learning, evolutionary algorithms, and differentiable optimization, but often incurs prohibitive computational cost.This cost motivates the use of lower-fidelity evaluation signals.
  • Low-fidelity proxies such as early stopping, learning-curve extrapolation, and zero-cost signals reduce the expense of repeatedly training candidate architectures.
  • This work applies single-epoch accuracy to shape a generative LLM’s behavioral priors rather than optimizing within a static handcrafted search space.
  • Code-capable LLM frameworks generate complete network implementations using mutation, quality-diversity search, iterative refinement, performance-guided evolution, or reflective reasoning.The cited systems include LLMatic, LEMONADE, SEKI, and RZ-NAS.
  • Useful LLM-generated architectures require both executable validity and downstream learning quality, while MinHash–Jaccard filtering limits redundant code motifs.The method separates parsing, instantiation, and forward-pass validity from learning performance.

3. Method

The method runs a 22-cycle closed loop in which a code-focused LLM generates PyTorch models, candidates are executed and evaluated with fixed early-training protocols, novel high-performing models augment the corpus, and LoRA fine-tuning updates the generator. The procedure is replicated across CIFAR-10, CIFAR-100, and SVHN with dataset-specific inclusion thresholds.

  • The pipeline generates PyTorch candidates, checks validity, measures first-epoch performance, filters novelty, and fine-tunes the LLM on accepted outputs across 22 cycles.The primary evaluation uses CIFAR-10, with cross-dataset generalization assessed separately.
  • Figure 1 summarizes the generate–evaluate–select–fine-tune loop, including validity, first-epoch accuracy, and novelty as candidate filters.
  • The LLM is prompted to generate CIFAR-10 classifiers under fixed PyTorch API, operation, shape, and parameter-budget constraints.The models receive 32×32 RGB inputs, produce 10 logits, and use at most 500,000 parameters.
  • The initial LEMUR corpus contains 849 unique architectures represented by 1,698 prompt–code examples, after MinHash/LSH deduplication and chat-format conversion.Generated candidates are compared using token-level code shingles against training and same-cycle generation sets.
  • Candidates whose MinHash-estimated Jaccard similarity exceeds approximately 0.9 to either comparison set are rejected as near duplicates.Sampling continues until a valid architecture is sufficiently dissimilar from both the training corpus and earlier same-cycle generations.
  • 455 unique high-accuracy models are added across 22 cycles, expanding the corpus from 1,698 to 2,153 examples.Inclusion requires compilation and training success, dataset-specific first-epoch thresholds, and passing the novelty filter.
  • Validity checks reject parsing, instantiation, or dummy-forward failures before candidates undergo a standardized fixed training protocol.The protocol holds the data split, input resolution, optimization schedule, and batch size constant.
  • The primary performance signal is top-1 validation accuracy after one training epoch, with per-cycle summaries also reporting validity, mean and variation, and above-threshold proportions.

4. Results

Across 22 cycles, the synthesis loop improved proxy performance and shifted CIFAR-10 outputs toward valid, above-threshold architectures while retaining structural novelty. The same qualitative pattern appeared on CIFAR-100 and SVHN, and ablations attributed the gains to the combined novelty filter, accuracy threshold, and iterative fine-tuning.

  • CIFAR-10 reliability: 50.6% mean valid generation rate across 22 CIFAR-10 cycles, peaking at 74.5% before stabilizing mostly between 51% and 60%.Validity began at 44.0% in cycle 1 and declined to 41.8% in cycle 22.
  • CIFAR-10 proxy quality: 28.06% → 49.48% mean first-epoch accuracy from cycle 1 to cycle 22, with a 50.99% mean at the strongest cycle-18 checkpoint.The best cycle-18 model reached 63.98% first-epoch accuracy.
  • CIFAR-10 proxy quality: 2.04% → 96.81% of models exceeded 40% first-epoch accuracy from cycle 1 to cycle 18, making above-threshold architectures typical rather than rare.The fraction exceeded 90% throughout cycles 16–20 and ended at 92.86% in cycle 22.
  • Novelty and corpus growth: 455 structurally novel architectures were admitted to the fine-tuning corpus, which grew from 1,698 to 2,153 prompt–code pairs by cycle 22.Admitted models ranged from compact single-block designs to deeper residual and dropout-based networks.
  • Cross-dataset generalization: CIFAR-100 and SVHN reproduced the main trajectory: rising mean first-epoch accuracy and threshold-exceeding fractions alongside broadly comparable validity patterns.CIFAR-100 mean accuracy rose from 9.0% to 20.5%, while SVHN rose from 51.0% to 70.5%.
  • Ablation study: Removing novelty filtering reduced distinct admissions from 455 to 220, removing the accuracy threshold lowered above-40% outputs from 51.1% to 34.0%, and removing fine-tuning left metrics at cycle-1 levels.The ablations support complementary roles for exploration, corpus quality, and feedback-driven improvement.

5. Conclusion and Future Work

The study tracks an LLM’s architecture distribution across 22 iterative synthesis cycles, finding improved executability and early performance while retaining code-level diversity. Cross-dataset results support similar qualitative trends, while ablations and open questions delimit the method’s scope.

  • Conclusion: Across 22 cycles, iterative generate–evaluate–select–fine-tune shifts the architecture distribution toward more executable, better early-learning models while retaining structural diversity.The novelty criterion measures code-level diversity rather than guaranteed functional diversity.
  • Generalization: Cross-dataset experiments on CIFAR-100 and SVHN reproduce the qualitative pattern across class counts, visual domains, and task difficulty.These experiments support generality beyond CIFAR-10 within the evaluated benchmark scope.
  • Future work: Open questions include whether single-epoch proxy rankings persist after full training, whether architectures compete with NAS methods under matched budgets, and whether code-level diversity extends broadly.These questions define concrete directions for evaluating the method beyond its current controlled setting.
  • Future work: Future extensions include coupling the refined generator to explicit search frameworks, evaluating higher-resolution and non-classification tasks, using richer feedback, and adding multi-objective constraints.The proposed directions aim to broaden both evaluation and optimization of the learned architectural prior.

6. Limitations

The study’s limitations concern benchmark scope, reliance on first-epoch accuracy, incomplete comparisons with established NAS, limited qualitative coverage, fixed adaptation settings, and token-level novelty measurement.

  • Benchmark scope: Evaluation covers only three low-resolution 32×32 image-classification datasets, leaving transfer to higher-resolution, non-visual, segmentation, and detection tasks unclear.The stated boundary is the benchmark domain and resolution, not a demonstrated failure on other tasks.
  • Proxy signal: First-epoch validation accuracy is the sole performance signal, and its ranking under full training schedules remains unvalidated.The study uses the proxy for inexpensive selection, but does not establish that it preserves final-training rankings.
  • Comparison scope: The work characterizes generator evolution rather than providing fully trained accuracy comparisons against established NAS methods.Related fully trained results are included for context, not as matched-budget comparisons.
  • Qualitative coverage: Qualitative inspection finds diverse representative architectures, but broader systematic analysis across a larger sample is still needed.The current evidence spans examples from compact single-block designs to deep residual networks.
  • Fine-tuning rigidity: LoRA uses fixed hyperparameters and a constant acceptance threshold, while later-cycle plateauing suggests alternative curricula or regularization may help.This limitation concerns adaptation rigidity and the observed plateau, not a tested superiority of alternatives.
  • Novelty definition: Token-level source-code novelty can label functionally equivalent models as novel and miss structurally distinct models with similar token patterns.The novelty measure therefore does not directly establish functional or computation-graph diversity.

7. Ethical Considerations

The study is a controlled methodological investigation using source-code data without human participants, personal data, or end-user deployment.

  • Study scope: The experiments use DeepSeek-Coder-7B-Instruct-v1.5 in a closed, controlled architecture-synthesis framework and contain no human participants or personal data.The LEMUR corpus consists of source code and technical metadata without personally identifiable information.
  • Deployment: Real-world deployment of LLM-generated code would require comprehensive security reviews.This is stated as a deployment requirement beyond the controlled methodological study.

Supplementary Material

The supplementary material details corpus preparation, MinHash–LSH novelty filtering, evaluation choices, and cycle management for the iterative synthesis loop.

  • Novelty filtering: The supplement describes tokenizing Python/PyTorch code into lexical shingles, with each architecture represented as contiguous 10-token sequences.The shingle representation supports subsequent approximate similarity calculations.
  • Novelty filtering: MinHash uses 256 permutations to approximate Jaccard similarity, while LSH retrieves potentially similar candidates before verification.The retrieval threshold is 0.85, followed by MinHash-estimated similarity checks.
  • Novelty filtering: Architectures are marked near-duplicates when estimated Jaccard similarity exceeds τ = 0.90, consistently across curation and sampling.The same threshold governs lexical or structural duplicate checks.
  • Evaluation order: Novelty filtering occurs only after successful parsing, instantiation, forward execution, and one training epoch.This ordering avoids spending LSH queries on invalid candidates.
  • Novelty filtering: A cycle-local MinHash archive stores accepted candidates to support efficient within-cycle novelty computation.The archive is maintained in memory during each sampling cycle.
  • Diagnostics: The supplement records rejected candidates before each accepted non-duplicate architecture as a measure of near-duplicate proposal propensity.This rejection count is measured under fixed decoding settings.
  • Corpus preparation: MinHash–LSH deduplication reduces 109,913 raw LEMUR records to 1,065 unique records by retaining one representative per near-duplicate cluster.The reduction removes redundant variants while preserving architectural diversity for fine-tuning.
  • Corpus preparation: The retained 849 records span convolutional, residual, depthwise-separable, lightweight, and multi-branch architecture families across image-classification tasks.This diversity initializes the generator without strong family-level bias.

F. Detailed Ablation Analysis

The ablations show that novelty filtering preserves structural exploration, performance filtering accelerates learning-oriented refinement, and iterative fine-tuning drives progressive gains before late-cycle saturation.

  • Removing the novelty filter: Without novelty filtering, accuracy trends remain qualitatively similar, but repeated motifs accumulate and genuinely distinct code patterns decline.The ablation identifies novelty filtering as central to sustaining exploration beyond a narrow family of designs.
  • Removing the accuracy threshold: Without the 40% accuracy threshold, the fraction of models exceeding 40% improves more slowly and stabilizes lower.Novelty alone admits valid models across a broad performance range, so performance-based selection is required to align corpus growth with empirical learning behavior.
  • Removing iterative fine-tuning: Without iterative fine-tuning, valid generation and first-epoch accuracy remain near early-cycle behavior, while high-accuracy models never dominate later generations.Novel architectures can still appear, but they do not influence future generations when the feedback loop is disabled.
  • Late-cycle saturation: After cycle 18, best and mean first-epoch accuracies plateau, and the fraction above 40% stabilizes.This late-cycle saturation is consistent with diminishing returns from continued self-training as generated samples increasingly dominate the training set.

G.3. On Pre-training Bias and Architectural Exploration

The paper argues that iterative refinement is not severely confined to pretrained architectural motifs: novelty filtering rejects close code matches, novel admissions increase, and exploration transfers across datasets.

  • Pre-training bias: The base model was pretrained on general code rather than curated architecture data, biasing it toward valid Python and PyTorch idioms rather than one architectural family.Architectural patterns form only a small and heterogeneous portion of the pretraining distribution.
  • Novelty constraint: The MinHash–Jaccard filter rejects candidates exceeding τ = 0.90 similarity to architectures in the seed corpus, making accepted models code-level dissimilar.Across 22 CIFAR-10 cycles, 455 architectures were admitted under this novelty criterion.
  • Exploration dynamics: Novel admissions rise from 1 in cycle 1 to 40 in cycle 17, while structural fingerprints diversify across cycles.This pattern suggests the feedback loop expands rather than contracts the explored design region.
  • Cross-dataset evidence: Comparable novel-architecture admission rates on CIFAR-100 and SVHN indicate that exploration is not confined to CIFAR-10-specific motifs.The cross-dataset result extends the architectural exploration pattern across different visual domains and task difficulty levels.

H. Qualitative Analysis of Discovered Architectures

The generated architectures span compact and deep designs, residual and plain networks, and varied computational graphs while satisfying a fixed implementation contract.

  • Structural diversity: Architecture C combines one convolutional feature block, global average pooling, and a linear classifier, achieving 58.1% first-epoch accuracy.Architecture A instead uses four widening multi-stage Conv–BN–ReLU stages with downsampling and achieves 62.8%.
  • Emergence of residual connections: Architecture B independently produces residual identity shortcuts with conditional downsampling in a deep 3-stage layout.Its forward pass explicitly adds the identity branch and uses a separate downsampling path when dimensions change.
  • Variation in design philosophy: Architecture D uses uniform dual-Conv–BN–ReLU stages without skip connections, then applies global average pooling before a single linear layer.Its contrast with Architecture B shows diversity between plain and residual designs even among similarly organized models.
  • Summary: The examples exhibit combinatorial novelty and operational diversity, including residual shortcuts, dropout, and alternative classifier structures.These variations correspond to distinct computational graphs rather than superficial code edits.
  • Implementation contract: Every model must implement a fixed Net(nn.Module) API, supported hyperparameters(), and a single module definition without data loaders or training loops.Pretrained weights and external feature extractors are prohibited; standard convolutional and pooling components are permitted.

J. Full Hyperparameter Specifications

The experimental specification fixes LoRA and decoding settings, defines validity and accuracy statistics, documents compute requirements, and records reproducibility, privacy, and code-security boundaries.

  • LoRA fine-tuning: LoRA targets attention and MLP projections across all 24 Transformer layers with rank r=32, α=32, dropout 0.05, and five training epochs.Training uses causal language modeling over chat-format prompt–response pairs with learning rate 1 × 10−5 and effective batch size 4.
  • Decoding configuration: Generation uses temperature 0.20, top-k 50, nucleus p=0.9, sampling, and a 2,048-token maximum.These decoding settings remain fixed across cycles so behavioral changes are attributable to the evolving training corpus.
  • Generated architecture coverage: Table 8 contains representative architectures from different design families, each passing the MinHash–Jaccard novelty filter against the seed corpus and other generated models.The architectures were generated within the 22-cycle CIFAR-10 synthesis loop.
  • Validity and uncertainty: The valid generation rate is defined from generated candidates and the subset that compile and train successfully.The statistical protocol also forms accuracy sets over valid models and reports t-based 95% confidence intervals for sample means.
  • Validity and uncertainty: Wilson score confidence intervals are used for proportions such as validity or the share of models meeting an accuracy threshold.This complements the t-based intervals reported for sample means.
  • Compute requirements: On one RTX 4090, candidate generation takes 0.7–1.5 GPU-minutes per sample, proxy evaluation takes 1–5 GPU-minutes per valid model, and the full 22-cycle evolution is estimated at 90–266 GPU-hours.LoRA fine-tuning over 2,153 pairs takes approximately 2–4 hours.
  • Privacy and security: The initial corpus contains source code and technical metadata without PII, while generated artifacts are produced in an isolated execution environment.Real-world reuse of generated code would still require comprehensive security reviews for unsafe or vulnerable patterns.
  • Reproducibility: The documented setup covers the base LLM, LoRA fine-tuning, novelty filtering, and evaluation protocols to support independent verification and replication.The documentation is also intended to enable extensions of the methodology.
Loading 2601.02997v2…