PaperlayerView exact revision on arXiv

Source-linked AI summary

EvoPrompting: Language Models for Code-Level Neural Architecture Search

Angelica Chen, David M. Dohan, David R. So

arXiv:2302.14838v3cs.NEcs.AIcs.CLcs.LG

TL;DR

Neural architecture search remains difficult for language models using prompting alone, motivating a method that combines evolutionary prompt engineering with soft prompt-tuning. EVOPROMPTING produces effective architectures on MNIST-1D and CLRS, including models outperforming established baselines on 21 of 30 CLRS tasks, while the study notes limited comparison with standard NAS techniques.

  • Problem

    Language models perform poorly at designing novel and effective neural architectures through naive few-shot prompting, leaving this application insufficiently solved.

  • Method

    EVOPROMPTING uses evolutionary search to create and curate prompting examples while prompt-tuning a language model for neural architecture design.

  • Results

    EVOPROMPTING produces convolutional architectures superior to published manually designed models and graph neural networks outperforming Triplet-GMPNN on 21 out of 30 CLRS tasks.

  • Takeaways & Limitations

    EVOPROMPTING discovers architectures that optimize accuracy and model size across convolutional and graph neural network tasks and is described as adaptable beyond neural architecture search.

  • Takeaways & Limitations

    The study lacks an extensive comparison against standard NAS techniques because their differing search settings could introduce a confounder.

Abstract

from arXiv · show

Given the recent impressive accomplishments of language models (LMs) for code generation, we explore the use of LMs as adaptive mutation and crossover operators for an evolutionary neural architecture search (NAS) algorithm. While NAS still proves too difficult a task for LMs to succeed at solely through prompting, we find that the combination of evolutionary prompt engineering with soft prompt-tuning, a method we term EvoPrompting, consistently finds diverse and high performing models. We first demonstrate that EvoPrompting is effective on the computationally efficient MNIST-1D dataset, where EvoPrompting produces convolutional architecture variants that outperform both those designed by human experts and naive few-shot prompting in terms of accuracy and model size. We then apply our method to searching for graph neural networks on the CLRS Algorithmic Reasoning Benchmark, where EvoPrompting is able to design novel architectures that outperform current state-of-the-art models on 21 out of 30 algorithmic reasoning tasks while maintaining similar model size. EvoPrompting is successful at designing accurate and efficient neural network architectures across a variety of machine learning tasks, while also being general enough for easy adaptation to other tasks beyond neural network design.

1 Introduction

EVOPROMPTING combines evolutionary search with in-context prompting and prompt-tuning to improve language models' ability to design neural architectures. It outperforms naive few-shot prompting and produces strong convolutional and graph neural architectures across MNIST-1D and CLRS tasks.

  • Method: The search begins from manually designed program seeds, evaluates generated architectures, selects fit examples, and prompt-tunes on evaluated individuals before repeating.This loop combines candidate generation, task evaluation, evolutionary selection, and prompt-tuning.
  • Results: EVOPROMPTING produces conventional convolutional architectures superior to published manually designed models on MNIST-1D.The experiments use MNIST-1D as a relatively low-compute setting for extensive testing and analysis.
  • Method: EVOPROMPTING evolves in-context examples and prompt-tunes a language model to generate improved neural architectures.The method uses evolutionary search to create and curate data for prompting, while remaining applicable to tasks using in-context learning or prompt-tuning.
  • Motivation: Naive few-shot prompting alone is unsuccessful for neural architecture design, whereas EVOPROMPTING enables novel and effective architectures.The contrast motivates combining prompting with evolutionary prompt engineering and prompt-tuning.
  • Results: EVOPROMPTING-generated graph neural networks outperform Triplet-GMPNN on 21 out of 30 CLRS Algorithmic Reasoning Benchmark tasks.The discovered architectures are novel and are evaluated against a current state-of-the-art architecture.

2 Related Work

Prior work established language-model prompting, code generation, and evolutionary neural architecture search as relevant foundations. EVOPROMPTING extends these directions by using language models to search over arbitrary code strings rather than relying only on hand-crafted architecture spaces.

  • Language models: Language models have demonstrated capabilities in code generation, mathematics, and complex reasoning, while their ability to create novel solutions remains limited.This literature motivates methods that improve language-model performance on complex design problems.
  • Prompting: In-context prompting steers language-model decoding with task instructions or examples without gradient updates.Prompt optimization and retrieval-based augmentation are related approaches for improving prompted performance.
  • Evolutionary algorithms: Evolutionary neural architecture search represents architectures as discrete genotypes and filters them using fitness metrics.Conventional methods typically depend on manually designed search spaces containing predefined components or primitives.
  • Evolutionary algorithms: EVOPROMPTING differs from conventional evolutionary NAS by searching over arbitrary strings of code rather than a hand-crafted search space.This places the language model in the role of generating architectural variations within an evolutionary process.

3 EVOPROMPTING Method

EvoPrompting combines evolutionary search with language-model crossover, mutation, and prompt-tuning to discover neural architectures in a broad code space. It evaluates, filters, selects, and uses generated architectures to iteratively improve the prompting model.

  • Search formulation: EvoPrompting generates neural architecture code with an LM and uses evolutionary search to iteratively score and select candidates.The method improves conventional evolutionary NAS by using an LM for crossover and mutation operations.
  • Search formulation: The search space includes any neural network architecture representable in Python rather than a manually specified discrete component space.This removes the need to restrict evolution to hand-crafted modules, primitives, or operations.
  • Evaluation and selection: Fitness is defined as the negative product of validation error and model size, balancing predictive performance against architectural compactness.A validation-error threshold filters candidates before fitness-based selection, limiting exploitation through arbitrarily small models.
  • Meta-learning loop: The algorithm initializes a population with seed architectures, generates children through few-shot prompts, evaluates them, and retains top candidates across evolutionary rounds.Potential parents are selected from prior high-fitness examples, while generated children provide data for subsequent prompt-tuning.
  • Mutation and crossover: Parent source code and evaluation metrics condition prompts toward children targeted to higher validation accuracy and smaller model size.The target model size is set to 90% of the parents’ minimum size, while target validation accuracy is set to 102% of their maximum.
  • Meta-learning loop: After evaluation, the highest-fitness candidates are selected for crossover, while other trainable children prompt-tune the LM for the next evolutionary round.Selected parents are permanently removed from the population as reusable crossover parents.

4 Experiments and Results

Across MNIST-1D and CLRS, EvoPrompting searches neural architectures with evolutionary prompting, prompt-tuning, and language-model-generated code. It finds smaller, more accurate convolutional models, improves sample-efficient search, and discovers CLRS architectures that generalize across tasks.

  • Evaluation setup: EvoPrompting evaluates neural architectures on lightweight MNIST-1D and the 30-task CLRS algorithmic reasoning benchmark.MNIST-1D enables more extensive analysis, while CLRS tests generalization across algorithms and architecture classes.
  • Search procedure: The search uses a 62B-parameter code-pretrained PaLM crossover operator, prompt-tuned between evolutionary rounds.Unless otherwise stated, each run uses 10 rounds, 10 prompts per round, and 16 samples per prompt.
  • MNIST-1D results: EvoPrompting’s Pareto frontier is closest to the origin, finding top-20 MNIST-1D models with lower test error and substantially smaller size than seed models and baselines.Many discovered models are orders of magnitude smaller than alternatives while still achieving lower test error.
  • MNIST-1D results: EvoPrompting especially improves convolutional architectures by producing narrower, deeper models with smaller strides, less padding, and no dense layers.These designs consistently outperform shallower, denser, wider convolutional architectures found earlier in the search.
  • Search efficiency and ablations: EvoPrompting reaches the highest fitness plateau, whereas random-parent search plateaus near 200 generated individuals at significantly worse fitness.Removing prompt-tuning or evolutionary parent selection also reduces performance, showing that the full combination matters.

5 Conclusion

EVOPROMPTING improves language-model performance for neural architecture design across convolutional and graph architectures, optimizing accuracy and model size. The study’s comparison with standard NAS techniques remains limited because open-ended search introduces a potential confounder.

  • The method optimizes both accuracy and model size when discovering convolutional and graph neural architectures.
  • The approach is presented as adaptable to reasoning tasks beyond neural architecture search, but those adaptations remain future work.
  • Extensive comparison with standard NAS techniques is limited because open-ended and conventional searches differ, creating a potential confounder.

A.1 EVOPROMPTING Hyperparameters

This section identifies the hyperparameter values used in EVOPROMPT, but the supplied passage provides only the table caption.

  • Table 2 reports the hyperparameter values used in EVOPROMPT.
  • The supplied passage does not enumerate the individual hyperparameter settings.
  • The table concerns EVOPROMPT rather than the broader EVOPROMPTING search procedure.

A.2 MNIST-1D Seed Models

The MNIST-1D search uses four seed-model implementations: one hand-designed convolutional model and three baselines from Greydanus.

  • Four seed-model source-code implementations are provided for the MNIST-1D model search.
  • One seed model is a hand-designed convolutional model.
  • The convolutional baseline from Greydanus is implemented with Flax.
  • The GRU baseline from Greydanus is implemented with Flax.
  • The fully connected baseline from Greydanus is implemented with Flax.

A.3 Trajectory of search for MNIST-1D models

Figure 4 tracks the average model size and test error of child models across search rounds. Rounds closer to the origin produced more fit models.

  • Figure 4 plots average model size against test error for child models produced in each search round.
  • Data points closer to the origin represent rounds yielding more fit models.
  • The figure summarizes search-round trajectories using child-model averages rather than individual model results.

A.4 Newly Discovered CLRS GNNs

This section presents five newly discovered graph neural network models and compares their maximum fitness scores with a baseline across eight CLRS tasks.

  • Five newly discovered models are compared against a baseline on eight CLRS tasks using maximum fitness scores.
  • The newly discovered models are provided as Python source code for inspection and reuse.
  • QUADNODEMINMAX is one of the newly discovered triplet representations.
  • CONCATREP is one of the newly discovered triplet representations.
  • TANHEXPANDTRIPLETS, DIV2MEAN, and MAXMEAN are additional newly discovered triplet representations.

A.5 CLRS Seed Models

This section documents the CLRS seed models, compares discovered GNNs with Triplet-GMPNN across all 30 tasks, and reports broader evaluation boundaries and impacts.

  • CLRS Seed Models: The CLRS model search uses nine seed models, including the standard Triplet-GMPNN representation and variants changing output layers, aggregation, or node, edge, and graph representations.
  • CLRS Seed Models: Additional seed variants remove graph representations, tile a linear-layer output, or use a bilinear node, edge, and graph representation.
  • OOD Evaluation: Table 3 compares five newly discovered GNNs with Triplet-GMPNN on all 30 CLRS tasks using out-of-distribution accuracy.
  • OOD Evaluation: The comparison includes out-of-distribution accuracy from both the authors' Triplet-GMPNN implementation and the original paper.
  • NATS-Bench: The NATS-Bench comparison is limited because EVOPROMPTING is evaluated in a search space that removes the benefit of code pre-training and its open-ended search-space advantage.
  • NATS-Bench: EVOPROMPTING performs competitively against other standard neural architecture search techniques on the NATS-Bench size search space.
  • Broader Impacts: The paper notes potential benefits of improving and reducing language models alongside risks including toxic language, misinformation, privacy disclosure, overreliance, and malicious use.
Loading 2302.14838v3…