PaperlayerView exact revision on arXiv

Source-linked AI summary

Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution

Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, Tim Rocktäschel

arXiv:2309.16797v1cs.CLcs.AIcs.LGcs.NE

TL;DR

Promptbreeder addresses the limitations of hand-crafted prompt strategies by evolving prompts for a domain through LLM-driven, self-referential mutation. It jointly improves task-prompts and the mutation-prompts governing them, outperforming established strategies on arithmetic and commonsense benchmarks while also evolving prompts for hate speech classification.

  • Problem

    Prompt strategies can substantially affect LLM performance, but existing strategies are often manually engineered and may be sub-optimal for a given domain.

  • Method

    Promptbreeder uses evolutionary search to mutate task-prompts and mutation-prompts, evaluating task-prompt fitness on training data and selecting units for later generations.

  • Results

    Promptbreeder outperforms state-of-the-art prompt strategies across arithmetic and commonsense reasoning benchmarks and evolves complex domain-specific prompts for hate speech classification.

  • Takeaways & Limitations

    Promptbreeder improves prompts while also improving how prompts are improved, enabling self-referential domain adaptation in natural language.

  • Takeaways & Limitations

    Promptbreeder adapts prompt content but keeps the prompting algorithm’s topology fixed, limiting its open-endedness relative to human thought processes.

Abstract

from arXiv · show

Popular prompt strategies like Chain-of-Thought Prompting can dramatically improve the reasoning abilities of Large Language Models (LLMs) in various domains. However, such hand-crafted prompt-strategies are often sub-optimal. In this paper, we present Promptbreeder, a general-purpose self-referential self-improvement mechanism that evolves and adapts prompts for a given domain. Driven by an LLM, Promptbreeder mutates a population of task-prompts, and subsequently evaluates them for fitness on a training set. Crucially, the mutation of these task-prompts is governed by mutation-prompts that the LLM generates and improves throughout evolution in a self-referential way. That is, Promptbreeder is not just improving task-prompts, but it is also improving the mutationprompts that improve these task-prompts. Promptbreeder outperforms state-of-the-art prompt strategies such as Chain-of-Thought and Plan-and-Solve Prompting on commonly used arithmetic and commonsense reasoning benchmarks. Furthermore, Promptbreeder is able to evolve intricate task-prompts for the challenging problem of hate speech classification.

1 INTRODUCTION

Promptbreeder addresses the limitations of manually engineered prompting by evolving domain-adaptive task-prompts and the mutation-prompts that improve them. It uses LLM-driven evolutionary search and reports stronger performance than established prompt strategies across reasoning benchmarks.

  • Motivation: Manually engineered prompt strategies can substantially affect LLM reasoning abilities, yet their phrasing is often hand-designed and potentially sub-optimal.Prompting also influences multimodal processing, tool use, distillation, and agentic behavior.
  • Motivation: Promptbreeder treats prompts as an LLM’s modifiable program and applies self-referential improvement to both task-prompts and the process that modifies them.This extends the analogy from changing model behavior through prompts to using the LLM itself to change those prompts and their mutation process.
  • Method: Promptbreeder initializes mutation-prompts, thinking-styles, and a domain-specific problem description, then selects evolutionary units according to task-prompt fitness on training data.Each unit contains task-prompts and an associated mutation-prompt, with selected units transmitted to later generations.
  • Results: Promptbreeder improves over state-of-the-art prompt strategies on commonly used arithmetic and commonsense reasoning benchmarks.The paper also investigates how self-referential components contribute to the reported results.
  • Method: Promptbreeder evolves increasingly domain-adaptive task-prompts while simultaneously improving the mutation-prompts that govern their evolution.The overview describes mutations to both prompt types across multiple generations using five mutation-operator classes.

2 RELATED WORK

Related work shows that prompting improves LLM capabilities but is largely domain-agnostic and manually designed. Promptbreeder instead automates domain adaptation through evolutionary mutation of task-prompts and mutation-prompts, differing from existing optimization and self-improvement approaches.

  • Prompting: Chain-of-Thought and related strategies improve arithmetic, commonsense, symbolic, and other reasoning abilities by structuring intermediate work.The literature includes zero-shot, self-consistent, tree-structured, graph-structured, and plan-and-solve variants.
  • Automated Prompt Engineering: Existing prompt-engineering approaches are generally domain agnostic and hand designed, motivating automated adaptation to the domain at hand.Automatic methods such as Auto-CoT and APE reduce manual effort but remain distinct from Promptbreeder’s broader evolutionary process.
  • Automated Prompt Engineering: Unlike APE, Promptbreeder initializes mutation-prompts compositionally, mutates them online, uses population and elite-history information, and maintains diversity.These design choices are presented as addressing diminishing returns and diversity loss in APE.
  • Automated Prompt Engineering: Unlike OPRO’s single complex mutation-prompt and fixed small training set, Promptbreeder evolves multiple LLM-generated mutation-prompts and evaluates random subsets of the whole training set.The comparison concerns both the mutation mechanism and the fitness-evaluation procedure.
  • Self-Referential Self-Improvement: Prior self-referential systems modify model parameters, whereas Promptbreeder uses natural language prompts as the substrate of self-referential improvement.This distinction is presented as avoiding costly parameter updates for modern LLMs.
  • Open-Endedness and LLMs: Promptbreeder also relates to open-ended systems such as Picbreeder but explores prompts without humans in the loop and proposes mutations to itself.This connects prompt evolution with the transition from learning from data to learning what data to learn from.

3 PROMPTBREEDER

Promptbreeder evolves task-prompts and the mutation-prompts that generate them, using LLM-driven evolutionary operators to adapt prompt strategies to a domain. Its population includes task-prompts, mutation-prompts, and, in few-shot settings, correct workings out.

  • 3 PROMPTBREEDER: Promptbreeder automatically explores task-prompts for a given domain and can adapt the same system across domains.
  • 3 PROMPTBREEDER: A task-prompt is evaluated by sampling a batch of 100 Q&A pairs from the domain’s entire training set.
  • 3 PROMPTBREEDER: Task-prompts mutate as P′ = LLM(M + P), while hyper-mutation produces mutation-prompts as M′ = LLM(H + M).
  • 3 PROMPTBREEDER: Each evolutionary unit contains task-prompts, one mutation-prompt, and, in few-shot settings, correct workings out.
  • 3.1 PROMPTBREEDER INITIALIZATION: Initialization combines thinking styles, mutation-prompts, and a domain description to generate a diverse population of task-prompts.
  • 3.2 MUTATION OPERATORS: Nine mutation operators across five classes explore prompt strategies by changing problem framing and retrieving linguistic mental models.
  • 3.2.1 DIRECT MUTATION: Zero-order generation creates task-prompts from the domain description without depending on previously found task-prompts.
  • 3.2.2 ESTIMATION OF DISTRIBUTION MUTATION: Population-level operators use filtered prompt sets, fitness ordering, or chronological elite lineages to generate new task-prompts.

4 EXPERIMENTS

Promptbreeder is evaluated with an evolutionary population on datasets spanning arithmetic reasoning and other commonly used reasoning benchmarks.

  • Promptbreeder uses a population of 50 units evolved for typically 20–30 generations.Each generation forms random pairs of all population members and makes them compete.
  • Evaluation uses datasets spanning arithmetic reasoning, including GSM8K, SVAMP, MultiArith, AddSub, and AQuA-RAT.

5 RESULTS AND DISCUSSION

Promptbreeder outperforms the strongest Plan-and-Solve comparison across the evaluated reasoning benchmarks, with additional few-shot gains and strong domain adaptation on hate speech classification. Analyses also indicate that its mutation operators and self-referential components contribute materially to performance.

  • Promptbreeder outperforms PS+, the best Plan-and-Solve prompting technique, on the reported reasoning benchmarks.
  • Zero-shot Promptbreeder accuracy exceeds PS+ on all datasets except AddSub, while few-shot examples further improve performance.
  • 89% accuracy on ETHOS exceeds the hand-designed hate-speech prompt’s 80%, demonstrating intricate domain adaptation.
  • Mutation-prompt scores measure how often applying a mutation-prompt produces a better task-prompt.
  • All mutation operators are important for Promptbreeder, including hypermutation operators responsible for self-referential self-improvement.
  • Removing any self-referential operator is harmful under nearly all circumstances, with initialization-time task-prompt redescription providing the greatest benefit.
  • Randomly sampling mutation-prompts at initialization is harmful for GSM8K, the sole reported harmful operator-task combination.

6 CONCLUSION AND FUTURE WORK

Promptbreeder evolves both domain-specific task-prompts and the mutation-prompts that govern their improvement. The paper frames this language-based approach as a step toward more open-ended self-referential self-improvement, while acknowledging important scope limits.

  • Promptbreeder evolves task-prompts and mutation-prompts, thereby improving both prompts and the process used to improve them.
  • Future work could evolve more complex conditional thought processes, promote prompt diversity, or use self-play to evolve pre-prompts for competing policies.
  • Promptbreeder remains limited because its prompting topology is fixed and it adapts prompt content rather than the prompting algorithm itself.
  • The approach uses language as the substrate for self-referential improvement instead of relying on parameter updates.

B A TYPICAL EVOLUTIONARY RUN

A typical evolutionary run evolves a two-prompt strategy for the word-in-context task, with fitness improving across evaluations. The run illustrates how mutation prompts and thinking styles generate increasingly specialized task prompts.

  • A TYPICAL EVOLUTIONARY RUN: The word-in-context task asks whether a homograph has the same meaning in two sentences, using a two-prompt strategy evolved over 2000 evaluations.
  • A TYPICAL EVOLUTIONARY RUN: Fitness increases considerably during the run, with blue dots representing individual training-set evaluations and the red line representing population mean fitness.
  • A TYPICAL EVOLUTIONARY RUN: The best evolved strategy uses two sequential prompts that independently ask for a yes-or-no judgment about whether the word has the same sense or meaning.
  • MUTATION PROMPTS: Mutation prompts include requests to rephrase instructions, encourage creativity, add perspectives, seek detailed responses, combine prompts, and decompose complex questions.
  • THINKING STYLES: Thinking styles supply general heuristics such as experimentation, progress measurement, assumption checking, risk analysis, and problem decomposition.
  • PROMPTBREEDER AS SELF-REFERENTIAL SELF-IMPROVEMENT SYSTEM: Promptbreeder’s self-referential pathways let prompts, mutation-prompts, workings out, and LLM parameters influence the generation of later prompts.

I.1 CONTROL TASK-PROMPTS

The appendix documents control prompts and benchmark datasets used across arithmetic, commonsense, hate-speech, and instruction-induction evaluations. It also lists evolved two-stage task-prompts for arithmetic tasks.

  • I.1 CONTROL TASK-PROMPTS: Table 5 lists control task-prompts for Chain-of-Thought, Plan-and-Solve, PS+, zero-shot APE, and OPRO.
  • I.2 ARITHMETIC REASONING: The arithmetic evaluation covers GSM8K, SVAMP, MultiArith, AddSub, AQuA-RAT, and SingleEq.
  • I.3 COMMONSENSE REASONING: Commonsense reasoning is evaluated with CommonsenseQA and StrategyQA, including multiple-choice commonsense questions and yes/no reasoning tasks.
  • I.4 HATE SPEECH CLASSIFICATION: The hate-speech experiment optimizes a long prompt on ETHOS, paralleling a prior self-referential error-detection and error-fixing approach.
  • I.5 INSTRUCTION INDUCTION: Instruction Induction contains 24 language-understanding tasks spanning spelling, morphosyntax, similarity, causality, style transfer, and sentiment analysis.
  • I.5 INSTRUCTION INDUCTION: Table 6 lists two-stage task-prompts evolved for different arithmetic tasks.

J EXAMPLE RESULTS

The example-results section describes ETHOS prompts, experimental settings, and mutation-prompt analysis. Promptbreeder uses evolved prompts and reports mutation operators by their improvement rates.

  • J.1 ETHOS EVOLVED PROMPT: ETHOS examples show evolved prompts defining hate speech through derogatory, generalized, hostile, discriminatory, or violence-inciting language.
  • J EXAMPLE RESULTS: Promptbreeder uses a population of 50, evaluates accuracy on random batches of 100 training examples, and splits unsplit datasets into equal training and test sets.
  • J EXAMPLE RESULTS: Redescriber generates prompts, Inducer generates responses, and Evaluator produces final outputs, with maximum sampled-token budgets of 50, 30, and 5.
  • J EXAMPLE RESULTS: Experiments stop after training fitness plateaus, with runs generally using 1–2k fitness evaluations.
  • J EXAMPLE RESULTS: Diversity maintenance uses random prefix strings, BERT-similarity fitness sharing, and evolving mutation temperature when local optima trap the system.
  • J.2 PROMPT EVOLUTION MATHS RESULTS: The paper does not provide model-based comparisons with PoT, PS, and Auto-CoT because PS and PS+ were the strongest Plan-and-Solve prompts.
  • J.3 EVOLVED MUTATION PROMPTS: Table 7 ranks mutation-prompts by the probability that applying them produces an improved prompt, while Table 8 compares mutation-operator improvement proportions on GSM8K.

J.5 ADDSUB

The AddSub examples illustrate evolved prompts and contexts across generations. In few-shot evolution, contexts often dominate fitness while task-prompts may drift into nonsense.

  • J.5 ADDSUB: After 1600 mutations, the individual uses Prompt 0 to produce working-out and Prompt 1 to produce the final answer, as in Plan-and-Solve.
  • J.5 ADDSUB: In few-shot evolution, contexts dominate, and task-prompts frequently drift into nonsense while becoming less decisive for fitness.
  • J.5 ADDSUB: Another evolved context solves the paint problem as 6689 - 660 = 6029 gallons of blue paint.
  • J.5 ADDSUB: The evolved contexts include arithmetic solutions, irrelevant game instructions, and prompts directing the model to solve or confirm multiple-choice problems.
  • J.5 ADDSUB: A separate context combines 6 + 8 + 7 seashells to obtain 21 total seashells despite intervening mutated text.
  • J.5 ADDSUB: For the seashell example, the solution computes 7 - 4 = 3 unbroken seashells.

J.6 AQUA

The AQuA examples show evolved arithmetic prompts ranging from direct solving instructions to generic or unrelated text. Several contexts still produce worked solutions and selected answers.

  • J.6 AQUA: The evolved AQuA prompts include instructions to understand multiple-choice problems, solve them, and choose among options.
  • J.6 AQUA: One ratio example computes 3/8 × 32000 = 12000 and 5/8 × 32000 = 20000, selecting option A.
  • J.6 AQUA: A lizard-speed example derives a 1.5 speed ratio and concludes with option E.
  • J.6 AQUA: A simple-interest example computes a principal of Rs. 8925 and selects option B.
  • J.6 AQUA: Other evolved prompts ask for Arabic-numeral answers, recall similar problems, or solve without a calculator.
  • J.6 AQUA: The examples include correct arithmetic solutions for allowance, furniture, sibling, bracelet, and cookie-cost problems.
  • J.6 AQUA: Some evolved prompts are unrelated quotations or inverse-variation instructions, and contexts contain unrelated text alongside arithmetic questions.

K APE INSTRUCTION INDUCTION TASKS

Promptbreeder was evaluated on all 24 Instruction Induction datasets used in APE experiments, evolving both task-prompts and few-shot contexts. It matched or surpassed APE on 21 of 24 tasks, with improvements up to 21%.

  • K APE INSTRUCTION INDUCTION TASKS: 21 of 24 Instruction Induction tasks matched or surpassed APE, with improvements up to 21%.The comparison used Promptbreeder with PaLM2-L against APE controls, including published text-davinci-002 results and an APE-style Promptbreeder variant.
  • K APE INSTRUCTION INDUCTION TASKS: Promptbreeder evolved few-shot contexts as well as task-prompts across all 24 Instruction Induction datasets.The LLM used in these experiments was not instruction tuned, unlike text-davinci-002 in the published APE results.
  • K APE INSTRUCTION INDUCTION TASKS: Because these datasets lack problem descriptions, task-prompt initialization used randomly chosen induction input examples.Each induction input combines a fixed prompt with several training examples from which possible problem descriptions can be inferred.
  • K.1.1 FIRST LETTER: The evolved First Letter prompt instructs the model to write the first letter of each input.Its accompanying context compares candidate responses and gives the answer for “drummer” as “d.”
  • K.1.2 SECOND LETTER: The evolved Second Letter prompt repeatedly instructs the model to give the input’s second letter.Examples for “month,” “asylum,” and “heel” provide the corresponding second-letter answers.
  • K.1.3 LIST LETTERS: For List Letters, one evolved prompt inserts spaces after consonants, while another returns the first word containing a specified letter.Examples apply these instructions to words and sentences, including “gale,” “accompaniment,” “credibility,” and review text.
  • K.1.5 PLURALIZATION: The Pluralization prompts use either a direct pluralization instruction or a rule replacing final y with ies and otherwise adding s.The rule produces “forages,” “miles,” and “touches” for the displayed examples.
  • K.1.6 PASSIVIZATION: For Passivization, evolved prompts swap noun-phrase positions, insert “by,” and conjugate the verb with an “ed” ending when appropriate.Displayed examples transform active sentences into forms such as “The presidents were stopped by the authors.”

K.1.21 SENTIMENT ANALYSIS

The sentiment-analysis examples use prompts for classifying review language and counting positive examples. The displayed contexts apply these instructions to reviews with positive or negative wording.

  • K.1.21 SENTIMENT ANALYSIS: One prompt asks whether each review uses generally positive or negative language.It frames the task as sentiment classification for individual reviews.
  • K.1.21 SENTIMENT ANALYSIS: A second prompt asks the model to write the number of positive examples.The displayed context applies this instruction to the review “I simply can’t recommend it enough.”
  • K.1.21 SENTIMENT ANALYSIS: The displayed examples classify “I simply can’t recommend it enough” and “There’s a lot to recommend Read My Lips” as positive.Both contexts present the answer label as “positive.”

L ABLATIONS

Promptbreeder ablations test how its self-referential initialization and mutation mechanisms affect fitness. Across most cases, removing mutation operators reduced performance, with the Lamarckian operator especially important on ETHOS.

  • L ABLATIONS: The ablation study evaluates Promptbreeder’s mutation operators and self-referential mechanisms against the complete algorithm.Runs use populations of 10 for 200 evaluations, comparing ablations with the same population size and no ablations.
  • L ABLATIONS: The tested removals include random initial prompts, random initial mutation-prompts, context-to-task-prompt mutation, and meta-mutation.The latter means mutating mutation-prompts rather than applying the default prompt mutation operation.
  • L ABLATIONS: Most mutation operators had a positive impact on fitness, so their removal generally produced lower fitness at equivalent points in the run.Figure 4 interprets values near −100% as lower fitness after ablation and values near 100% as harmfulness of the removed operation.
  • L ABLATIONS: 81.6% was Promptbreeder’s ETHOS score, while removing Lamarckian context-to-prompt mutation reduced it to 64.6%.Removing several initialization and hyper-mutation mechanisms together yielded 68.7%, with additions restoring performance to 70.4% and then 73.7%.
Loading 2309.16797v1…