Prompt Engineering a Prompt Engineer

1 Introduction

• Prompt engineering requires examining model errors, identifying what is missing or misleading, and expressing the task clearly, yet existing meta-prompts are argued to provide insufficient guidance.The introduction presents prompt engineering as a demanding language-generation task and motivates more structured meta-prompts.Source: §1 · pp. 1–2
• PE2 adds detailed two-step task descriptions, context specification, and a step-by-step reasoning template to guide automatic prompt engineering.The paper presents these components as support for the prompt proposal process.Source: §1 · p. 2
• With text-davinci-003 as the task model, PE2 surpasses the zero-shot chain-of-thought prompt by 6.3% on MultiArith and 3.1% on GSM8K, and matches or outperforms Iterative APE and APO in multiple settings.Figure 2 reports consistent improvements over prompt initialization and the most significant gain on Counterfactual Eval.Source: §1 · p. 2, Figure 2 · p. 2
• PE2 can amend erroneous or incomplete prompts, add details, and devise multi-step plans, while its base-8 example also raises a concern about shortcut learning.For movie recommendation, PE2 proposed considering genre, plot, and style; its base-8 arithmetic rules were partially correct.Source: §1 · p. 2

2.1 Prompt Engineering

• Prompt engineering seeks a prompt that maximizes task-model performance on validation examples, using a per-example evaluation function.The paper describes datasets as textual input-output pairs and distinguishes training, development, and test sets.Source: §2.1 · pp. 2–3

2.2 Automatic Prompt Engineering with LLMs

• Automatic prompt engineering initializes candidate prompts manually or by inducing instructions from examples, then uses a meta-prompt to propose new candidates.Manual initialization can use an existing prompt; induction uses training examples and an initialization prompt to generate candidates.Source: §2.2 · p. 3
• Because a proposed prompt can underperform its predecessor, the framework retains the best-performing candidates and generates further prompts from them.At each step, it selects n prior candidates and samples m batches of errors for each; constructing a better meta-prompt is the study’s focus.Source: §2.2 · p. 3

3 Prompt Engineering a Prompt Engineer

• The paper focuses on improving the meta-prompt because it affects the quality of proposed prompts and the overall automatic prompt engineering process.PE2 is named as a prompt-engineered prompt engineer using the introduced components.Source: §3 · p. 3
• Figure 3 provides an example illustrating the meta-prompt components introduced in this section.Source: §3 · p. 3, Figure 3 · p. 4
• PE2’s two-step task description clarifies the steps and expectations upfront, while also guiding the model through specific steps during prompt generation.This contrasts with APO’s brief explanations of inspecting the current prompt and batch, then generating an improved prompt.Source: §3 · pp. 3–4
• Context specification explicitly states how the prompt and input text are laid out together, accommodating prompts that appear before or after the input.Source: §3 · p. 4
• A list of questions guides the proposal model to inspect examples, assess whether the prompt describes the task correctly, and suggest actionable edits when needed.Source: §3 · p. 4

Other Meta-prompt Components We Tried

• The authors also tried verbalized batch-size, step-size, and momentum concepts and a prompt-engineering tutorial, but observations on these additions were mixed.Those results are reported in Appendix B.Source: Other Meta-prompt Components We Tried · p. 4

4.1 Tasks

• The task suite covers math reasoning, instruction induction, BIG-bench Hard, counterfactual arithmetic, chess and syntax, and an internal production classification task.The production task classifies user queries into domains, intents, and slots and uses an engineer-written prompt exceeding 5,000 tokens.Source: §4.1 · pp. 4–5

4.2 Compared Methods

• PE2 is compared with APE, Iterative APE, and APO, which respectively select initialized candidates, paraphrase prompts iteratively, or inspect errors and generate textual feedback before proposing prompts.Source: §4.2 · p. 5

4.3 Expeirment Details

• The default proposal and task models are GPT-4 and text-davinci-003, with GPT-4-turbo and GPT-3.5-turbo-instruct used for later BIG-bench Hard experiments; comparisons use the same model pair across methods.Source: §4.3 · p. 5
• Initialization differs by task: Math Reasoning and BIG-bench Hard use “Let’s think step by step.”, Instruction Induction uses induction initialization, Counterfactual Eval tests both, and production uses an experienced engineer’s prompt.With induction initialization, 30 prompts are generated by pinit for the initial candidate set P (0); T = 3, selecting n = 4 best-performing prompts and proposing m = 4 from each at each timestamp.Source: §4.3 · p. 5

Improved baselines with more recent LLMs.

• Rerunning Zero-shot CoT and APE with text-davinci-003 narrowed their gaps from 3.3% to 1.0% on MultiArith and 2.3% to 0.6% on GSM8K, indicating reduced sensitivity to prompt paraphrasing.Source: Improved baselines with more recent LLMs · p. 5
• PE2 reaches 92.3% accuracy on MultiArith (+6.3% compared to Zero-shot CoT), 64.0% on GSM8K (+3.1%), and improves accuracy by 6.9% over APO across counterfactual tasks.PE2 outperforms APO on 11 out of 12 counterfactual tasks when induction initialization is used.Source: Improved baselines with more recent LLMs · pp. 5–6, Table 2 · p. 6
• With Mistral-7B-Instruct-v0.2 as the task model and gpt-4-turbo as the proposal model, PE2 performs competitively with or surpasses other automated methods.The authors frame this setting as evidence of applicability to open-weight models and a basis for studying prompt sensitivity.Source: Improved baselines with more recent LLMs · pp. 6–7, Table 3 · p. 6

5.2 Ablation Study

• Removing any of PE2’s three meta-prompt components significantly reduces final accuracy, increases proposal-quality variance, and more often yields low-quality prompts.Back-tracking and hard negative sampling show slightly positive effects and are retained; additional explored components were not included in the final method.Source: §5.2 · p. 7, Table 5 · p. 7

5.3 Case Study

• PE2 corrects an erroneous rhyming instruction and breaks movie similarity into criteria such as genre, plot, and actor; it also identifies the need to reference “today” in date understanding.The examples illustrate learning from failures by summarizing key steps and incorporating them into improved prompts.Source: §5.3 · p. 7, Table 6 · p. 8
• On base-8 addition, PE2 can infer a partially correct shortcut that reaches 37% test accuracy, versus 17% before and 28% after optimization with explicit base-8 instructions.The shortcut works on some test cases but is not the intended solution, suggesting similar shortcut-learning failure modes may occur in automatic prompt optimization.Source: §5.3 · p. 8

5.4 Additional Analysis

• PE2 can recover from misleading or irrelevant initial prompts, but its final prompt is worse than with instructive initialization; induction initialization matches “Let’s think step by step” on MultiArith.Source: §5.4 · p. 8
• Automatic prompt engineering has limited effect on multiple-choice Date Understanding but brings significant gains in generative format; optimized prompts show no consistent cross-model generalization.The authors conclude that the optimization method applies to various task models, while final prompts remain specific to the underlying task model.Source: §5.4 · p. 8

• Automatic prompt engineering includes incremental editing, reinforcement learning, algorithmic search, adaptive in-context demonstrations, and LLM meta-prompting.The paper discusses limitations in meta-prompting methods and introduces new components in PE2.Source: §6 · pp. 8–9
• Reasoning research uses intermediate-step prompting and self-reflection, while this work frames prompt engineering itself as language generation requiring complex reasoning.Human prompt engineers inspect failures, form hypotheses, and compose revised prompts.Source: §6 · p. 9
• Self-training work uses LLM-generated reasoning chains for fine-tuning; this paper instead treats textual prompts as parameters and optimizes them with LLMs.The paper describes this approach as a possible case of self-improving.Source: §6 · p. 9

7 Conclusion

• PE2 refines human-written prompts and surpasses established automatic prompt engineering baselines across varied scenarios, notably counterfactual tasks and a production application.The paper also demonstrates applicability with different LLMs.Source: §7 · p. 9
• PE2 makes targeted prompt edits and generates high-quality prompts, as illustrated by the paper’s analyses and case studies.The conclusion attributes improved performance to three introduced meta-prompt components.Source: §7 · p. 9
• Instruction-following improvements and hallucination mitigation remain important, while shortcut learning warrants deeper study of LLM failure modes.The authors also identify optimizing PE2’s own meta-prompt as a future direction.Source: §7 · p. 9

Limitations

• The prompt search budget was small (T = 3, m = 4, n = 4) for cost reasons, and larger budgets or different settings could yield new insights.Performance tended to plateau after three optimization steps, though the common initialization might have been near-optimal and confounded convergence.Source: Limitations · p. 10
• Using proprietary models raises reproducibility concerns because they may be upgraded or discontinued, as well as concerns about possible data contamination.The authors report support for model-agnostic concepts from experiments with different model pairings and a Mistral task model.Source: Limitations · p. 10
• The study focuses predominantly on English tasks, apart from three translation tasks in Instruction Induction, and identifies multilingual research as future work.Source: Limitations · p. 10

A.1 Effect of Initialization

• With induction initialization, PE2 discovered a high quality prompt scoring 84% on MultiArith-Dev, almost matching the 85% achieved by a prompt designed by highly experienced human prompt engineers.Source: §A.1 · p. 13

A.2 Effect of Task Format

• For Date Understanding, optimization adds little in multi-choice format but brings significant gains in the more challenging generative format.The authors hypothesize that multi-choice narrows the output space and, with Zero-shot CoT, brings performance close to saturation.Source: §A.2 · p. 14, Figure 7 · p. 14
• For Movie Recommendation, prompt optimization brings significant performance gains in both tested formats.Optimized prompts include multi-step plans, while outputting an option letter versus an option string can mildly affect accuracy.Source: §A.2 · p. 14, Figure 7 · p. 14
• The effectiveness of automatic prompt optimization depends on factors including task format, difficulty, output-space size, and the task model’s instruction-following ability.Source: §A.2 · p. 14

A.3 Do optimized prompts generalize to other LLMs?

• Optimized prompts did not generalize consistently: they usually beat the original zero-shot CoT on three tested models, but not on gpt-3.5-turbo-instruct.The authors conclude that current methods tend to find model-specific prompts; the study tests five prompts across four models.Source: §A.3 · pp. 14–15, Figure 8 · p. 14

B Other Meta-Prompt Components We Tried

• Additional meta-prompt components produced mixed, inconclusive results: the tutorial showed no significant improvement, while optimizer-inspired concepts helped only occasionally.The 2,500-plus-token tutorial was excluded from the final PE2 because it slowed runtime.Source: B Other Meta-Prompt Components We Tried · p. 16, Providing Detailed Instructions and… · p. 17, Table 11 · p. 17

Providing Detailed Instructions and Context.

• The tutorial and optimizer-inspired additions were not consistently effective under the tested models and settings, and the authors recommend revisiting them as model capability and efficiency improve.Momentum requires multiple optimization steps to accumulate, while these experiments were restricted to three for cost reasons.Source: Providing Detailed Instructions and… · p. 17

C.1 Recent Works

• The authors relate PE2 to OPRO and other prompt-engineering approaches, distinguishing their focus on meta-prompt design from methods such as prompt evolution, tree search, and collaboration.A direct OPRO comparison was unavailable because of access and budget constraints, and the experimental setups differed.Source: §C.1 · pp. 17–18

C.2 Discussion on using PE2 to optimize its own meta-prompt

• Applying PE2 to optimize its own meta-prompt is proposed, but would require substantial task-history data, noisy evaluation without ground-truth labels, and much greater cost.Source: §C.2 · p. 18

D.2 Controlling Prompt Length

• The study uses 50-token prompt limits by default and 200 tokens for counterfactual tasks.The authors note that conclusions about generalization may differ without this constraint.Source: §A.3 · p. 15, §D.2 · p. 18

D.3 Infrastructure and Runtime

• Experiments used OpenAI API models and locally ran Mistral, Yi, and MPT models; one GPT-4/text-davinci-003 optimization took about 90 minutes and cost about $25.A later GPT-4-turbo/gpt-3.5-turbo-instruct setup reduced the reported cost to about $3 per experiment.Source: §D.3 · pp. 18–19

D.4 Tasks and Data

• MultiArith uses a 100/100/400 train/dev/test split, while GSM8K uses 100 training and 100 validation examples sampled from its original training set.The MultiArith split creates a slight discrepancy from past reported results; all methods use the same fixed split.Source: §D.4 · p. 19
• Instruction Induction uses five random 100/20/100 train/dev/test splits per subtask, excluding near-perfect baseline tasks and two tasks with fewer than 50 examples.The exclusions reflect APE accuracy of 95%+ on eight tasks and insufficient examples for splitting two others.Source: §D.4 · p. 20
• BIG-bench Hard uses 100/100/50 splits for standard tasks and 100/100/500 samples for alternative Date Understanding and Movie Recommendation formats.The standard BBH repository provides 250 examples per task; the two alternative-format experiments use the original repository's larger collections.Source: §D.4 · p. 20
• Counterfactual evaluation covers arithmetic, chess, and syntax, with five random 100/20/100 train/dev/test splits for each subtask.The dataset details are listed in Table 15.Source: §D.4 · p. 20, Table 15 · p. 21
• The production task uses over 150 sampled human-annotated queries and labels derived from user-reported errors, split 50%/25%/25% for training, validation, and testing.Outputs are evaluated with F1-score, reporting the absolute change from the initialization prompt.Source: §D.4 · p. 21

E Additional Result Figures and Tables

• The appendix catalogs notable PE2 prompt edits and directs readers to task-level breakdowns for Instruction Induction, counterfactual evaluation, and BIG-bench Hard.The listed breakdowns use Figures 10–13 and Tables 17–20; Table 16 continues examples of notable edits.Source: E Additional Result Figures and Tables · p. 22, Table 16 · p. 22
• APO and PE2 perform similarly on most Instruction Induction tasks; the authors hypothesize that easier tasks lead to performance saturation.Figure 10 and Table 17 provide task-level results, while the table reports means and standard deviations across five runs.Source: Figure 10 · p. 22, Table 17 · p. 25

F Meta-prompts

• The meta-prompts are implemented with the guidance toolkit, which supports multi-round conversations and handlebars-style workflow control.The toolkit is identified in the implementation description.Source: F Meta-prompts · p. 28

F.1 Initialization Prompt pinit

• The shared induction initialization prompt infers an instruction from a batch of input-output examples; initialization is shared across Iterative APE, APO, and PE2.Its template asks what instruction produced the provided examples.Source: §F.1 · p. 28

F.2 APE

• APE proposes a variation of an instruction while preserving its semantic meaning and limiting its length.The prompt requests only the new instruction as output.Source: §F.2 · p. 28

F.3 APO

• APO first generates reasons a prompt failed on examples, then uses those reasons to refine the prompt.The two prompt stages request a gradient and an improved prompt, respectively.Source: §F.3 · pp. 28–29

F.4 PE2

• PE2 asks the model to inspect examples and the prompt-template context, assess correctness and task fit, then use that reasoning to produce an edited prompt.Its example-by-example template distinguishes output correctness, adherence to the existing prompt, and whether the prompt describes the task; refinement history and editing constraints may also be supplied.Source: §F.4 · pp. 29–30

Prompt examples

• PE2’s chess prompt specifies swapped knight-and-bishop starting positions and movements, and asks whether an opening is legal under the variant rules.It explicitly excludes strategic soundness from the judgment.Source: G Prompt Optimization Results · p. 34
• Prompt examples for subject-and-verb extraction specify different word orders and output conventions, including returning the main subject with the last verb in English order.The examples address OVS and OSV sentence patterns and vary in what counts as the target verb.Source: G Prompt Optimization Results · pp. 34–35
• For the causal task, APO asks for the direct cause, while PE2 asks whether the action was intentional and a contributing factor.The PE2 prompt specifies a yes-or-no response.Source: G Prompt Optimization Results · p. 36
• PE2’s examples replace generic stepwise cues with task-specific directions for ranking objects, checking math, comparing movies, and tracking movement.The movie prompt names genres, themes, narrative styles, tone, and characters as comparison dimensions.Source: G Prompt Optimization Results · p. 36