PaperlayerView exact revision on arXiv

Source-linked AI summary

PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization

Xinyuan Wang, Chenxi Li, Zhen Wang, Fan Bai, Haotian Luo, Jiayou Zhang, Nebojsa Jojic, Eric P. Xing, Zhiting Hu

arXiv:2310.16427v2cs.CL

TL;DR

Expert-level prompts require detailed instructions and domain insights that existing automated methods often miss while inefficiently exploring the prompt space. PromptAgent treats prompt optimization as strategic planning, using MCTS, error feedback, and self-reflection to refine prompts. On 12 tasks across BBH and domain-specific and general NLP, it outperforms strong CoT and prompt-optimization baselines, including APE across multiple base models.

  • Problem

    Existing prompt optimization methods struggle to automate expert-level prompts because they overlook deep domain knowledge and inefficiently explore the expansive prompt space.

  • Method

    PromptAgent formulates prompt optimization as strategic planning with MCTS, refining prompt states through model-error feedback and LLM self-reflection.

  • Results

    Across 12 tasks spanning BBH, domain-specific NLP, and general NLP, PromptAgent significantly outperforms CoT and recent prompt-optimization baselines, including APE.

  • Takeaways & Limitations

    PromptAgent autonomously produces expert-level prompts that integrate domain-specific knowledge while efficiently navigating complex prompt spaces.

  • Takeaways & Limitations

    The study focuses on a traditionally zero-shot setting in which task prompts exclude training samples.

Abstract

from arXiv · show

Highly effective, task-specific prompts are often heavily engineered by experts to integrate detailed instructions and domain insights based on a deep understanding of both instincts of large language models (LLMs) and the intricacies of the target task. However, automating the generation of such expert-level prompts remains elusive. Existing prompt optimization methods tend to overlook the depth of domain knowledge and struggle to efficiently explore the vast space of expert-level prompts. Addressing this, we present PromptAgent, an optimization method that autonomously crafts prompts equivalent in quality to those handcrafted by experts. At its core, PromptAgent views prompt optimization as a strategic planning problem and employs a principled planning algorithm, rooted in Monte Carlo tree search, to strategically navigate the expert-level prompt space. Inspired by human-like trial-and-error exploration, PromptAgent induces precise expert-level insights and in-depth instructions by reflecting on model errors and generating constructive error feedback. Such a novel framework allows the agent to iteratively examine intermediate prompts (states), refine them based on error feedbacks (actions), simulate future rewards, and search for high-reward paths leading to expert prompts. We apply PromptAgent to 12 tasks spanning three practical domains: BIG-Bench Hard (BBH), as well as domain-specific and general NLP tasks, showing it significantly outperforms strong Chain-of-Thought and recent prompt optimization baselines. Extensive analyses emphasize its capability to craft expert-level, detailed, and domain-insightful prompts with great efficiency and generalizability.

1 INTRODUCTION

PromptAgent addresses the difficulty of automatically producing expert-level prompts by treating prompt optimization as strategic planning. Across 12 tasks and multiple base models, it generates detailed, domain-informed prompts that outperform human, CoT, and APE baselines.

  • Expert-level prompts: Figure 1 illustrates that expert prompts contain richer biomedical details and structured guidance than ordinary human-written and sampling-based prompts.The example concerns disease-entity extraction in the NCBI biomedical task and shows the expert prompt leading to the correct prediction.
  • Challenges: Existing prompt optimization methods struggle to reach expert-level prompts because they use local edits and lack principled exploration of the prompt space.These methods often overlook the domain knowledge and nuanced instructions embedded in expert prompts.
  • Approach: PromptAgent reformulates prompt optimization as strategic planning with Monte Carlo Tree Search, using error feedback to refine prompts and prioritize high-reward paths.The framework combines trial-and-error exploration, model-error collection, LLM self-reflection, and iterative prompt refinement.
  • Results: Across 12 tasks spanning BBH, domain-specific NLP, and general NLP, PromptAgent enhances initial human prompts and surpasses strong CoT and prompt-optimization baselines.The evaluation covers three practical domains and emphasizes exploration efficiency and generalizability.
  • Results: PromptAgent consistently outperforms human and APE baselines across GPT-3.5, GPT-4, and PaLM 2, improving over APE by 9.1%, 7.7%, and 6%, respectively.Figure 2 compares prompt performance across the three base models.

2 RELATED WORKS

Prior work optimizes prompts through soft-prompt training, discrete-token search, sampling, or LLM augmentation with reasoning and planning. PromptAgent differs by combining self-reflection and Monte Carlo Tree Search specifically for prompt optimization.

  • Prompt optimization: Open-source LLMs support gradient- or reinforcement-learning-based optimization of soft and discrete prompts, but these approaches are less feasible for closed-source models.The limitation motivates gradient-free methods for API-based LLMs.
  • LLM augmentation: LLMs have been augmented with external reasoning, tools, and planning modules to address limitations including long-term coherence and the absence of an internal world model.These approaches add capabilities beyond the base model through external modules.
  • Planning with LLMs: Planning-based methods such as Tree of Thoughts, CoRe, and RAP enhance reasoning, but PromptAgent is presented as the first framework combining self-reflection and planning for prompt optimization.Tree of Thoughts uses DFS/BFS, whereas CoRe and RAP use MCTS to navigate reasoning paths.

3 METHODOLOGY

PromptAgent formulates prompt optimization as strategic planning over prompt states and error-based actions, using MCTS to search an expansive prompt space for high-reward revisions. Its framework combines model-error feedback, prompt transitions, reward evaluation, and four MCTS operations to balance exploration with exploitation.

  • PromptAgent Framework Design: Prompt optimization seeks a natural-language prompt that maximizes a base LLM’s performance on a target task, but the prompt space is infinite and intractable to enumerate.The formulation starts from an initial prompt and evaluates candidate prompts with a task-performance reward such as accuracy.
  • PromptAgent Framework Design: PromptAgent defines each prompt version as a state and uses expert-informed modifications as actions to steer revisions toward expert-level prompts.The framework seeks to integrate expert prior knowledge while exploring the expansive prompt space efficiently.
  • PromptAgent Framework Design: Error-based actions use base-model mistakes to generate feedback that guides the optimizer in revising prompts to avoid observed pitfalls.The optimizer collects errors from task samples, reflects on them, and produces feedback used for subsequent prompt refinement.
  • PromptAgent Framework Design: PromptAgent models optimization as an MDP in which an optimizer LLM generates error-feedback actions, transitions them into revised prompt states, and evaluates task-performance rewards.The MDP consists of state space S, action space A, transition function T, and reward function r; the transition uses a second meta-prompt to generate the updated prompt.
  • Strategic Planning for Prompt Optimization: MCTS navigates prompt space by balancing exploitation of high-value paths with exploration of less-visited nodes, then expands candidates, simulates future trajectories, and back-propagates rewards.Selection uses UCT; expansion samples diverse error feedbacks and retains the highest-reward new node, simulation continues expansion toward a terminal state, and back-propagation updates Q values from future trajectories.

4 EXPERIMENTS

PromptAgent is evaluated across 12 tasks and three domains against human, Chain-of-Thought, and prompt-optimization baselines. Results show stronger performance, transfer across base LLMs, efficient strategic search, and progressively refined expert prompts.

  • Experimental setup: PromptAgent is evaluated on 12 tasks spanning BIG-Bench Hard, domain-specific biomedical tasks, and general NLP tasks.The setup compares against human prompts, CoT prompts, and recent prompt optimization methods.
  • Comparison with prompting baselines: 28.9%, 9.5%, and 11.2% relative improvement over human zero-shot, CoT, and APE baselines, respectively, is reported on BBH tasks.PromptAgent outperforms baselines overall and wins five of six BBH tasks; CoT surpasses it only on Object Counting.
  • Comparison with prompting baselines: +7.3% average improvement over APE on domain-specific tasks and +16.9% and +9% over CoT and APE on general NLP tasks are reported.The results are presented as evidence that PromptAgent can induce domain knowledge and generalize across NLP applications.
  • Prompt generalization: GPT-4 improves with transferred PromptAgent prompts in 11/12 tasks, while PaLM 2 still exceeds both baselines in 7/12 tasks despite a dramatic performance drop.Prompts optimized on GPT-3.5 are directly applied to GPT-4 and PaLM 2.
  • Ablation on search strategies: MCTS yields a 5.6% relative overall improvement over the best alternative search baseline across the ablation tasks.Greedy and Beam improve over directionless Monte Carlo search, but lack MCTS’s strategic exploration, future-outcome evaluation, and backtracking.
  • Efficiency and convergence: PromptAgent’s explored prompts cluster at higher performance and lower search cost, while training and testing performance stabilize after depth 3 on the Epistemic task.The exploration analysis compares explored-node counts with task performance, and the convergence analysis tracks averages across tree depth.
  • Qualitative analysis: The NCBI qualitative trace shows PromptAgent using error feedback to transition from initial prompts toward expert prompts containing domain-specific insights and structured guidance.The analyzed prompt aspects include task description, term clarification, solution guidance, exception handling, priority, emphasis, and formatting.

5 CONCLUSION

PromptAgent autonomously crafts expert-level prompts by treating prompt optimization as strategic planning and using MCTS to traverse complex prompt spaces. Across 12 diverse tasks, its prompts incorporated domain-specific details and outperformed human-written, Chain-of-Thought, and other optimized-method baselines.

  • PromptAgent views prompt optimization as strategic planning and uses MCTS to efficiently traverse complex prompt spaces.It incorporates task-specific knowledge through trial-and-error and LLM self-reflection.
  • Across 12 diverse tasks spanning three domains, PromptAgent produced prompts enriched with domain-specific details and guidance.
  • PromptAgent significantly outperformed human-written, Chain-of-Thought, and other optimized-method baselines.The analyses also reported superior transferability, exploration efficiency, and prompt quality.

Inputs:

The planning procedure initializes a prompt state and search components, repeatedly selects or expands actions, simulates transitions, and back-propagates future rewards through the tree.

  • The algorithm initializes an initial prompt state, transition and reward functions, action generation, search depth, iterations, and exploration weight.
  • It represents state-to-action and child mappings, rewards, state-action values, and state visit counters.
  • Selection: When actions exist, selection transitions to a child state, obtains its reward, and increments the state visit counter.
  • Expansion and simulation: During expansion and simulation, the procedure generates multiple actions and transitions until an early-stopping state is reached.
  • Back-propagation: After a terminal state, it back-propagates future rewards along the path and updates each state-action value.

A.1 INPUT FORMULATION

The model input combines the optimized prompt with optional task context, the question, optional answer choices, and an optional answer-format specification.

  • The normal model input consists of a prompt, task prefix, question, task suffix, and answer format.
  • The prompt is the optimization target, while the optional task prefix supplies task-specific background information.
  • The question is the main body of the task, and the optional task suffix contains answer options such as yes/no or multiple-choice labels.
  • The optional answer format is designed to constrain or caption the model's response.

A.2 DATA SPLIT

The data-splitting procedure uses predefined test sets when available, caps oversized tests at 1,000 examples, and otherwise creates approximate test and training partitions before holding out reward data.

  • Predefined testing sets are used directly, except that sets exceeding 1,000 examples are sampled down to 1,000.
  • Without a default testing set, the data are shuffled and approximately half is allocated for testing.
  • A subset of the remaining training data is sampled for reward calculation, with a default size of 150.

A.3 MORE IMPLEMENTATION DETAILS

PromptAgent uses MCTS to search prompt states, evaluates rewards on held-out performance, and selects a high-reward prompt from the best path.

  • Experimental settings: The experiments use documented data splits and PromptAgent hyperparameter settings alongside the MCTS implementation.The supplied appendix references a data-split table and a hyperparameter-settings table.
  • PromptAgent-MCTS: PromptAgent performs MCTS planning in prompt space with terminal states defined by a depth limit and rewards based on held-out-set performance.An early-stopping rule can avoid unnecessary exploration after depth exceeds 2 when rewards fall below a minimum threshold or exceed a maximum threshold.
  • Initialization and expansion: The algorithm initializes its root node with an initial prompt, evaluates it, formats model errors, and uses error feedback to guide prompt expansion.For BBH, dataset task descriptions generally provide the initial prompts, with a custom format for Object Counting; other tasks use prompts crafted from objectives or answer formats.
  • Initialization and expansion: Each state transition supplies the expanding prompt, selected-path trajectory, and summarized errors to an optimizer that generates new prompt samples.This transition representation combines the current prompt context with accumulated search history and error information.
  • Output strategy: The output strategy selects the path with the highest average reward and then chooses the highest-reward prompt on that path as the final output.This prioritizes the strongest overall trajectory while allowing the best prompt to occur before the path’s final node.

A.4 BASELINES IMPLEMENTATION DETAILS

The appendix specifies comparison procedures for sampling, beam, greedy, and APE baselines, while documenting PromptAgent’s meta-prompt formats.

  • Monte Carlo: Monte Carlo samples multiple one-step prompt alternatives and selects the best, using PromptAgent’s sampling method with search depth limited to one.The search ablation samples 72 new prompts per task.
  • Beam Search: Beam Search expands nodes into three children, retains the best three nodes at each depth, and generates 72 prompts across a depth-eight search.The root expands into nine prompts, while later nodes expand into three each and the beam width remains three.
  • Greedy Search: Greedy Search uses beam width one, with Greedy-S and Greedy-L differing in expansion width and generating 34 and 72 prompts, respectively.Both experiments use search depth eight; their expansion widths are three and nine.
  • APE: APE uses one iterative optimization round, initially generating 100 candidate prompts and then evaluating 50 candidates in the iterative proposal step.Med QA uses only 25 initial candidates because of longer processing time.
  • Meta formats: PromptAgent’s meta-formats define the base-model input, error examples, error feedback, and state-transition prompts containing error information and trajectory prompts.These formats separate task inputs from optimizer guidance and the prompt sequence used during search.

B TASK INPUT EXAMPLES

The appendix illustrates task inputs across multi-choice selection, named-entity recognition, and direct-answer matching using three representative tasks.

  • Task categories: The examples cover three task-input categories: multi-choice selection, named entity recognition, and direct answer matching.Penguins in A Table, NCBI, and Subjective serve as representative examples.
  • Representative formats: Figure 6 presents the input format for the Penguins in A Table task.
  • Representative formats: Figure 7 presents input formats for the NCBI and Subjective tasks.

C CONVERGENCE OBSERVATION DETAILS

The appendix reports convergence-plot settings and compares PromptAgent’s optimized prompts with human-written and APE prompts across several tasks.

  • Convergence settings: The Wide convergence setting uses expand width = 3, num samples = 2, and depth limit = 6; plots show average-path reward, variance, and best-path reward.The best node is the highest-reward node on the best-reward path.
  • Convergence plots: The convergence figures define average reward as the mean reward of paths and represent its variance with a blue area.
  • Convergence settings: The Lite convergence setting uses expand width = 3, num samples = 1, and depth limit = 4.
  • Convergence settings: The Standard convergence setting uses expand width = 3, num samples = 1, and depth limit = 8.Its plots use the same average-reward-path, variance, and best-reward-path definitions as the other settings.
  • Prompt comparisons: PromptAgent’s optimized prompts are compared with ordinary human-written and APE prompts across Geometric Shapes, Penguins in A Table, Epistemic Reasoning, Object Counting, and Temporal Sequences.The comparisons emphasize differences in prompt structure and domain-specific content.
  • Prompt comparisons: The Geometric Shapes examples contrast APE’s concise instruction with PromptAgent’s detailed treatment of SVG path commands and shape reconstruction.PromptAgent’s instructions address M, L, and A commands, multiple sub-paths, side counting, and final recounting.
  • Prompt comparisons: Additional prompt comparisons cover Causal Judgment, NCBI, Biosses, and Med QA.The supplied table descriptions state that PromptAgent prompts contain more complex structures and domain-specific insights than both baselines.
  • Prompt comparisons: The Causal Judgment examples contrast APE’s binary intent classification with PromptAgent’s analysis of highlighted entities, multifactorial causes, and immediate versus incidental origins.
Loading 2310.16427v2…