PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Automatic Prompt Engineering: An Optimization Perspective

Wenwu Li, Xiangfeng Wang, Wenhao Li, Bo Jin

arXiv:2502.11560v1cs.AIcs.LG

TL;DR

Existing surveys do not comprehensively organize automated prompt engineering across modalities and prompt spaces. This paper develops a unified optimization-theoretic survey covering variables, objectives, and methods, concluding that multi-level, multi-objective, and online optimization are important directions for emerging foundation models.

  • Problem

    Existing surveys remain fragmented across modalities and do not unify prompt components, objectives, and optimization strategies across discrete, continuous, and hybrid spaces.

  • Method

    The survey formalizes prompt optimization as performance maximization over discrete, continuous, and hybrid prompt spaces and classifies variables, objectives, and optimization methods.

  • Results

    The survey establishes a unified optimization-theoretic foundation spanning instructions, soft prompts, exemplars, task objectives, and FM-based, evolutionary, gradient-based, and reinforcement-learning methods.

  • Takeaways & Limitations

    The survey identifies tighter integration of multi-level, multi-objective, and online optimization as pivotal for prompt design in emerging foundation models.

  • Takeaways & Limitations

    Current prompt optimization lacks robust frameworks for inter-task similarity and trade-offs such as negative transfer, while multi-turn agents and domain-specific models impose additional constraints.

Abstract

from arXiv · show

The rise of foundation models has shifted focus from resource-intensive fine-tuning to prompt engineering, a paradigm that steers model behavior through input design rather than weight updates. While manual prompt engineering faces limitations in scalability, adaptability, and cross-modal alignment, automated methods, spanning foundation model (FM) based optimization, evolutionary methods, gradient-based optimization, and reinforcement learning, offer promising solutions. Existing surveys, however, remain fragmented across modalities and methodologies. This paper presents the first comprehensive survey on automated prompt engineering through a unified optimization-theoretic lens. We formalize prompt optimization as a maximization problem over discrete, continuous, and hybrid prompt spaces, systematically organizing methods by their optimization variables (instructions, soft prompts, exemplars), task-specific objectives, and computational frameworks. By bridging theoretical formulation with practical implementations across text, vision, and multimodal domains, this survey establishes a foundational framework for both researchers and practitioners, while highlighting underexplored frontiers in constrained optimization and agent-oriented prompt design.

1 Introduction

Foundation models make conventional fine-tuning increasingly costly, while manual prompt design remains laborious, sensitive to phrasing, static, and difficult to align across modalities. This survey addresses these limitations by framing automated prompt engineering as a unified optimization problem spanning prompt spaces, objectives, and computational methods.

  • Foundation models’ scale makes conventional fine-tuning computationally prohibitive, especially for edge devices and time-sensitive applications.
  • Prompt engineering steers foundation-model behavior through strategic input design rather than neural-weight modification, reducing dependence on retraining and backpropagation.
  • Manual prompting requires expert trial-and-error, is sensitive to minor input changes, and cannot readily adapt to evolving inputs.
  • Automated approaches explore combinatorial prompt spaces through foundation-model optimization, evolutionary strategies, gradient-based methods, and reinforcement learning.
  • Existing surveys are fragmented across methodological and modal boundaries, leaving no comprehensive treatment of automated prompt engineering across modalities.
  • The survey formalizes automated prompt engineering as maximizing performance over discrete, continuous, and hybrid prompt spaces and organizes methods by variables, objectives, and computational frameworks.

2 Related Work

Prior surveys examine prompt learning, compression, multimodal applications, or efficiency in isolation. This survey unifies prompt components, objectives, and optimization strategies across discrete, continuous, and hybrid spaces within one optimization-theoretic framework.

  • Earlier surveys address foundational prompt learning, prompt compression, multimodal prompting, manual design, or efficiency-focused aspects separately.
  • Existing work remains fragmented by methodology or modality and does not comprehensively unify prompt optimization across discrete, continuous, and hybrid spaces.
  • This survey formulates prompt engineering as an overarching optimization problem maximizing task-specific performance for discrete, continuous, and mixed prompts.
  • Its taxonomy jointly organizes optimization variables, objective functions, and computational methods across modalities.
  • The resulting framework synthesizes diverse studies into a single cohesive foundation for advancing automated prompt engineering.

3 Optimization Problem Formulation

The paper formalizes prompt optimization for LLMs and VLMs over discrete, continuous, and hybrid prompt spaces. It specifies prompt forms, optimization variables, and a performance-maximization problem over a validation set.

  • Prompt spaces: Prompt functions map inputs to conditioning patterns, and the prompt space is partitioned into discrete, continuous, and hybrid subspaces.The hybrid space combines discrete and continuous components.
  • Discrete prompt forms: Discrete prompts include instructions, thought sequences, few-shot exemplars, and, for VLMs, spatial annotations such as regions with area specifications and labels.Region specifications may include bounding boxes, markers, pixel masks, polygons, or curves.
  • Continuous prompt forms: Continuous prompts concatenate input embeddings with m learnable vectors in a d-dimensional embedding space.The embedding representation is produced by an embedding function, while the vectors are trainable prompt parameters.
  • Hybrid prompt forms: Hybrid prompts jointly optimize discrete regions and continuous embeddings.This formulation combines human-readable prompt components with trainable vectors.
  • Optimization objective: Given a black-box foundation model and validation set, prompt optimization maximizes expected performance under discrete, continuous, or hybrid prompt-space constraints.The performance metric compares model predictions with ground truth, yielding three corresponding optimization subclasses.

4 Optimization Spaces

The survey organizes prompt optimization into discrete, continuous, and hybrid spaces, covering instructions, reasoning thoughts, exemplars, annotations, and trainable embeddings across modalities.

  • Prompt optimization is analyzed across discrete, continuous, and hybrid spaces, enabling a principled taxonomy across modalities.
  • Discrete Variables: Discrete prompts include instructions, thoughts, few-shot examples, and visual annotations with distinct optimization roles.Instructions express objectives, thoughts structure reasoning, exemplars demonstrate behaviors, and annotations provide visual cues.
  • Discrete Variables: Exemplar optimization selects, orders, or generates input-output examples to improve task performance.
  • Discrete Variables: Visual annotations such as bounding boxes, markers, and pixel masks remain exceptionally underexplored in prompt optimization.
  • Continuous Variables: Continuous prompts use learnable embeddings optimized by gradients, avoiding model-parameter updates and requiring fewer adaptation resources.Soft prompts can be appended to input representations or applied as prefixes and layer-spanning embeddings.
  • Hybrid Variables: Hybrid prompts combine human-readable instructions or exemplars with trainable embeddings to balance interpretability, domain specificity, and flexibility.Jointly refining these complementary components is reported to produce more robust and adaptable behavior.

5 Objective Functions

The survey frames downstream evaluation as maximizing expected performance metrics for black-box foundation models, then extends this objective with task-specific measures and constraints.

  • The core objective maximizes expected performance g(f(P(x)), y) over prompts for a black-box foundation model.The formulation covers discrete, continuous, and hybrid prompt spaces.
  • Downstream Tasks: Instruction induction uses BERTScore-F1 and exact/set match to assess whether models extract and generalize underlying instructions.
  • Downstream Tasks: Text classification commonly evaluates sentiment, topic, and subjectivity tasks with classification accuracy and Macro-F1.
  • Downstream Tasks: Reasoning objectives evaluate numeric outputs, perplexity, exact match, or specialized intermediate-step and chain-of-thought correctness.These measures cover arithmetic, commonsense, and multi-hop reasoning tasks.
  • Downstream Tasks: Multimodal objectives include image-text alignment, retrieval accuracy, CLIP, and Aesthetics Score across captioning, retrieval, and visual question answering.
  • Constrained Objectives: Constrained objectives add structural, semantic, or length constraints to support prompt editing and compression beyond raw performance maximization.Prompt editing restricts reformulations, while compression imposes a token-length or embedding-size budget.

6 Optimization Methods

The survey classifies automated prompt optimization into FM-based, evolutionary, gradient-based, and reinforcement-learning paradigms, further distinguishing discrete, continuous, and hybrid prompt variables.

  • Four major paradigms organize the literature: FM-based optimization, evolutionary computing, gradient-based optimization, and reinforcement learning.
  • The taxonomy additionally distinguishes methods optimizing purely discrete, purely continuous, or hybrid prompts.Some approaches combine paradigms, such as FM-generated mutations with genetic-algorithm operators.
  • FM-based Optimization: FM-based methods iteratively use a foundation model to propose revised prompts from performance feedback, meta-prompts, self-reflection, or search.
  • Evolutionary Computing: Evolutionary methods mutate or cross over discrete prompts and retain offspring according to task-performance fitness.FM-assisted approaches can act as evolutionary operators, while some methods co-evolve task prompts and mutation prompts.
  • Gradient-Based Optimization: Gradient-based methods approximate gradients for discrete tokens or optimize continuous soft-prompt parameters because discrete tokens are not directly differentiable.
  • Gradient-Based Optimization: Soft-prompt tuning optimizes trainable vectors concatenated with inputs or inserted as prefixes and hidden-state components.
  • Reinforcement Learning: Reinforcement learning treats prompt edits as actions in a Markov decision process, with task performance serving as a reward or learned proxy.The framework supports discrete, continuous, and hybrid prompt adaptation.
  • Reinforcement Learning: RL extensions address adaptive test-time editing, query-specific rewards, and conflicting objectives such as style versus accuracy.

7 Future Directions

Future work must extend automated prompt optimization beyond offline, single-task, single-objective, and predominantly textual settings. Key challenges include constraints, nonstationarity, competing goals, heterogeneous modalities, agent interactions, and hierarchical reasoning.

  • Constraint optimization: Constraint-aware optimization must encode human-value alignment, resource bounds, and readability while preserving search flexibility and linguistic quality.The challenge is especially pronounced for discrete prompts in high-dimensional symbolic spaces.
  • Multi-task prompt optimization: Multi-task optimization lacks formal definitions of inter-task prompt similarity and must address negative transfer between tasks.Improvements on one task can degrade performance on another, motivating frameworks that codify these trade-offs.
  • Online prompt optimization: Online optimization must handle shifting user intentions, stable performance in nonstationary environments, discrete-search complexity, and uncertain real-time feedback.Current techniques generally prioritize offline scenarios, while online updates demand stronger convergence analyses.
  • Multi-objective prompt optimization: Multi-objective optimization must balance competing goals such as accuracy and interpretability rather than relying primarily on single-metric objectives.Promising directions include Pareto-based methods, multi-criteria decision-making, preference-space geometry, and game-theoretic arbitration.
  • Heterogeneous modality optimization: Prompt optimization remains comparatively underexplored for visual cues such as bounding boxes and pixel-level annotations.Future methods must clarify cross-modal coupling and when modalities should be optimized jointly or separately.
  • Bi-level prompt optimization: Bi-level prompt optimization remains open for thought-driven models because small prompt changes can drastically alter reasoning trajectories.Open questions include the existence of stable equilibria and sensitivity to prompt perturbations.
  • Broader application scenarios: Multi-turn agents and vertical-domain models introduce sequential, nonstationary, collaborative, competitive, and domain-specific constraints.Agent-oriented settings may require equilibrium concepts, while vertical domains require specialized constraints beyond standard prompt optimization.

8 Discussion and Conclusion

The survey establishes a unified optimization-theoretic foundation for automated prompt engineering across modalities and prompt spaces. It synthesizes variables, objectives, and optimization methods while identifying multi-level, multi-objective, and online optimization as important directions.

  • Discussion and Conclusion: The survey unifies automated prompt engineering across discrete, continuous, and hybrid prompt spaces, including instructions, soft prompts, and exemplars.It frames these variables as systematically optimizable under shared theoretical principles.
  • Discussion and Conclusion: Its taxonomy covers task objectives and FM-based, evolutionary, gradient-based, and reinforcement-learning optimization methods.The framework connects theoretical inquiry with realistic application across modalities.
  • Discussion and Conclusion: Future progress depends on tighter integration of multi-level, multi-objective, and online optimization for emerging foundation models.The conclusion identifies these integrations as pivotal for future prompt design.
Loading 2502.11560v1…