PaperlayerView exact revision on arXiv

Source-linked AI summary

promptolution: A Unified, Modular Framework for Prompt Optimization

Tom Zehle, Timo Heiß, Moritz Schlager, Matthias Aßenmacher, Matthias Feurer

arXiv:2512.02840v2cs.CL

TL;DR

Prompt quality strongly shapes LLM task performance, but prompt optimizers are scattered across difficult-to-maintain research repositories. promptolution brings multiple discrete optimizers into a modular, LLM-agnostic framework with reproducible benchmarking; most evaluated optimizers outperformed unoptimized prompts, with CAPO strongest on both datasets.

  • Problem

    Prompt quality and example choice affect LLM task performance, while optimizers are scattered across research repositories that are often poorly maintained and difficult to compare or adapt.

    Source: §1 · pp. 1–2

  • Method

    promptolution combines contemporary discrete optimizers in a modular, LLM-agnostic framework with interchangeable components, reproducible experiments, and prompt-string outputs for integration into existing applications.

    Source: §1 · p. 2, §2 · pp. 2–3, §3 · p. 4

  • Results

    With exceptions GEPA on SST5 and OPRO on GSM8K, optimizers substantially outperformed unoptimized prompts, with accuracy gains up to 15%p for CAPO on GSM8K; CAPO led both datasets.

    Source: §4 · p. 6

  • Takeaways & Limitations

    The library provides at least one strong optimizer per task, lets users switch when performance disappoints, and supports optimizer comparisons with minimal manual effort.

    Source: §4 · p. 7

    promptolution optimizes prompts rather than whole application pipelines; DSPy better suits end-to-end development, but couples system design to its abstractions and reduces portability.

    Source: §5 · p. 7

Abstract

from arXiv · show

Prompt optimization has become crucial for enhancing the performance of large language models (LLMs) across a broad range of tasks. Although many research papers demonstrate its effectiveness, practical adoption is hindered because existing implementations are often tied to unmaintained, isolated research codebases or require invasive integration into application frameworks. To address this, we introduce promptolution, a unified, modular open-source framework that provides all components required for prompt optimization within a single extensible system for both practitioners and researchers. It integrates multiple contemporary discrete prompt optimizers, supports systematic and reproducible benchmarking, and returns framework-agnostic prompt strings, enabling seamless integration into existing LLM pipelines while remaining agnostic to the underlying model implementation.

1 Introduction

Prompt performance is sensitive to prompt formulation and example choice, making optimization useful, while manual prompt engineering is tedious and unreliable. promptolution addresses practical adoption barriers with a modular, thoroughly tested framework for prompt optimization.

1 Introduction

  • Prompt performance depends on formulation and example choice and order; semantically similar GSM8K prompts can perform substantially differently.Table 1 presents example prompts and their test accuracy on GSM8K with Llama-3.3-70B.Source: §1 · p. 1, Table 1 · p. 2
  • Manual prompt engineering is tedious and unreliable, while automatic prompt optimization offers a lightweight alternative to expensive, data-intensive fine-tuning.Source: §1 · p. 2
  • promptolution combines tested, interchangeable optimization components with an iterative workflow that evaluates prompts on development data and returns optimized prompts for test evaluation.The framework also provides logging and callbacks to track the process.Source: Figure 1 · p. 1, §1 · p. 2

2 Background & Related Works

The section situates promptolution among discrete prompt optimizers and existing tools, highlighting its modular, lower-level, portable role for practice and reproducible research.

2 Background & Related Works

  • Automatic prompt optimization systematically explores prompt spaces; promptolution focuses on LLM-agnostic discrete methods that iteratively refine textual prompts, often using a meta-LLM.Source: §2 · p. 2
  • OPRO (Yang et al., 2024) uses a meta-LLM to refine scored candidates, while EvoPrompt and CAPO (Zehle et al., 2025) use evolutionary approaches.CAPO also applies AutoML techniques for cost-efficiency and jointly optimizes instructions and few-shot examples.Source: §2 · pp. 2–3
  • Siloed, often poorly maintained optimizer repositories make practical use and scientific comparison difficult; promptolution offers interchangeable optimizers for arbitrary LLMs.Source: §2 · p. 3
  • Unlike DSPy’s high-level pipeline compilation, promptolution exposes iterative optimization for finer control and returns prompt strings that can replace prompts in arbitrary applications.DSPy integration into other implementations requires larger refactoring.Source: §2 · p. 3
  • CoolPrompt emphasizes zero-configuration, while PromptBench targets evaluation and OpenPrompt predates modern LLMs; promptolution instead gives researchers close control over setup.Source: §2 · p. 3
  • promptolution unifies multiple contemporary discrete optimizers in a customizable, extensible framework for single-prompt use and systematic, reproducible benchmark studies.Source: §2 · p. 4

3 System Design

promptolution organizes prompt optimization around interchangeable components with shared Base-class interfaces and an ExperimentConfig for configuring them together. Its components cover LLM access, prediction, tasks, optimization, experiment execution, and supporting utilities.

3 System Design

  • Shared Base-class interfaces make four components interchangeable, while ExperimentConfig and helpers configure them together.Figure 2 depicts component implementations, functions, and conceptual connections.Source: §3 · p. 4, Figure 2 · p. 4

3.1 LLM

  • LLM interfaces support API-hosted models, local transformers models, and high-throughput vLLM inference; the base class enables parallelization and monitors token use.The API examples include OpenAI and Anthropic.Source: §3.1 · p. 4

3.2 Predictor

  • Predictors extract predictions from LLM outputs: FirstOccurrencePredictor finds the first class label, while MarkerBasedPredictor extracts text between predefined markers.The paper illustrates marker-based extraction with a GSM8K response.Source: §3.2 · p. 4

3.3 Task

  • Tasks hold datasets, descriptions, evaluation procedures, and subsampling strategies, with implementations for classification, LLM-as-a-judge scoring, and custom reward functions.Subsampling can use random samples, a dataset slice, or the full dataset.Source: §3.3 · pp. 4–5

3.4 Optimizer

  • Optimizers iteratively evaluate predictions and refine prompts; implementations include OPRO (Yang et al., 2024), EvoPromptDE/GA, and CAPO (Zehle et al., 2025).Prompt caching and evaluation on a data subset reduce optimization cost; researchers can add optimizers by implementing a custom _step() method.Source: §3.4 · p. 5

3.5 Experiment Configuration & Helper

  • ExperimentConfig provides structured configuration for larger benchmarks, and helpers run optimization and evaluate prompts on unseen test data.run_experiment combines optimization and evaluation across datasets and optimizers; unspecified arguments use defaults, while unused specified arguments trigger warnings.Source: §3.5 · p. 5

3.6 Supporting Modules and Utilities

  • Utilities support post-hoc random or random-search exemplar selection, initial-prompt creation from task descriptions, base prompts, or data, and callbacks for tracking, stopping, token limits, and file output.Callbacks can access optimizer state at each step; file output records prompts and scores for later analysis.Source: §3.6 · pp. 5–6

4 Evaluation

The evaluation compares promptolution optimizers, other prompt-optimization frameworks, and unoptimized prompts on GSM8K and SST-5 using Gemma3-27B-it. Most optimizers outperform the unoptimized baseline, and promptolution includes the best-performing optimizer on both datasets.

Setup

  • The comparison evaluates optimized prompts on an unseen test set after selecting each optimizer’s best prompts by development-set performance.The setup uses GSM8K and SST-5 with Gemma3-27B-it, default optimizer parameters, and a combined in- and output-token budget of at most one million.Source: §4 · p. 6

Results

  • Except GEPA on SST5 and OPRO on GSM8K, all optimizers substantially outperform the unoptimized baseline; CAPO on GSM8K improves accuracy by up to 15%p.CAPO is the best-performing optimizer on both datasets, and each runner-up is implemented in promptolution.Source: Table 4 · p. 6, §4 · p. 6
  • The library may not have the top optimizer on every task, but users can switch to an alternative and compare optimizers with minimal manual effort.The authors attribute the low-effort comparisons to dedicated support for systematic benchmark experiments.Source: §4 · pp. 6–7

5 Use Cases & Anti-Use Cases

The paper frames framework choice around performance, integration constraints, optimization scope, and tolerance for lock-in. It distinguishes use cases for existing pipelines, end-to-end application development, optimizer benchmarking, and LLM benchmarking.

Case I: Integration in Existing Pipelines

  • For an existing pipeline, promptolution returns a plain-text optimized prompt that can be substituted without refactoring application logic, retaining flexibility across providers and deployment modes.The text contrasts this with framework-specific prompts that require rewriting pipelines into structures such as DSPy’s.Source: §5 · p. 7

Case II: End-to-End LLM Application Development

  • For end-to-end application development, promptolution only optimizes prompts, while DSPy supports joint pipeline composition and optimization at the cost of abstraction coupling and reduced portability.The paper presents DSPy as better suited to building a system from scratch when prompt logic, program structure, and possibly training stages are designed together.Source: §5 · p. 7

Case III: Prompt Optimizer Benchmarking

  • For optimizer research, promptolution’s low abstraction and modular design support comparative evaluation, reproducible experiments, and extensions to optimizers, tasks, or evaluation strategies.Callbacks and logging expose full trajectories, intermediate prompts, and token usage for detailed analysis.Source: §5 · p. 7

Case IV: Integration in LLM Benchmarking

  • For LLM benchmarking, promptolution integrates prompt optimization into benchmark pipelines, supports exchanges of backends and datasets, and targets controlled, reproducible studies across multiple seeds.The stated motivation is to account for prompts’ strong influence on performance and reduce variance from arbitrary prompt choices.Source: §5 · p. 7

6 Conclusion & Future Directions

The paper concludes that promptolution provides a modular framework for practitioners and researchers and reports utility and competitiveness in comparative evaluation. Future work includes higher-level interfaces, experiment-management and visualization tools, multi-agent prompt optimization, and optimizer ensembling.

6 Conclusion & Future Directions

  • Comparative evaluation verified promptolution’s utility and competitiveness as a modular framework for practitioners and researchers.The paper also describes concrete use and anti-use cases for choosing a prompt optimization framework.Source: §6 · p. 7
  • Planned interfaces with higher-level frameworks and configuration-management tools aim to simplify combining framework strengths and managing complex experiments.The paper names DSPy and Hydra as examples.Source: §6 · p. 7
  • Planned additions include real-time experiment tracking and visual analysis, plus optimization of system prompts and interaction protocols across agents.The paper connects the multi-agent direction to the recent rise of multi-agent systems.Source: §6 · pp. 7–8
  • The authors plan to explore prompt-optimizer ensembling and continue incorporating new state-of-the-art optimization methods.The proposed ensembling direction is inspired by AutoML and likened to ELPO.Source: §6 · p. 8

Broader Impact

Promptolution unifies prompt optimizers that had been scattered across research repositories, with potential benefits for practitioners and researchers. Its experiment-friendly design may also support faster progress and methodological comparability in prompt optimization.

Broader Impact

  • Unifying methods scattered across research repositories makes prompt optimization more accessible to practitioners in real-world industry applications.The paper links this access to practitioners leveraging prompt optimization to improve LLM performance.Source: Broader Impact · p. 8
  • Modular, extensible, experiment-friendly implementations make promptolution useful for researchers benchmarking new prompt optimization algorithms.The passage identifies reimplementing competing methods and setting up rigorous benchmarks as time-consuming.Source: Broader Impact · p. 8
  • The framework has the potential to accelerate prompt-optimization research and enhance methodological comparability across the field.Source: Broader Impact · p. 8

A.1 Details on Prompt Optimization Tool Categorization

The categorization contrasts open-source prompt-optimization tools by optimizer variety, extensibility, abstraction, and integration. The supplied descriptions show different trade-offs across these dimensions.

A.1 Details on Prompt Optimization Tool Categorization

  • DSPy supports multiple extensible optimizers but uses a high-level declarative interface and requires integration through its modules and signatures.CoolPrompt also offers multiple optimizers and non-invasive integration, but is only partly extensible and emphasizes zero configuration.Source: §A.1 · p. 12
  • Promptomatix uses DSPy optimizers through a fixed, non-extensible class, while prompt-ops offers basic and advanced optimizers with its advanced option intended exclusively for Llama models.Promptomatix is described as high-abstraction and invasive; prompt-ops is low-abstraction, configurable through one file, and returns a prompt string.Source: §A.1 · p. 12
  • TextGrad and AdalFlow each provide a single optimizer and are described as non-extensible and invasive to integrate.TextGrad requires wrapped prompts and LLM calls; AdalFlow requires pipelines to be rearchitected into PyTorch-like component classes.Source: §A.1 · p. 12
  • PromptWizard and PromptIM each provide a single optimizer, are not extensible, and are characterized as low-abstraction and non-invasive.PromptIM can optionally commit an optimized prompt to LangChain Hub; PromptWizard is described as fine-grainedly parametrizable.Source: §A.1 · p. 12

A.2 Delimitation from DSPy

Promptolution focuses on prompt optimization rather than end-to-end LLM application construction, and differs from DSPy in workflow control, target users, and portability. Its standalone prompt strings and experiment-oriented extensibility are central distinctions.

A.2 Delimitation from DSPy

  • Promptolution focuses exclusively on prompt optimization, whereas DSPy treats optimization as one component of a broader compilation workflow tied to program structure.Source: §A.2 · p. 12
  • Promptolution’s iterative optimizer-specific steps and callbacks expose intermediate candidates, scores, and token use, supporting early stopping, custom logging, and budget-aware termination.Source: §A.2 · p. 12
  • Promptolution targets researchers and advanced ML practitioners, while DSPy primarily targets developers building end-to-end workflows such as agents or RAG systems.Source: §A.2 · p. 12
  • Promptolution supports custom components and large-scale benchmarking; its standalone prompt strings can be reused across systems with minimal friction.DSPy’s program abstraction can create lock-in when subtasks move to another system, potentially requiring adaptation.Source: §A.2 · p. 13

A.3 Experiment Details

The appendix documents the experimental setup and software practices, then shows how promptolution can be extended and used in an email-generation workflow.

A.3 Experiment Details

  • Token limits were enforced for DSPy and AdalFlow by raising an exception when exceeded and returning the last suggested prompt.The dataset overview and zero-shot prompts are presented in Tables 5 and 6.Source: §A.3 · p. 13, Table 5 · p. 13, Table 6 · p. 14
  • Evaluation uses a shared LangChain interface, exact-match label comparisons, a development set for optimization, and a holdout test set for final prompts.GEPA outputs required manual tag cleanup, while AdalFlow API calls were intercepted because system- and user-prompt extraction was faulty.Source: §A.3 · p. 13
  • The experiment code and raw results are publicly available for reproducibility.The LLM was hosted on local servers and accessed through API calls.Source: §A.3 · p. 13

A.4 Quality Standards

  • The package uses tests, reviewed pull requests, pre-commit checks, documentation standards, and CI/CD pipelines for quality and releases.All tests must pass before a release is published.Source: §A.4 · p. 13

A.5 Documentation & Tutorials

  • Documentation covers major components and includes tutorials for a first prompt-optimization use case.The paper presents this as supporting framework accessibility and usability.Source: §A.5 · p. 14

A.6 Extensibility of promptolution

  • New optimizers implement two hooks, while the base class supplies the optimization loop and shared infrastructure such as logging, callbacks, and budget-aware termination.The same extension pattern applies to task, LLM, and predictor components.Source: §A.6 · p. 14

A.7 System Demonstration

  • The email-generation demonstration derives candidate prompts from a task description, uses an LLM judge for subjective outputs, and iteratively refines prompts with CAPO for six steps.The example task description specifies persona, tone, and formatting constraints; the final optimized prompts are returned for email generation.Source: §A.7 · pp. 14–15

Related papers

Loading 2512.02840v2…