PaperlayerView exact revision on arXiv

Source-linked AI summary

$p1$: Better Prompt Optimization with Fewer Prompts

Zhaolin Gao, Yu, Wang, Bo Liu, Thorsten Joachims, Kianté Brantley, Wen Sun

arXiv:2604.08801v1cs.LGcs.CL

TL;DR

Prompt optimization works inconsistently because response randomness can obscure differences between system prompts, especially as heterogeneous datasets grow. The paper introduces p1, which filters for user prompts with high cross-system-prompt reward variance, substantially improving optimization on reasoning benchmarks and producing a broadly generalizing prompt from two AIME 24 examples.

  • Problem

    Prompt optimization varies widely across tasks, motivating study of when system-prompt quality differences are sufficiently distinguishable for optimization.

  • Method

    p1 selects a small subset of user prompts with high reward variance across candidate system prompts to strengthen the optimization signal.

  • Results

    Filtering substantially improves prompt optimization on reasoning benchmarks and outperforms training on the full dataset and strong baselines such as GEPA.

  • Takeaways & Limitations

    Training on only two AIME 24 prompts can produce a system prompt that generalizes well to other reasoning benchmarks.

  • Takeaways & Limitations

    The analysis assumes binary rewards, and the relationship between selected-subset performance and full-distribution performance remains incompletely understood.

Abstract

from arXiv · show

Prompt optimization improves language models without updating their weights by searching for a better system prompt, but its effectiveness varies widely across tasks. We study what makes a task amenable to prompt optimization. We show that the reward variance across different system prompts can be decomposed into two components: variance among responses, which captures generation stochasticity, and variance among system prompts, which captures differences in system prompt quality. Prompt optimization succeeds when variance among system prompts is sufficiently large, but fails when variance among responses dominates the variance of the system prompts. Surprisingly, we further show that scaling to more user prompts can hurt optimization by reducing variance among system prompts, especially on heterogeneous datasets where different user prompts favor different system prompts. Motivated by this insight, we propose $p1$, a simple user prompt filtering method that selects a small subset of user prompts with high variance across candidate system prompts. This subset of user prompts allows one to distinguish a good system prompt from a bad one, making system optimization easier. Experiments on reasoning benchmarks show that $p1$ substantially improves prompt optimization over training on the full dataset and outperforms strong baselines such as GEPA. Notably, training on only two prompts from AIME 24 yields a system prompt that generalizes well to other reasoning benchmarks.

1 Cornell University, 2 Microsoft, 3 Harvard University, 4 Databricks AI Research · 1 Introduction

The introduction frames system prompts as a weight-free interface for improving LLM behavior and studies why automatic prompt optimization succeeds inconsistently. It proposes p1, which filters user prompts by reward variance to strengthen optimization signals while using fewer examples.

  • 1 Introduction: Prompt optimization can substantially improve task performance without modifying model weights by shaping reasoning style, formatting, and instruction adherence.This motivates prompt optimization as an alternative to full model training.
  • 1 Introduction: The study focuses on reinforcement-learning prompt optimization, where a prompt-generation policy proposes candidate system prompts and a frozen LLM’s accuracy supplies the reward.The introduction also situates this setting among evolutionary and reinforcement-learning approaches to automatic prompt optimization.
  • 1 Introduction: Performance is strikingly inconsistent across tasks: optimized system prompts can yield clear gains on some tasks but barely improve others despite substantial compute.The paper investigates mechanisms governing this variability.
  • 1 Introduction: Reward variance across system prompts decomposes into variance among responses and variance among system prompts, separating generation stochasticity from differences in prompt quality.This decomposition provides the paper’s mechanism for analyzing when optimization has a useful signal.
  • 1 Introduction: Increasing dataset size can reduce variance among system prompts because different user prompts favor different prompts, causing preferences to cancel and candidate prompts to appear statistically identical.The dilution is especially severe on heterogeneous datasets.
  • 1 Introduction: p1 selects a small subset of user prompts with high reward variance across candidate system prompts, improving optimization by emphasizing prompts that distinguish strong from weak prompts.The method uses fewer training examples than optimizing over the full training set.

2 Problem Setup

Prompt optimization searches for a system prompt that maximizes expected correctness over a dataset, while modeling system-prompt generation as reinforcement learning. The formulation fixes the language-model response policy and trains a system-prompt policy using an on-policy GRPO variant.

  • Objective: Prompt optimization finds a system prompt x′ maximizing expected binary reward across dataset D, where correctness is evaluated from sampled responses y conditioned on x′ and user prompt x.The reward is r(x, y) ∈ {0, 1}, with 1 indicating a correct response.
  • RL formulation: Following prior work (Deng et al., 2022; Kwon et al., 2024), the method casts prompt optimization as reinforcement learning over a system-prompt generation policy π′ conditioned on meta-prompt s.Training aims to make π′ generate increasingly effective system prompts.
  • RL formulation: With response policy π fixed, each system prompt receives expected reward r(x′), and training maximizes its expectation under system-prompt policy π′.This rewrites the objective as Ex′∼π′(·|s)[r(x′)].
  • Optimization algorithm: The implementation uses on-policy GRPO without KL regularization or standard-deviation-based advantage normalization, with a value baseline equivalent to RLOO and matching standard policy-gradient updates (Gao et al., 2025).The formulation is selected for a clean RL objective.

3 Prompt Learnability

This section studies which user prompts are learnable through system-prompt optimization, using Monte Carlo reward estimates and preliminary experiments on IFBench and AIME. Optimization improves IFBench but not AIME, motivating analysis of reward variance in the single-prompt setting.

  • 3 Prompt Learnability: The preliminary experiments frame prompt learnability as a question of which user-prompt classes can be effectively improved by better system prompts.This framing connects the contrasting benchmark behavior to the subsequent variance analysis.
  • 3 Prompt Learnability: Reward collection samples N system prompts and M responses per user prompt, producing KM responses per system prompt and KNM responses per optimization step.The Monte Carlo estimator approximates each system prompt’s dataset correctness while marginalizing response-policy stochasticity through repeated sampling.
  • 3.1 Initial Investigations: The experiments use 64 IF-RLVR questions for training and evaluate on IFBench’s 58 out-of-distribution constraints, alongside AIME.Qwen3-4B-Instruct-2507 serves as both the frozen response policy and the updated system-prompt policy.
  • 3.1 Initial Investigations: Using N = 16 and M ∈ {1, 2}, reward collection remains expensive, taking around 1,400 seconds per training step even when M = 2.The evaluation averages accuracy across four system prompts generated from the trained policy.
  • 3.1 Initial Investigations: Optimization improves IFBench but not AIME: training reward and evaluation accuracy rise on IFBench, while both remain flat on AIME; GEPA shows the same contrast.These results motivate identifying which user-prompt properties make optimization more effective.
  • 3.2 Variances for One Prompt: For a single user prompt, each fixed system prompt induces a Bernoulli reward distribution with mean p, estimated from M sampled responses.The section then considers variance across rewards from N generated system prompts and its expectation over response-sampling randomness.

M | {z } among responses

Prompt optimization depends on separating system-prompt differences from response-generation noise: it is easier when system prompts produce distinguishable rewards. Increasing dataset size can erase this signal, motivating p1, which selects a small high-variance prompt subset.

  • Variance decomposition: Reward variance decomposes into response-sampling variance and variance among system prompts, with repeated sampling reducing the former to clarify system-prompt quality.The response term reflects Monte Carlo estimation, while the system-prompt term reflects true differences in expected reward.
  • Prompt sensitivity: IFBench has substantially higher system-prompt variance than response variance, whereas AIME is dominated by generation stochasticity and has overlapping response spaces across prompts.Thus, IFBench provides a clearer learning signal, while AIME is more challenging for prompt optimization.
  • Prompt learnability: Single-prompt AIME optimization improves training reward when the selected prompt has sufficient variance among system prompts, linking prompt learnability to system-prompt sensitivity.This explains why individual high-variance prompts can be learnable even when optimizing on the full AIME 24 dataset yields no improvement.
  • Dataset size and signal-to-noise: Increasing the number of user prompts reduces response variance as 1/(KM), but also decreases system-prompt variance because heterogeneous prompts favor different system prompts.As datasets grow, good and bad system prompts become less distinguishable; the signal-to-noise ratio therefore worsens, especially when increasing K reduces M.
  • p1 prompt filtering: p1 retains a small subset of user prompts with the largest variance across candidate system prompts, using Ktop = 2 by default to preserve a distinguishable optimization signal.The method estimates system-prompt variance after accounting for response variance and selects the highest-scoring subset.

5 Experiments

Across reasoning benchmarks, p1 outperforms GEPA and full-dataset RL by filtering for user prompts that better distinguish system-prompt quality. Its advantage is strongest on heterogeneous reasoning tasks, whereas IFBench favors GEPA and RL and reveals overfitting risks from overly small subsets.

  • Experimental setup: The experiments compare p1 with GEPA and full-training-set RL while keeping the overall optimization budget roughly constant across methods.For p1, reducing Ktop is offset by increasing M so that KM remains approximately constant.
  • Reasoning benchmarks: p1 outperforms GEPA and full-dataset RL on AIME and HMMT, showing that filtered prompt subsets provide a stronger optimization signal than standard full-dataset training.Table 1 evaluates p1 using top-ranked subsets with Ktop ∈ {1, 2, 4}.
  • Reasoning benchmarks: Using Ktop = 1 improves training reward but fails to generalize, whereas slightly larger subsets produce much stronger generalization.The single-prompt setting overfits, illustrating that maximizing training reward alone can mislead prompt optimization.
  • IFBench: On IFBench, GEPA and RL perform similarly while p1 underperforms, indicating that filtering is less effective on this relatively homogeneous task.Smaller Ktop values increase training-reward gains but do not improve evaluation performance.
  • Prompt analysis: Qualitative inspection shows that p1 produces broadly applicable reasoning guidance, while GEPA produces more domain- and example-specific instructions.The learned prompts suggest that GEPA memorizes training-set patterns whereas p1 discovers more transferable reasoning behaviors.

6 Related Work

Prompt optimization is presented as a lightweight alternative to updating model weights, with prior work spanning evolutionary, search-based, and reinforcement-learning-based methods.

7 Limitation & Conclusion · Appendix · A Preliminary Investigation Details

The study attributes prompt-optimization success to variance among system prompts and proposes p1 to exploit high-variance user subsets, while noting limitations of binary rewards and incomplete understanding of subset-to-distribution generalization.

  • 7 Limitation & Conclusion: p1 selects a small subset of user prompts with high variance among system prompts, leading to substantially better prompt optimization on reasoning benchmarks.The method is motivated by the finding that increasing user prompts reduces system-prompt variance, particularly on heterogeneous mathematical-reasoning tasks.
  • 7 Limitation & Conclusion: Prompt optimization succeeds on some tasks but fails on others partly because its effectiveness depends on variance among system prompts.
  • 7 Limitation & Conclusion: Increasing the number of user prompts can reduce variance among system prompts, especially on heterogeneous tasks such as mathematical reasoning.
  • 7 Limitation & Conclusion: The analysis is limited to binary rewards, whose clean variance formulation does not cover dense-reward environments.
  • 7 Limitation & Conclusion: The results suggest that selected high-variance subsets can produce prompts that generalize well.
  • 7 Limitation & Conclusion: A fuller understanding of when subset performance correlates with full-distribution performance remains an important direction for future work.

A.1 Dataset Details … A.3.1 Generated System Prompt from Experiments in Sec. 3.1

The appendix specifies training and evaluation datasets for AIME and IFBench, the Qwen3-4B-Instruct-2507 training setup, and generated system prompts with their observed accuracies. Across 500 steps, IFBench and AIME prompts show modest accuracy increases on their respective evaluation sets.

  • A.1 Dataset Details: AIME uses 30 AIME 24 questions for training and 30 AIME 25 questions for evaluation, while IFBench uses 64 IF-RLVR questions for training and 294 IFBench questions for evaluation.
  • A.2 Model Details: Training uses Qwen3-4B-Instruct-2507 with full-parameter updates on four H100 GPUs and a fixed learning rate of 1e-6.One GPU updates π′ while three generate responses from π.
  • A.3.1 Generated System Prompt from Experiments in Sec. 3.1: The initial IFBench prompt directs strict compliance with task constraints and requires responses to begin and end as specified.
  • A.3.1 Generated System Prompt from Experiments in Sec. 3.1: IFBench accuracy rises from 34.69 at step 0 to 38.61 at step 500 under the generated system prompts.The prompts emphasize exact adherence to formatting, stylistic, content, keyword, capitalization, word-limit, and structural constraints.
  • A.3.1 Generated System Prompt from Experiments in Sec. 3.1: By step 500, the IFBench prompt adds explicit controls for sentence structure, capitalization, word limits, required keywords, and preventing extra text after the answer.
  • A.3.1 Generated System Prompt from Experiments in Sec. 3.1: AIME 25 accuracy rises from 45.21 at step 0 to 46.15 at step 500 under the generated system prompts.The prompts frame the model as a rigorous competition-level mathematical problem solver emphasizing precision, logic, insight, and edge-case handling.
  • A.3.1 Generated System Prompt from Experiments in Sec. 3.1: The step-500 AIME prompt presents MathMaster Pro as a patient, relentlessly logical solver that anticipates hidden patterns and edge cases.

A.3.2 Generated System Prompt from Experiments in Sec. 3.2

Experiments in Section 3.2 generated competition-mathematics system prompts with training rewards ranging from 0.4375 to 0.75. The strongest prompts emphasize specialized contest expertise, rigorous step-by-step reasoning, and efficient or insightful solutions.

  • Generated system prompts: Other generated competition-math prompts received lower training rewards of 0.5, 0.46875, and 0.4375.These prompts similarly describe expertise in advanced contest domains including algebra, geometry, number theory, combinatorics, probability, and functional equations.
  • Generated system prompts: A competition-level mathematics prompt emphasizing accurate, rigorous, clear, and step-by-step solutions achieved a training reward of 0.6875.Its stated coverage includes contests such as AMC, AIME, Olympiad, USAMO, IMO, and Putnam, along with advanced mathematical topics.
  • Generated system prompts: A generated “MathMaster Pro” system prompt achieved the highest reported training reward of 0.75.It frames the model as a specialized solver for AMC, AIME, ARML, and Olympiad-style problems, emphasizing accurate, efficient, elegant, and insightful reasoning.

B Theoretical Analysis … D Experiment Details

The theoretical appendices derive variance terms for system-prompt rewards, while the p1 pseudocode selects informative user prompts before optimizing system prompts on the selected subset.

  • B.1 Proof for Sec. 3.2: The Sec. 3.2 proof computes the relevant variance components separately under independence assumptions.The derivation explicitly uses independence among sampled responses and evaluates the terms separately.
  • B.2 Proof for Sec. 3.3: The Sec. 3.3 proof defines expected reward for each system prompt and analyzes variance across the N sampled system prompts.The proof separates the across-prompt variance calculation from the response-level term.
  • B.2 Proof for Sec. 3.3: The Sec. 3.3 derivation models each reward estimate as an average of KM independent Bernoulli variables.This establishes the response-level variance contribution used in the subsequent calculation.
  • C Pseudo-code for p1: The p1 procedure first samples candidate system prompts and evaluates them on user prompts using a filtering budget.Its inputs include N candidate prompts, Mfilter, Mtrain, Ktop, and T, and it outputs a trained system-prompt generator.
  • C Pseudo-code for p1: p1 scores every Ktop-sized user-prompt subset by subtracting response variance from total reward variance, then selects the maximizing subset S⋆.The selection rule is S⋆ ← arg max over subsets of size Ktop of the resulting score.
  • C Pseudo-code for p1: After filtering, p1 repeatedly samples system prompts and reward-bearing responses on S⋆ for T optimization updates.Training uses the separate Mtrain budget for each selected user prompt and candidate system prompt.

D.1 Dataset Details … E Related Work

The appendix reports dataset, model, and optimization settings, then shows generated system prompts and their benchmark accuracies for RL and p1. Related work situates the approach among automatic, search-based, reinforcement-learning, inductive, retrieval, and robustness-focused prompt optimization methods.

  • D.1 Dataset Details: The AIME training set contains 30 AIME 24 questions, while the IFBench training set contains 64 questions sampled from IF-RLVR.These experiments use extended context for Qwen3-1.7B.
  • D.2 Model Details: Experiments use full-parameter training on four H100 GPUs with Qwen3-4B-Instruct-2507 and Qwen3-1.7B; Qwen3-1.7B runs in thinking mode with thinking text excluded from reward computation.One GPU updates π′ while the other three generate responses y from π.
  • D.3 Hyperparameter Details: The appendix specifies three-day maximum training, N = 16, learning rate 1e-6, and temperature, top-p, and top-k settings for π′, π, RL, and p1.GEPA uses reflection minibatches of 4; p1 and RL use Qwen3-4B-Instruct-2507 in the listed hyperparameter settings.
  • D.4 Generated System Prompts: 37.41 IFBench accuracy for p1 exceeds 31.97 for RL with Qwen3-1.7B, while RL reaches 39.46 with Qwen3-4B-Instruct-2507.For Qwen3-1.7B, another p1 prompt records 29.25 IFBench accuracy.
  • E Related Work: Automatic prompt optimization improves prompts algorithmically rather than through manual engineering, building on work showing that carefully designed prompts can elicit strong model capabilities (Zhou et al., 2023).The related-work discussion presents automatic optimization as a lightweight alternative to weight updates for downstream adaptation.
  • E Related Work: Search-based methods optimize prompts through mutation, reflection, selection, or textual feedback, including PromptBreeder (Fernando et al., 2023), Self-Taught Optimizer (Zelikman et al., 2024), and EvoPrompt (Guo et al., 2025).A separate line formulates prompt optimization as reinforcement learning, including RLPrompt (Deng et al., 2022), StablePrompt (Kwon et al., 2024), PReWrite, and PromptMII.
  • E Related Work: Related approaches induce or retrieve prompts from examples, while other studies examine cross-model transfer and robustness or security implications, including prompt attacks.Instruction induction includes Honovich et al. (2022), retrieval includes UPRISE (Cheng et al., 2023), transfer includes Wang et al. (2025), and prompt attacks include Zhao et al. (2026).
Loading 2604.08801v1…