Source-linked AI summary
Coding Agents are Strong Prompt Optimizers
Agamdeep Singh, Srishti Gautam, Priyanshu Gupta, Nikita Mehrotra, Tanmay Bakshi, Sumit Gulwani
TL;DR
Prompt optimizers commonly rely on iterative edits, fresh rollouts, and validation, while small trajectory samples can miss corpus-wide behavioral patterns. CASD replaces that loop with a coding agent’s offline analysis of a static rollout corpus; under matched data access, it beats reflective search on most benchmarks at low cost.
Problem
Search-based prompt optimization repeatedly validates edits through fresh environment rollouts, while reflections over small trajectory samples can miss patterns that emerge across the corpus.
Source: §1 · p. 1Method
Coding-Agent Skill Distillation (CASD) uses an off-the-shelf coding agent to compute corpus-wide statistics, inspect informative episodes, and synthesize a prompt in one offline pass.
Source: §1 · p. 1Results
Under matched data access, CASD beats reflective search on three of four benchmarks and costs about $1.60 per prompt.
Source: §8 · p. 7Takeaways & Limitations
Corpus-scale offline reflection may serve as a first step in prompt optimization, with search reserved for final gains when additional interaction is inexpensive.
Source: §8 · p. 7CASD cannot discover behaviors absent from its logged rollouts, and very small or failure-free corpora may leave little to distill.
Source: §7 · p. 7
Abstract
from arXiv · showhide
Search-based prompt optimizers improve prompts through iterative search: they propose edits, execute fresh rollouts, score the resulting trajectories, and retain only edits that improve a validation metric. We show that this optimization loop is unnecessary. Given only a static corpus of agent trajectories, an off-the-shelf coding agent can directly synthesize an optimized prompt, requiring neither environment access nor validation data. We call this approach \textit{Coding-Agent Skill Distillation} (CASD). The key insight is reflection scope. Rather than reasoning over a small batch of trajectories at each optimization step, the coding agent writes and executes analysis code to compute corpus-wide statistics, identifies systematic failure modes, inspects representative episodes, and distills the resulting insights into behavioral rules. Across four agentic benchmarks (ALFWorld, $τ^2$-bench retail and telecom, and SpreadsheetBench-Verified), under matched data access, a single CASD pass outperforms GEPA, a state-of-the-art reflective prompt optimizer, on three of four benchmarks and outperforms validation-gated reflective search (SkillOpt) on all four, improving the unoptimized baseline by 16.6 percentage points on average versus 10.9 for GEPA and 5.3 for SkillOpt. Because CASD performs a single offline analysis pass rather than iterative search, producing an optimized prompt costs approximately \$1.60---over $22\times$ cheaper than validation-gated search. Even when competing methods are granted additional validation data and unrestricted environment access, CASD remains ahead on two of four benchmarks. These results suggest that corpus-scale statistical reflection is a viable alternative to iterative search for prompt optimization.
1 Introduction
The introduction argues that iterative prompt search relies on costly fresh rollouts and small trajectory samples, then presents CASD as a single offline pass that analyzes a static corpus. Under matched data access, CASD outperforms reflective search on most benchmarks; it remains competitive even when those baselines receive extra validation data and environment access.
1 Introduction
- Search-based optimizers repeatedly propose edits, run fresh environment rollouts, and retain prompts that improve a validation objective.Even reflective search diagnoses failures using sampled trajectories, while each revision still depends on repeated rollouts and validation.Source: §1 · p. 1
- Small trajectory samples can miss corpus-scale patterns, while validating each prompt revision requires fresh environment interaction and makes costs grow with candidate edits.The introduction illustrates the missed patterns with duplicated tool calls and a task category that failed in every attempt.Source: §1 · p. 1
- CASD replaces iterative search with one offline pass by an unmodified coding agent over static trajectories, without validation data or additional environment interaction.The resulting prompt costs about $1.60 to produce.Source: §1 · pp. 1–2
- Corpus-scale reflection computes statistics such as pass rates, tool-call and duplicate-call counts, then uses them to select informative episodes for inspection.The approach grounds prompts in measured corpus-wide evidence rather than a handful of trajectories.Source: §1 · p. 1
- Under matched data access, CASD has the largest average improvement across four benchmarks, beating GEPA on three and SkillOpt on all four.With extra validation data and unrestricted environment access, it still outperforms search baselines on two of four benchmarks.Source: §1 · pp. 1–2, Figure 1 · p. 2
2 Related Work
Prior approaches search over prompts using validation scores, reflect on individual trajectories, or repurpose coding agents and offline-improvement ideas. CASD differs by using executable analysis over a frozen corpus to guide targeted inspection.
2 Related Work
- Prompt-search methods propose and score candidate instructions or edits, while evolutionary methods mutate prompt populations; GEPA adds reflection over sampled trajectories and selects using a validation-set Pareto front.The shared scoring gate requires fresh rollouts and rescoring for each candidate.Source: §2 · p. 2
- Reflexion and Self-Refine use per-episode feedback, while skill and memory systems accumulate reusable insights across episodes; these approaches reflect trajectory by trajectory.CASD instead uses executable analysis to compute exact corpus-wide counts that direct targeted reading, rather than relying on context-window recall.Source: §2 · p. 2
- Coding agents interleave code execution, file inspection, and editing; this paper repurposes an unmodified agent to edit prompts using frozen rollouts as its test suite.Source: §2 · p. 3
- CASD is presented as a textual analogue of offline improvement: one support-constrained update over a fixed dataset, without an evaluation gate.The cited offline-RL work discusses constraining improvements to data support and cases where one step can suit small datasets.Source: §2 · p. 3
3 Method: Coding-Agent Skill Distillation
CASD starts from the initial prompt and rollout corpus, then lets an unmodified coding agent determine its analysis process and write a skill file used as the optimized prompt. Across logged runs, the agent alternated corpus-level statistical analysis with targeted episode inspection before synthesis.
3.1 Problem Setup
- The target agent’s model parameters stay fixed while the system prompt is optimized, and executing the initial prompt on training tasks produces the rollout corpus.Each trajectory records the agent’s messages, tool calls and outputs, rewards, and execution metadata.Source: §3.1 · p. 3
3.2 The Distillation Pass
- The coding agent receives the rollout corpus and initial prompt, chooses its own analysis process, and writes a skill markdown file that serves directly as the optimized prompt.The instruction specifies no analysis pipeline, optimization procedure, evaluation metric, or intermediate representation.Source: §3.2 · p. 3
3.3 Inside the distillation process
- Every run measured rewards or pass rates and episode lengths; 96% quantified termination and failure modes, 92% analyzed tool use, and 79% broke performance down by task category.One third explicitly searched for duplicated tool invocations.Source: §3.3 · p. 3
- The 12 CASD skills contained 54 quantitative evidence citations, compared with none across four GEPA prompts and one across four SkillOpt prompts.The CASD skills averaged 4.6 citations per 1,000 words.Source: §3.3 · p. 3
- Across 24 runs, the agent frontloaded exploration, alternated corpus-statistics analysis with targeted episode inspection, and wrote the skill file last.The workflow used statistics to identify episodes for inspection before synthesizing evidence-backed behavioral rules.Source: §3.3 · p. 3, Figure 2 · p. 4
4 Prompt Optimization Through Corpus-Scale Reflection
Prompt optimizers share the goal of improving expected reward, but differ in how they infer prompt changes: sampled, validated search versus offline corpus-scale reflection. Their statistical trade-off makes offline distillation especially suited to limited budgets, while iterative search becomes more attractive with abundant interaction.
4 Prompt Optimization Through Corpus-Scale Reflection
- Prompt optimizers share the objective of maximizing expected target-agent reward, but differ in how they infer prompt improvements from execution experience.Search repeatedly estimates updates from sampled trajectories and validates them with additional rollouts; CASD makes one offline inference from corpus-level statistics.Source: §4 · p. 4
4.1 A Unified View of Prompt Optimization
- In the unified search framework, a proposal mechanism modifies the prompt from execution feedback, and an acceptance rule decides whether to retain the update.Search uses sampled rollout batches and validation estimates, both Monte Carlo estimates from sampled trajectories.Source: §4.1 · p. 4
- CASD instead uses a fixed rollout corpus, corpus statistics, and selected representative trajectories as its feedback for prompt improvement.Source: §4.1 · p. 4
4.2 Bias–Variance Analysis of Prompt Optimization
- Search-based validation feedback is a Monte Carlo estimate whose reliability improves only with additional rollout evaluations.Its variance decreases with the number of validation episodes; per-episode reward variance is denoted σ^2.Source: §4.2 · pp. 4–5
- Once collected, corpus statistics eliminate minibatch sampling variance, but CASD incurs bias when the fixed corpus misses important behaviors.Search instead has decreasing variance as additional rollouts are collected.Source: §4.2 · p. 5
4.3 Operating Regimes
- Under limited rollout budgets, CASD can improve optimization effectiveness at substantially lower cost by replacing iterative search with one offline distillation step.The analysis frames reduced estimator variance as more valuable than eliminating asymptotic bias in this regime.Source: §4.3 · p. 5
- As rollout data and environment interaction grow, search variance falls while fixed-corpus bias remains, making iterative search increasingly attractive.The relative advantage of offline distillation is therefore expected to diminish.Source: §4.3 · p. 5
- Offline corpus-scale reflection and iterative search occupy complementary regimes: offline distillation suits constrained budgets, while iterative search is expected to benefit from abundant interaction.Source: §4.3 · p. 5
5 Experimental Setup
The experiments cover four agentic benchmarks with benchmark-specific held-out accuracy measures, and compare no-skill, CASD, GEPA, and SkillOpt under limited-data and head-to-head regimes. Evaluations use GPT-5.4-mini agents and Claude Sonnet 5 optimizers, with seed-level dispersion reported.
Benchmarks
- The evaluation spans ALFWorld household tasks, τ^2-bench retail and telecom customer service, and SpreadsheetBench-Verified spreadsheet manipulation, using benchmark-specific held-out accuracy measures.The measures are win rate on 50 games, pass@1 on 40 tasks, and modified accuracy on 50 items, respectively.Source: §5 · p. 5
Data regimes
- Table 1 reports held-out test accuracy (%) as mean over three seeds with standard deviation when all optimizers use the static pool.ALFWorld SkillOpt uses its native scaffold and is not directly comparable to the other methods in that column.Source: Table 1 · p. 5
Models
- Target agents use GPT-5.4-mini with reasoning disabled, optimizers use Claude Sonnet 5, and test accuracy is averaged over three seeds with sample standard deviations.At these evaluation-set sizes, binomial standard error alone is approximately seven points per seed.Source: §5 · p. 5
Methods
- The methods are no-skill baseline, CASD, GEPA evolutionary reflective optimization, and SkillOpt reflective search with a validation gate.CASD averages three synthesized skills per benchmark; its standard deviation measures skill-to-skill variation, unlike the baselines’ seed-to-seed variation. GEPA uses 120 metric calls; SkillOpt runs 3–5 epochs with a validation gate.Source: §5 · p. 5
Data regimes
- In the limited-data regime, each optimizer receives the same fixed rollout pool; in the head-to-head regime, GEPA and SkillOpt additionally receive validation data and unlimited candidate-rollout environment access.CASD does not use that additional validation set.Source: §5 · p. 5
6 Results
With matched access to the same static rollout pool, one CASD pass outperforms GEPA on three of four benchmarks and SkillOpt on all four. The results also examine CASD’s cost, corpus-scale reflection, and whether distilled prompts recover benefits of test-time reasoning.
6.1 Matched data access: one pass beats the loops
- CASD improves the baseline by +16.6 points on average, versus +10.9 for GEPA and +5.3 for SkillOpt, and leads GEPA on three of four benchmarks and SkillOpt on four.It scores 83.3 vs. GEPA’s 74.0 on ALFWorld, 40.0 vs. 39.2 on retail, and 39.2 vs. 17.5 on telecom, while ranking second on SSB at 51.3 vs. GEPA’s 60.7.Source: §6.1 · p. 5, Figure 3 · p. 6
6.3 Cost
- A CASD pass costs approximately $1.60 per skill, making it 22× cheaper than SkillOpt’s $142.5 total and less costly than GEPA’s $8.4.CASD requires no new environment rollouts because its corpus comes from rollouts already generated.Source: Table 3 · p. 6, §6.3 · p. 6, §6.1 · p. 6
6.2 Head-to-head: search buys back some ground, at a price
- With extra validation data and environment access, GEPA reaches 46.7 on retail and SkillOpt reaches 44.2 on telecom, while CASD still wins ALFWorld and stays within noise of the best telecom cell.GEPA still scores 15.8 on telecom, while SkillOpt fixes telecom at a gated-search cost of $74.7.Source: §6.2 · p. 6
6.4 Why does corpus-scope reflection win?
- Corpus-wide analysis can reveal absent behaviors: only CASD detects that no rollout invokes make_payment and writes the billing rule needed to solve the two-cause telecom ticket.The device issue is visible in individual trajectories and appears in all three optimizers’ prompts; the billing issue is visible only as a corpus-level absence.Source: §6.4 · p. 6, Figure 4 · p. 7
6.5 Ablation: does the skill recover thinking?
- CASD skills exceed think-mode accuracy on ALFWorld (83.3 vs. 71.3) and retail (40.0 vs. 35.0), while using no reasoning tokens.They recover 78% and 55% of the no-think-to-think gap on telecom and SSB, respectively, with output at or below the no-think budget; skills distilled from no-think rollouts alone perform comparably to those distilled from think rollouts. Residual telecom and SSB gaps suggest that some instance-specific deduction is not recovered by a static prompt.Source: Table 4 · p. 7, §6.5 · p. 7
7 Limitations
The evaluation has scope and measurement limits, and CASD’s ability to distill useful behavior depends on what its corpus contains.
7 Limitations
- CASD’s skill-to-skill SD is not identical to the baselines’ seed variance for a single prompt.CASD variance uses three independently distilled skills with one evaluation each; baselines report seed variance.Source: §7 · p. 7
- Evaluation covers one target model and one coding agent, leaving sensitivity to distiller capability untested.The target is GPT-5.4-mini and the coding agent is Claude Sonnet 5/Claude Code.Source: §7 · p. 7
- CASD cannot discover behaviors absent from its logged rollouts, and very small or failure-free corpora may leave nothing to distill.Source: §7 · p. 7
8 Conclusion
A stock coding agent can optimize prompts from frozen rollouts without environment interaction, using corpus-scale analysis instead of minibatch reflection and validation gating. Under matched data access, it outperforms reflective search on most tested benchmarks at low cost.
8 Conclusion
- On 3 of 4 agentic benchmarks, a stock coding agent beats state-of-the-art reflective search under matched data access.It does not touch the environment and costs about $1.60 per prompt.Source: §8 · p. 7
- The approach analyzes frozen rollouts with a one-paragraph instruction and uses the resulting corpus-scale reflection to optimize the prompt.It executes analysis code over the whole corpus rather than relying on minibatch reflection and validation gating.Source: §8 · p. 7
- The authors suggest offline corpus-scale reflection may become prompt optimization’s default first step, with search reserved for final gains when additional interaction is inexpensive.Source: §8 · p. 7