PaperlayerView exact revision on arXiv

Source-linked AI summary

Reflective Prompt Optimization of a Deep-Research Agent: A Topic-Disjoint, Multi-Seed Protocol for Measuring Transferable Gains and Prompt-Level Overfitting

Elias Lumer, Sahil Sen, Vamse Kumar Subbiah, Kevin Paul

arXiv:2610.12601v1cs.CL

TL;DR

On factual tasks, prompt optimization can copy training answers, so an in-distribution gain may not transfer. This paper applies GEPA to both prompts of a multi-agent research agent and evaluates it with topic-disjoint testing and verifier controls. Held-out gains replicate across seeds, while decomposed per-dimension feedback distinguishes transfer from scalar-reward optimization.

  • Problem

    On factual tasks, prompt optimization can copy training answers, so in-distribution gains do not establish transferable behavior.

    Source: §1 · p. 1

  • Method

    The study applies GEPA to both prompts of a two-module research agent and tests it with topic-disjoint evaluation and verifier controls.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    0.079 held-out gain on 500 topic-disjoint tasks replicated across 3 seeds at mean 0.107; scalar feedback makes transfer negative.

    Source: §5 · p. 14, §7 · p. 19

  • Takeaways & Limitations

    Decomposed per-dimension feedback supplies diagnostic information for transfer, whereas collapsing it to a scalar removes that channel.

    Source: §5 · p. 14

    The protocol detects prompt-level overfitting but does not prevent the reflector from writing memorized answers into prompts.

    Source: §6 · p. 18

Abstract

from arXiv · show

Prompt optimization is an attractive alternative to reinforcement learning for adapting language-model agents, since it is cheap and produces human-readable artifacts. On factual tasks, however, it carries a failure mode that scalar-reward reinforcement learning does not. An optimizer that reads the training examples can write their answers into the prompt, so an in-distribution gain may be memorization rather than transferable behaviour. We apply reflective prompt evolution (GEPA) to the two system prompts of a multi-agent deep-research agent and evaluate it under a topic-disjoint split in which no topic is shared across train, validation, and test. Held-out citation reward rises by 0.079, from 0.702 to 0.781, with a bootstrap 95\% confidence interval of [0.051, 0.107], and the gain replicates across three seeds (mean 0.107). Nearly half of the winning prompt is a memorized answer key, yet a controlled ablation shows this content earns no reward even on the exact training questions it encodes. The large validation-minus-held-out gap is therefore small-sample selection under a noisy judge, not memorization. Varying the verifier isolates what makes the gain transfer. Decomposed per-dimension feedback is the causal ingredient, and collapsing it to a single scalar reward makes held-out transfer negative. The gain persists under an out-of-family judge and is largely reproduced by a hand-written version of the discovered recipe. Credible evaluation of prompt optimizers therefore requires held-out topics, multiple seeds, and verifier ablations. We release the candidate prompts, lineage diffs, figures, tables, and analysis.

1 Introduction

On factual tasks, prompt optimization can encode training answers, making in-distribution gains insufficient evidence of transferable behavior. The paper uses topic-disjoint evaluation and controlled checks to distinguish transfer from memorization.

Motivation

  • Prompt optimization is presented as a cheaper, auditable alternative to reinforcement learning, which typically needs thousands of rollouts and compresses execution traces into scalar rewards.The optimizer can nevertheless copy training answers into prompts, creating a factual-task failure mode.Source: §1 · p. 1

Evaluation protocol

  • GEPA reflects on execution traces and diagnostic feedback to propose targeted prompt rewrites, while its Pareto frontier preserves multiple candidates.The topic-disjoint protocol is designed to separate transferable rules from specific facts copied into prompts.Source: §1 · p. 2
  • Topic-disjoint testing, a topic-shared ablation, and multi-seed replication find that the memorized answer key does not improve reward.The validation-minus-held-out gap is consistent with small-sample selection under a noisy judge, not memorization.Source: §1 · p. 2
  • Across 3 seeds, held-out gain averaged 0.107 and was significant in every run on 500 tasks.The protocol also tests whether the validation-minus-held-out gap reflects memorization; the answer key does not improve reward, and the gap is consistent with small-sample selection under a noisy judge.Source: §1 · p. 2

Verifier ablation

  • Decomposed per-dimension feedback is the causal ingredient behind transfer; collapsing it to a scalar makes held-out transfer negative.Finer reward resolution and repeated scoring do not raise transfer, while reward gaming tracks the weakest verifiers rather than the strongest.Source: §1 · pp. 2–3

Prompt autopsy

  • Nearly half of the winning prompt is a memorized answer key, but that content earns no reward even on the exact training questions it encodes.The prompt autopsy identifies literal memorization rather than benign paraphrase.Source: §1 · p. 3

2 Related work

Related work situates GEPA among prompt optimizers, multi-module agent optimization, and methods using natural-language feedback. The paper distinguishes its focus on whether prompt-optimization gains transfer to unseen factual topics.

Automatic prompt optimization

  • GEPA extends prompt-search methods by reflecting on full execution traces in natural language, and this study uses its reference implementation.The passage situates GEPA after black-box instruction search, evolutionary prompt methods, and structured instruction-and-demonstration search.Source: §2 · p. 3

Optimizing multi-module language-model programs

  • The system is a two-module orchestrator-researcher program, within a field that also optimizes across multiple language-model modules.Related approaches combine fine-tuning with prompt search, align local rewards to global objectives, or compose policy gradients with prompt optimization.Source: §2 · p. 3

Reflection and natural-language feedback

  • Reflective prompt evolution follows the premise that written diagnoses carry more information than a single reward number.Related work includes textual critiques propagated through language-model calls and methods that accumulate reusable lessons in external memory.Source: §2 · p. 3

Reinforcement learning and verifiable rewards for agents

  • Unlike weight-focused reinforcement learning on retrieval agents, this study optimizes prompts against a language-model-judged rubric and measures citation quality.The related-work discussion notes that rubric-style rewards are prone to hacking and that prompt optimization introduces leakage risk.Source: §2 · p. 4

Citation faithfulness and source attribution

  • The task concerns whether cited sources support claims, a source-attribution problem previously documented for deep-research agents [Onweller et al., 2026].The paper distinguishes its measurement of transfer from prior prompt-optimization studies reporting in-distribution gains.Source: §2 · p. 4
  • Prior prompt-optimization studies report in-distribution gains, while this work tests whether gains survive on topics unseen by the optimizer.The paper frames that distinction as separating transferable behavioral change from answers memorized from training examples.Source: §2 · p. 4

3 Method

The study optimizes both prompts of a multi-agent research system against a gated citation-quality reward using GEPA. Topic-disjoint splits and controlled split, seed, and verifier experiments test transfer and the properties of the reward signal.

3.1 System under optimization

  • The optimized system has an orchestrator that plans and delegates to a researcher, which searches a frozen corpus, consolidates citations, and writes a cited report.Both system prompts are optimized; the orchestrator prompt covers planning, delegation, synthesis, report structure, and citation rules, while the researcher prompt covers search strategy, evidence standards, and response format.Source: §3.1 · p. 4

3.2 Reward

  • The reward is a gated composite in [0, 1] targeting documented source-attribution failure modes [Onweller et al., 2026].A wrong answer zeroes the reward; citation levers are per-citation scores averaged over a task, covering link existence, relevance, factual support, and numbering.Source: §3.2 · pp. 4–5

3.3 GEPA configuration

  • GEPA uses Pareto candidate selection, a reflection minibatch of 3, and a budget of 240 metric calls.Its adapter runs candidate prompts, scores with textual feedback naming each lever and weakest-citation reasons, and records traces for the reflector.Source: §3.3 · p. 5

3.4 Topic-disjoint split

  • Train and validation each contain 24 tasks over 12 topics; test contains 500 tasks over 254 topics, with no article family shared across splits.Whole topic groups are assigned to test, then validation, then train; the held-out test set is evaluation data, not optimization data.Source: §3.4 · p. 5, Table 1 · p. 5

3.5 Ablation protocols

  • The split-and-seed experiments repeat optimization under 3 seeds and score the selected prompt on training-family questions to test whether memorized answers earn reward.The verifier experiments vary one verifier property at a time, then rescore optimized prompts on 120 topic-disjoint tasks with a canonical verifier.Source: §3.5 · pp. 5–6

3.6 Models

  • The policy agent is GPT-5.4 and the reflector is GPT-5.5 at high reasoning effort; citation judges use GPT-5-mini for relevance and GPT-5.4-mini for factual support.These per-dimension judge choices follow benchmarking against human labels for deep-research source attribution [Leung et al., 2026].Source: §3.6 · p. 6

4 Experiments

The optimized prompt improved topic-disjoint held-out reward, and the gain replicated across seeds. Tests of the validation gap found no reward benefit from the memorized answer key; the gap was consistent with small-sample selection under a noisy judge.

4.1 Experimental setup

  • The evaluation scored the seed and best prompts on all 500 held-out tasks and used bootstrap intervals, paired significance tests, effect size, and win/tie/loss counts.Lever analysis rescored candidates on validation and the seed and best prompts on a 200-task held-out subset.Source: §4.1 · p. 7

4.2 Results

  • GEPA searched 240 rollouts, raising validation reward from 0.672 at the seed to 0.874 at the best candidate, found at rollout 156.Pareto selection retained shorter candidate 4 after the two longer candidates regressed on validation.Source: §4.2 · p. 7, Figure 1 · p. 8
  • 0.079: the optimized prompt beats the seed on 500 topic-disjoint held-out tasks, with reward rising from 0.702 to 0.781.The held-out gain is smaller than the validation gain.Source: §4.2 · p. 7, Figure 1 · p. 8
  • On the 200-task held-out decomposition, improvement spanned all three citation-quality levers, with factual support the largest gain but still the lowest lever.Factual support rose from 0.53 to 0.62; the answer gate moved little and citation numbering was saturated.Source: §4.2 · pp. 7–8

4.3 Robustness across seeds

  • Across three seeds, held-out gains were positive and significant in every run, averaging 0.107 and ranging from 0.079 to 0.130.The validation-minus-held-out gap varied from 0.034 to 0.160, consistent with optimizer-selection and judge variance rather than a stable memorization signal.Source: §4.3 · p. 8

4.4 Does the memorized answer key earn reward?

  • The 24 exact training questions produced only a 0.090 gain, statistically indistinguishable from the 0.079 held-out gain and far below the 0.202 validation gain.Fresh same-topic questions gained less in memorized families than non-memorized families (0.071 versus 0.154); the answer-key blocks behaved as prompt bloat, not useful lookups.Source: §4.4 · pp. 8–9

4.5 Which verifier property drives transfer?

Verifier ablations identify decomposed per-dimension feedback as the ingredient associated with positive held-out transfer; finer or repeated scoring did not improve it. A hand-written recipe recovered about 57% of the total held-out gain on its evaluation tasks.

4.5 Which verifier property drives transfer?

  • +0.075 held-out transfer with decomposed feedback became −0.021 when feedback was collapsed to a scalar reward.The scalar variant either found no accepted improvement or improved validation while regressing on held-out tasks.Source: §4.5 · p. 9
  • Mean transfer was 0.079 with coarse scoring, 0.075 with fine scoring, and 0.059 with fine scoring repeated 3x.Repetition produced the tightest seed-to-seed spread (0.018 versus 0.037 for the reference and 0.079 for coarse scoring), at 3x the judge calls.Source: §4.5 · p. 10
  • The gaming gap was largest for low-information verifiers: 0.071 for coarse and 0.061 for scalar, versus 0.043 for the fine decomposed reference.Held-out rescoring exposed the scalar case, where a positive validation delta hid a held-out regression.Source: §4.5 · p. 10

4.6 A hand-written recipe baseline

  • The hand-written recipe recovered 0.047 of the 0.083 total held-out gain on 100 tasks, about 57%.The full prompt added 0.036, but that increment was not individually significant at this sample size (95% confidence interval [−0.021, +0.096]).Source: §4.6 · p. 11

4.7 Cost and latency

The optimized prompt added no inference machinery or observed latency penalty, though it increased input-token use. Spot-checks found the gain persisted under an out-of-family judge, with the authors noting limits to those checks.

4.7 Cost and latency

  • The optimized prompt averaged 138.8 seconds per query versus 159.9 for the seed, with output length unchanged at about 3200 tokens.Mean total policy tokens rose about 32%, from 146.0k to 193.1k per query, almost entirely from an added 47k input tokens; the one-time search used 240 policy rollouts, about 40M policy tokens.Source: §4.7 · p. 12

4.8 Judge validation and self-preference

  • On 12 held-out tasks, the out-of-family judge gave a seed-to-best gain of 0.129, versus 0.069 in-family on the same tasks.The out-of-family judge matched human decisions on 91.1% of relevance and 80.0% of factual-support decisions; the spot-check used 48 pairs and one annotator.Source: §4.8 · p. 13

5 Discussion

The validation gain overstates transfer, while the memorized answer key does not account for the gap; decomposed feedback and topic-disjoint evaluation help identify what transfers and where errors remain.

5 Discussion

  • Decomposed verifier feedback surfaces specific citation failures for the reflector to fix, while collapsing feedback to one scalar makes held-out transfer negative.The diagnostics identify unresolved links, off-topic pages, or pages that do not support the claim; the scalar ablation points to per-dimension feedback, not search machinery alone, as the gain's mechanism.Source: §5 · p. 14
  • The winning program devoted 52% of orchestrator and 39% of researcher prompt words to a memorized answer key, which added only 0.090 on the exact training questions it encoded, versus a 0.202 validation gain.Source: §5 · p. 15

5.1 Why the topic-disjoint protocol is necessary

  • A validation-only workflow would report a 0.202 gain against a transferable held-out gain of 0.079, an overstatement of about 2.6 times.The topic-disjoint split makes the held-out score a transfer estimate; the validation-minus-held-out difference measures the part that does not transfer.Source: §5.1 · p. 16
  • Memorized answer-key share and validation reward rose together (Pearson r = 0.83), but the topic-shared ablation found the key did not earn reward on encoded questions.The passages attribute their co-movement to memorization and transferable citation-discipline edits accruing along the same lineage, not to evidence that memorization caused the validation gain.Source: §5.1 · p. 16
  • The topic-shared ablation raised the apparent gain only to 0.112, short of the 0.202 validation gain, and memorized families gained less than non-memorized ones.The passage attributes the remaining gap to small-sample selection under a noisy judge.Source: §5.1 · p. 17

5.2 Error analysis

  • Factual support remained the lowest held-out lever at 0.62, with 59 of 200 tasks below 0.5; weak citations most often failed to support the specific claim.The corpus-only-URL rule largely worked, as out-of-corpus URLs were nearly gone.Source: §5.2 · p. 17, Table 12 · p. 17
  • On the paired 200-task subset, the answer gate gained on 12 tasks and lost on 6; its mean factual-support change was positive at 0.095.Losses could be large when the more assertive answer-first prompt committed to a wrong specific answer, but gains outnumbered losses.Source: §5.2 · p. 17

6 Limitations and Future Work

The evidence is bounded by a single agent, citation task, and news corpus, and several checks use small samples or few seeds. The authors identify broader validation and safeguards against prompt-level memorization as future work.

Judge validation

  • The reward is scored by language-model judges, and the study's human-agreement check is a small, single-annotator spot-check.A larger multi-annotator study on optimized outputs would strengthen validation; the reported gains are against judges with demonstrated human agreement.Source: §6 · p. 17

Constraining the optimizer

  • The topic-disjoint protocol detects prompt-level overfitting but does not prevent answer-key edits; the study proposes constrained, validation-gated updates as a complementary safeguard.Here the memorized key did not improve reward, making its cost prompt hygiene and auditability; in a domain where it transfers, the same edit could be hazardous.Source: §6 · p. 18

Small-sample validation

  • The optimizer selects among candidates using 24 validation tasks scored by a stochastic judge, leaving the winning validation reward vulnerable to winner's-curse inflation.Larger validation sets, selection penalties, or repeated scoring were not run and are left for future work.Source: §6 · p. 18

Seed replication

  • Three seeds replicate the held-out gain's direction, but the gain magnitude varies from 0.079 to 0.130 and three runs cannot characterize that spread precisely.The authors treat the validation-minus-held-out gap as a diagnostic, not a calibrated measurement.Source: §6 · p. 19

Verifier ablation

  • The verifier ablation uses two seeds per configuration and a single canonical verifier on 120 held-out tasks, so its cell means are directional rather than tight estimates.The authors identify more seeds and an out-of-family verifier as follow-up checks; the robust readings are that scalar feedback collapses transfer and reliability does not trend with transfer.Source: §6 · p. 19

Reward and corpus scope

  • The search tool queries a frozen corpus, so link-accessibility and URL-discipline conclusions are relative to that corpus.The paper also notes that reward weights, the answer-gate definition, and judge prompts should be stated in the paper.Source: §6 · p. 19

Scope

  • The study covers one multi-agent deep-research agent, one citation task, and one news corpus, so the transferable recipe and leakage size remain specific to this setting.Whether reflective optimization leaks as heavily on other tasks, agents, and domains is open; the authors identify narrow domains with recurring entities as a natural external-validity test.Source: §6 · p. 19

7 Conclusion

Reflective prompt optimization improved held-out citation reward across seeds. The conclusion attributes transfer to decomposed feedback and a portable prompting recipe, while ruling out the memorized answer key as the source of reward.

7 Conclusion

  • 0.079 held-out citation-reward gain on 500 topic-disjoint tasks without weight updates replicated across 3 seeds at a mean of 0.107.The gain spread across citation-quality levers rather than concentrating in one; factual support carried the largest share.Source: §7 · pp. 19–20
  • Decomposed per-dimension feedback is identified as the causal ingredient; collapsing feedback to a scalar makes held-out transfer negative.Finer or repeated scoring does not raise transfer.Source: §7 · p. 20
  • The nearly half-prompt memorized answer key earns no reward even on the exact training questions it encodes, leaving the validation gap consistent with small-sample selection under a noisy judge.The conclusion attributes the gap to selection, not memorized reward.Source: §7 · p. 20
  • A hand-written version of the discovered recipe recovers most held-out gain, supporting a transferable rule set rather than an effect unique to the search.The recipe uses answer-first structure, one claim per citation, verified page content, and corpus-only URLs.Source: §7 · p. 20

A Reproducibility

The reproducibility workflow separates optimization, re-analysis, and reporting over the released artifact. The analyzed run uses both components with 24 training and validation examples and 500 test examples.

A Reproducibility

  • Re-analysis derives the trajectory, decomposition, and error material, rescoring candidates on validation and seed and best prompts on a 200-task held-out subset.Per-citation reasons are enabled for held-out rescoring.Source: A Reproducibility · p. 25
  • The reporting step produces prompt lineage and diffs, paired held-out statistics, figures, tables, and error analysis.The commands are illustrative and use the released artifact layout.Source: A Reproducibility · p. 25
  • The analyzed optimization run uses both components, train-size 24, val-size 24, test-size 500, 240 maximum metric calls, and high reflection effort.The command runs against dataset.jsonl and corpus.db.Source: A Reproducibility · p. 25

B Artifacts

The released artifacts include prompt candidates and their lineage, analysis data, figures, tables, and write-ups.

B Artifacts

  • The release includes the seed prompt, all candidate prompts, and their pairwise lineage diffs.Source: B Artifacts · p. 26
  • The release includes machine-readable prompt lineage and per-lever analysis data.The lineage file is data/prompt lineage.json.Source: B Artifacts · p. 26
  • Figures, tables, and prompt-evolution and error-analysis write-ups are included, with paths given in the README.Source: B Artifacts · p. 26

C Reward weights, answer gate, and judge prompts

The appendix specifies the composite citation reward, its answer gate, and the model-judge prompts used in the run and re-evaluations.

C Reward weights, answer gate, and judge prompts

  • The composite reward weights link works at 0.25, relevant content at 0.30, fact check at 0.30, and citation numbering at 0.15.The first three are unweighted means over parsed attribution-citation pairs; a report with no parseable pair scores zero on them.Source: C Reward weights, answer gate, and… · p. 26
  • The answer gate cascades from a normalized gold-answer substring match to token-coverage thresholds, escalating only near misses to a language-model judge.Coverage at or above 1.0 passes, below 0.5 fails without a model call, and the near-miss band uses GPT-5-mini to judge whether the report answers with the same fact.Source: C Reward weights, answer gate, and… · p. 26
  • Source relevance and factual support are binary language-model judges that assess whether cited content relates to the attribution and supports its facts.The source-relevance judge is GPT-5-mini; factual support uses GPT-5.4-mini.Source: C Reward weights, answer gate, and… · pp. 26–27
  • Each citation judge receives the attribution, cited URL, and truncated URL content, and all judges see frozen-corpus content without live web fetching.Source: C Reward weights, answer gate, and… · pp. 26–27

Related papers

Loading 2610.12601v1…