PaperlayerView exact revision on arXiv

Source-linked AI summary

What Should the Reflector See? An Empirical Study of Evidence in Reflective Prompt Optimization

Xiaofan Zhou, Lu Cheng

arXiv:2609.32452v1cs.CLcs.LG

TL;DR

Which evidence should a reflector receive in prompt optimization remains unclear. The paper compares nine reflection strategies within a single-parent Pareto-guided search on five datasets using Qwen3.5-9B as both task model and reflector. Failures-only has the largest average test gain, while Balanced-mix and No-examples+Val share the best mean accuracy rank.

  • Problem

    Prompt revisions use examples of model behavior, but which evidence the reflector should receive remains unclear.

    Source: §1 · p. 2

  • Method

    The study compares nine reflection strategies varying evidence composition, example visibility, candidate selection and domain-knowledge policy within a single-parent Pareto-guided search.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Failures-only yields the largest average gain (8.0 points), while Balanced-mix and No-examples+Val share the best mean accuracy rank (4.0).

    Source: §5.1 · p. 6

  • Takeaways & Limitations

    No strategy wins on every task, so strategy choice should follow task-specific validation rather than aggregate gain alone.

    Source: §5.1 · p. 6, §6 · p. 9

    Using one model family for proposing and answering may make strategy rankings reflect model-specific strengths and shared failure modes, limiting generalizability.

    Source: §6 · p. 9

Abstract

from arXiv · show

Reflective prompt optimization revises instructions using examples of a model's behavior, but which evidence the reflector should receive remains unclear. We study evidence composition, visibility of examples, candidate selection and domain-knowledge policy within a single-parent Pareto-guided search. Using Qwen3.5-9B as both task model and reflector, we evaluate nine reflection strategies on five datasets. From these experiments, we find three distinct patterns. For performance improvement, Failures-only produces the largest mean test gain (+8.0 percentage points), while Balanced-mix and No-examples+Val share the best mean performance rank. For reflection effectiveness, No-examples achieves the best rank for improving sampled parents, yet yields only a 1.4-point mean test gain: local reflection success does not necessarily produce a stronger final prompt. For overfitting assessment, Failures-only and Balanced-mix share the lowest mean calibration-gap rank, while larger gaps on GPQA and IFBench show that calibration gains can overstate held-out improvement. This gap is a descriptive indicator, not a direct measure of overfitting. Together, these results show why reflection strategies should be assessed separately on final performance, parent improvement and calibration-to-test transfer.

1 INTRODUCTION

Prompt quality affects how much users obtain from LLMs, while writing effective prompts can demand expertise and repeated trial and error. This paper compares what evidence a reflector receives within a shared search procedure.

1 INTRODUCTION

  • LLMs can solve general-purpose tasks, but users’ realized capability often depends on prompts, especially for hard reasoning tasks.Writing effective prompts can require experienced prompt engineers and many rounds of trial and error; fine-tuning instead requires weight access and substantial compute.Source: §1 · p. 2
  • Prompt revisions depend on a small batch of examples read each round, making the choice of evidence central to what the method can learn.The loop runs a selected prompt on training examples, has an LLM revise it from the results, then keeps the revision if it improves held-out performance.Source: §1 · p. 2
  • Training-free prompt evolution keeps the model fixed while an LLM automatically proposes, tests and refines prompts.The paper positions its comparison of evidence design alongside prior work advancing search procedures and targeted evidence analyses.Source: §1 · p. 2
  • The study compares nine reflection strategies across five datasets, varying evidence composition, example visibility, candidate selection and domain-knowledge policy within one search backbone.Failures-only has the largest average gain, while Balanced-mix and No-examples+Val share the best mean accuracy rank; no strategy is best on every task.Source: §1 · p. 2

2 RELATED WORK

Prior work develops prompt optimizers, search procedures and feedback-driven revision; this paper holds one search procedure fixed to study the evidence its reflector reads.

2 RELATED WORK

  • Prompt optimizers such as APE, OPRO and GEPA propose different ways to generate and refine prompts, whereas this study isolates the evidence read by a reflector within one optimizer.The cited methods use demonstrations, earlier prompts with scores, or execution traces and textual feedback, respectively.Source: §2 · p. 2
  • Evolutionary prompt methods advance parent selection and candidate maintenance or combination, while this study fixes the search procedure and varies what the LLM sees.Examples include EvoPrompt’s crossover and mutation, GEPA’s Pareto front, and C-Evolve’s prompt group with majority-vote outputs.Source: §2 · pp. 2–3
  • Feedback-driven methods use critiques or task feedback to revise outputs or prompts, and StraGo separately analyzes positive and negative experiences.The paper compares evidence mixes and restrictions on reflector outputs under the same search procedure and iteration budget.Source: §2 · p. 3
  • Reflective optimizers can revise each module of a multi-call system from its gathered evidence, but this study isolates the question in a single-prompt setting.Related systems include DSPy, MIPRO and MASPOB, which optimize demonstrations, instructions and demonstrations, or multi-agent prompts.Source: §2 · p. 3

3 METHOD

The method uses a single-parent Pareto-guided search and varies evidence, example visibility, candidate selection and knowledge policy across reflection strategies.

3.1 SINGLE-PARENT PARETO-GUIDED SEARCH

  • The search keeps generated prompts and calibrated prompts with per-example scores, samples a parent from the strict Pareto front by mean calibration score, and falls back to uniform sampling when weights are all zero.A prompt is on the strict front when no other calibrated prompt is at least as good on every calibration example and strictly better on one.Source: §3.1 · p. 3
  • Each iteration evaluates one parent on 20 training examples, generates five children from five three-example groups, and returns the prompt with the highest calibration mean after 20 iterations.Empirical selection tests children against their parent on the same group, calibrates only strictly improved children, and excludes the held-out test set from search.Source: §3.1 · p. 3

3.2 REFLECTION STRATEGIES

  • Failures-only draws groups from parent failures and skips reflection and calibration when there are none, while Natural-mix samples the evaluated batch uniformly and Balanced-mix targets a near-equal mix of successes and failures.Failures are repeated to fill a group when only one or two are available; Balanced-mix uses a 2:1 or 1:2 split and fills shortages from the other class.Source: §3.2 · p. 3
  • No-examples generates from the task description and current instruction alone and sends its first parseable candidate directly to calibration; No-examples+Val adds empirical selection and a local gate.The blind No-examples strategy does not use the common parent batch for generation or selection.Source: §3.2 · p. 4
  • Alternating switches between blind and natural-evidence generation, while LLM-judge selects among children before the selected child faces a local improvement gate.The judge sees candidate texts and evidence, but not empirical child scores.Source: §3.2 · p. 4
  • No-knowledge and With-knowledge use natural evidence and empirical selection but respectively prohibit or request domain facts, formulas and mechanisms in reflections.Both allow procedures and output conventions; other data-grounded strategies use a neutral template on domain-specific knowledge.Source: §3.2 · p. 4

4 EXPERIMENTAL SETUP

The experiments compare nine strategies on five datasets using a shared model, examples and iteration budget, while realized model-call costs can differ.

4.1 RESEARCH QUESTIONS

  • The study asks which strategies improve held-out performance, which reflection choices drive differences, and how reliably local improvements advance the search.It compares test accuracy, improvement over the initial prompt and within-dataset rank, then examines local gains, search rounds and calibration-to-test relationships.Source: §4.1 · p. 4

4.2 DATASETS

  • The five datasets cover mathematics, graduate-level science, multi-hop question answering and instruction following, with disjoint training, calibration and test examples.The default split sizes are 100/150/150; HotpotQA uses a retrieve-then-read setup exposing three of ten candidate paragraphs, while IFBench accuracy is the fraction of constraints satisfied.Source: §4.2 · p. 4, Table 1 · p. 5

4.3 IMPLEMENTATION AND SEARCH BUDGET

  • Qwen3.5-9B performs task execution, reflection and judging in a 20-iteration search, with a shared initial instruction and examples within each dataset.Strategies share iteration count, parent batch size and candidate count, but realized model-call counts differ, so this is not an equal-token or equal-rollout comparison.Source: §4.3 · p. 5

4.4 EVALUATION

  • Final and seed prompts are evaluated three times on the same held-out examples, and test performance does not influence prompt selection.Table 2 reports three-evaluation mean accuracy and initial-paired-output adverse and beneficial correction rates; calibration-minus-test gain and discovery iteration are descriptive, not causal estimates of overfitting or computational efficiency.Source: §4.4 · pp. 5–6, Table 2 · p. 6

5 RESULTS AND ANALYSIS

Across five datasets, reflection strategies trade off final test performance, local parent improvement, and calibration-to-test transfer. Results vary by task, and no single strategy dominates every comparison.

5.1 MAIN RESULTS

  • No-examples has the best parent-improvement rank (3.1) but the worst accuracy rank (8.0); Failures-only gains on mathematics do not generalize to GPQA, where mean accuracy falls by 3.1 points.The discrepancy between gain averages and within-dataset ranks reflects the large LiveBench-Math improvement; neither summary replaces per-dataset results.Source: §5.1 · p. 6
  • Failures-only has the highest average gain (8.0 points), while Balanced-mix and No-examples+Val share the best mean accuracy rank (4.0); no strategy wins on every task.Failures-only leads on LiveBench-Math and OlympiadBench, while other strategies lead HotpotQA and IFBench.Source: §5.1 · p. 6, Figure 2 · p. 7

5.2 COMPARISONS BETWEEN REFLECTION CHOICES

  • Balanced-mix exceeds Natural-mix by 3.8 points on average, but adding successes to Failures-only lowers the average by 0.5 points, with gains and losses varying by dataset.The comparison does not support a consistent benefit from balanced evidence or protection of solved cases.Source: §5.2 · p. 7
  • No-examples+Val improves the mean over No-examples on all five datasets, averaging 3.1 points, but the comparison changes both candidate selection and the local calibration gate.Alternating exceeds Natural-mix by 1.5 points on average; blind generation can be useful when paired with validation.Source: §5.2 · pp. 7–8

5.3 CALIBRATION TRANSFER

  • Across 45 runs, calibration gain and mean test gain correlate at r = 0.70; the gap is 7.7 points on IFBench and 6.5 on GPQA, but −2.7 on each mathematics dataset.The gap mixes split difficulty, selection effects, and evaluation variability; it is not a direct estimate of selection bias.Source: Table 4 · p. 8, §5.3 · p. 8

5.2 COMPARISONS BETWEEN REFLECTION CHOICES

  • Natural-mix exceeds LLM-judge by 1.5 points on average, with three strict wins, one tie, and one loss; empirical selection helps on average, not uniformly.LLM-judge measures only its chosen child for the local gate, rather than all five candidates.Source: §5.2 · p. 8
  • With-knowledge exceeds No-knowledge by 0.9 points on average, but task-specific gains and losses do not support a universal benefit from explicitly writing domain knowledge.Other data-grounded strategies use a neutral reflection template without an instruction to include or exclude domain knowledge.Source: §5.2 · p. 8

5.4 ROUNDS TO CALIBRATION IMPROVEMENT

  • Failures-only requires the fewest rounds per parent improvement (3.2) and best-score improvement (10.3); No-examples has the best parent rank (3.1), while Failures-only has the best best-score rank (3.8).Every strategy takes more rounds to improve the best prompt than its sampled parent; assigning 20 rounds to unsuccessful runs is a convention, not an observed waiting time.Source: Table 5 · p. 9, §5.4 · p. 9

6 CONCLUSION

Across five datasets, reflection strategies show task-dependent effects: Failures-only has the largest average gain, while Balanced-mix and No-examples+Val share the best average accuracy rank. The study also notes that using one model family for both proposing and answering limits generalizability.

6 CONCLUSION

  • Failures-only provides the largest average gain, driven by LiveBench-Math; Balanced-mix and No-examples+Val share the best average accuracy rank.Balanced evidence does not uniformly improve on failure-only evidence, and explicit knowledge retention helps HotpotQA but hurts some other tasks.Source: §6 · p. 9
  • Empirical selection with natural evidence outperforms the judge on average, but not on every dataset, at a different computational cost.Validation and gating improve blind generation most clearly on LiveBench-Math and IFBench.Source: §6 · p. 9
  • Using a single model family for both proposing and answering may make the observed strategy rankings model-specific and limits their generalizability.The study calls for evaluation across multiple model families and sizes, including different models for reflection and task execution.Source: §6 · p. 9

A PROMPTS

The displayed prompts normalize benchmark-name references outside the five evaluated datasets and preserve the original executed texts in machine-readable result files.

A PROMPTS

  • Benchmark names outside the five evaluated datasets are displayed as descriptive task names, while original executed texts remain unchanged in machine-readable result files.Unsupported non-Latin characters are displayed as Unicode code-point escapes.Source: A PROMPTS · p. 12

A.1 TASK DESCRIPTIONS AND SEED PROMPTS

The appendix describes evaluated task formats and their answer requirements, including competition mathematics, graduate-level science, multi-hop question answering, and instruction following. The task prompts also specify output constraints for some datasets.

A.1 TASK DESCRIPTIONS AND SEED PROMPTS

  • The LiveBench and OlympiadBench math prompts require answers in the formats specified by each question, with no text after the final answer.The supplied task descriptions distinguish closed-form answers, option letters, and zero-padded three-digit integers.Source: §A.1 · p. 12
  • Olympiad-level mathematics covers algebra, combinatorics, geometry, and number theory; each problem requires one exact answer, such as an integer, fraction, radical, or other closed-form expression.The task description identifies problems from national and international olympiads and requires a single correct final answer.Source: §A.1 · p. 12
  • Graduate-level science questions span biology, physics, and chemistry, with four options and a required single-letter answer.They are described as hard for non-experts even with web access and provide no supporting context.Source: §A.1 · p. 12
  • HotpotQA requires combining facts across multiple retrieved paragraphs and returning a short answer span.Inputs contain three paragraphs that may or may not include the needed facts, and the prompt specifies an exact final-line format.Source: §A.1 · p. 13
  • IFBench tests verifiable instruction following on held-out constraint types, with responses required to satisfy every explicit machine-checkable constraint.The task description says its accuracy is not the fraction of fully correct outputs.Source: §A.1 · p. 13

A.2 REFLECTION PROMPTS

The reflection prompts vary whether the reflector sees examples and whether it is instructed to retain domain knowledge. Other prompts ask it to reason from the task description alone or to choose among generated candidates.

A.2 REFLECTION PROMPTS

  • The templates distinguish reflection with examples from reflection without examples, with Alternating switching between those modes across calls.The even iterations use examples and the odd iterations do not.Source: §A.2 · pp. 13–14
  • Example-based reflection asks the model to read inputs, responses, and feedback, then write actionable guidance as a new instruction.The prompt template says to output only the new instruction inside a single code block.Source: §A.2 · pp. 13–14
  • No-knowledge reflection asks for task procedures and answer conventions while explicitly excluding subject-matter facts, formulas, definitions, mechanisms, examples, and answers.The instruction is intended to leave domain knowledge to the task model rather than encode it in the prompt.Source: §A.2 · p. 14
  • With-knowledge reflection instead asks the model to record subject-matter facts, formulas, definitions, mechanisms, and concrete pitfalls revealed by the examples.It shares the procedural guidance requested by the no-knowledge version.Source: §A.2 · p. 14
  • No-examples reflection reasons from the task description alone about likely errors and the instruction that could prevent them.Its prompt calls for task-specific guidance and provides no observations of actual model behavior.Source: §A.2 · pp. 14–15
  • The judge receives the task, current instruction, observed correct and incorrect behavior with feedback, and candidate replacement instructions, then selects one for unseen instances.The judge is instructed to choose based on expected behavior rather than length and to return only the candidate number.Source: §A.2 · p. 15

A.3 EVOLVED PROMPTS

The appendix presents evolved prompts across LiveBench-Math, OlympiadBench, GPQA, HotpotQA, and IFBench, combining task-specific guidance with answer-format or constraint-following instructions. It also notes cases where the seed prompt was retained because no evolved candidate beat it on calibration.

A.3 EVOLVED PROMPTS

  • Each run’s final prompt is the highest-calibration candidate after 20 iterations, evaluated on the test set; HotpotQA evaluation appends a fixed suffix.Source: §A.3 · p. 15

A.3.1 LIVEBENCH-MATH

  • Some variants address explicit answer-duplication requests, while advising that the boxed answer may retain the original single-letter response for automated grading.Source: §A.3.1 · p. 17
  • Other prompt instructions emphasize rigorous derivation, checking answers against options, and verifying digit padding and domain constraints.Source: §A.3.1 · pp. 17, 19
  • Prompts call for concise, relevant reasoning, removal of filler, and termination immediately after the final answer in the matched format.Source: §A.3.1 · p. 18
  • The prompts distinguish symbolic answers in boxed LaTeX, multiple-choice option letters, and integers formatted with zero-padding when required.Source: §A.3.1 · p. 18
  • They direct the model to parse explicit format constraints first, including whether instructions require suppressing all reasoning text.Source: §A.3.1 · p. 18
  • LiveBench-Math domain guidance covers Vieta-based root counting, logarithmic-domain and sign checks, and geometric means involving negative values.The prompts emphasize distinct root sets, valid intervals and boundary conditions, and whether a real geometric mean exists.Source: §A.3.1 · p. 19
  • LiveBench-Math prompts require identifying the exact requested answer format, then outputting only the formatted result without conversational text.Instructions include matching formats such as duplicated letters, zero-padded integers, or boxed answers.Source: §A.3.1 · pp. 19–20

A.3.2 OLYMPIADBENCH

  • OlympiadBench prompts favor concise, direct derivations using relevant theorems or structural insights, and require a final answer enclosed in a box.One prompt prioritizes identifying core geometric or algebraic properties and avoiding exhaustive cases unless needed.Source: §A.3.2 · pp. 20–21
  • Natural-mix and LLM-judge kept the seed prompt because no candidate beat it on the calibration set.The passages do not report a corresponding outcome for Balanced-mix or No-knowledge.Source: §A.3.2 · pp. 20, 22
  • Other OlympiadBench prompts stress checking constraints, edge cases, and case validity before giving a simplified exact answer in the final box.They also call for subfield-specific checks, such as verifying algebraic roots or number-theoretic consistency.Source: §A.3.2 · pp. 21–22

A.3.3 GPQA

  • GPQA prompts direct the model to derive scientific constraints or mechanisms, evaluate options against them, and end with a single option letter.The guidance includes comparing conditions and identifying why incorrect options conflict with derived facts.Source: §A.3.3 · p. 23
  • GPQA prompt variants request domain-specific, step-by-step scientific deduction, distractor analysis, and a final answer consisting only of the option letter.Source: §A.3.3 · p. 23
  • No GPQA candidate beat the seed on the calibration set, so the seed prompt was kept.Source: §A.3.3 · p. 23
  • One detailed GPQA protocol emphasizes conservation checks, rigorous calculations, verification against the four options, and a single-letter final output.Source: §A.3.3 · p. 24

A.3.4 HOTPOTQA

  • No candidate beat the seed on HotpotQA calibration across Failures-only, Balanced-mix, No-examples, No-examples+Val, Alternating, LLM-judge, No-knowledge, and With-knowledge.Source: §A.3.4 · pp. 24–25
  • HotpotQA prompt variants range from concise context-based extraction to reasoning chains that bridge missing attributes using internal knowledge.Source: §A.3.4 · p. 25
  • A HotpotQA knowledge-bridging protocol limits world knowledge to retrieving attributes of entities identified in the text and requires concise answers.Source: §A.3.4 · pp. 25–26

A.3.5 IFBENCH

  • The IFBench prompt begins by directing the model to follow explicit, machine-checkable input constraints while answering accurately.Source: §A.3.5 · p. 26
  • They build the response around the constraints, prescribing exact formats and explicit handling of lexical, sequential, and other mechanically checkable rules.Examples include constructing the required structure before adding content and counting coordinating conjunctions explicitly.Source: §A.3.5 · pp. 26–28
  • The prompts require the response to answer the core request accurately while explicitly tallying statistical ratios and meeting specified counts or placements.They also instruct the model to prioritize explicit mechanical requirements over conversational tone.Source: §A.3.5 · pp. 27, 30
  • Before output, the model is told to audit formatting, counts, tags, sequencing, and completeness against the prompt’s constraints.Some variants direct it to revise a failing segment or discard and revise a draft until the checks pass.Source: §A.3.5 · pp. 27, 29
  • When constraints conflict or make a full answer impossible, the prompts prescribe a concise conflict response or a correct constrained answer fragment, and warn against filler.They also caution against spending the token budget on repetitive text while omitting the core answer.Source: §A.3.5 · pp. 27, 29
  • The final response should contain only the requested content, without introductions, explanations of the checking process, or other meta-commentary.For raw formats such as JSON, one variant specifies that the format itself must constitute the entire output.Source: §A.3.5 · pp. 27–29
  • The prompts direct the model to identify explicit constraints and classify them by type, including structural, lexical, logical, numeric, and formatting requirements.One variant also represents each constraint as an internal boolean checklist item.Source: §A.3.5 · pp. 28–29
  • The prompts prescribe a workflow of constraint analysis, strategic synthesis, execution, and self-verification before finalizing the response.Checks include format, sequential or recursive patterns, lexical counts or ratios, and whether the core question has been answered.Source: §A.3.5 · p. 30
  • Drafting guidance says to integrate constraints into the response structure, preserve the core narrative when possible, and prefer concise, direct statements over meandering justifications.It advises prioritizing constraints over coherence when they conflict, while maintaining the narrative thread as much as possible.Source: §A.3.5 · p. 30
  • Execution guidance calls for reaching a definitive answer early, avoiding unnecessary elaboration or repetition, and stopping once the required output is complete.It also recommends mentally monitoring token usage under a strict generation budget.Source: §A.3.5 · pp. 30–31
  • Output conventions require providing the response directly, applying requested formatting throughout, and ending immediately after the final required element.The prompts prohibit introductory or concluding process commentary and advise resolving implied conclusions, such as a requested comparison.Source: §A.3.5 · pp. 30–31
  • IFBench instructions make explicit, machine-checkable constraints the primary requirement, asking assistants to inventory them by type and favor specific structural constraints when requirements conflict.One prompt says not to omit required elements; another advises stating a correction when constraints are physically impossible.Source: §A.3.5 · p. 31
  • The detailed procedure checks feasibility before drafting, then verifies requested symbols, list structure, punctuation, sequencing, and completeness in a final audit.For example, it says to use specified symbols exactly, ensure required sub-bullets, and calculate target sentence word counts when length depends on the prior sentence.Source: §A.3.5 · p. 31
  • Examples extend constraint handling to domain details: distinguish the interrobang symbol, preserve requested list and screenplay layouts, use standard factual definitions, and keep creative tone within structural limits.The prompts also state that explicit constraint failures can make a response incorrect regardless of the quality of its creative or factual content.Source: §A.3.5 · pp. 31–32

B AI USAGE DISCLOSURE

The authors used GPT to assist with manuscript writing and code development.

B AI USAGE DISCLOSURE

Loading 2609.32452v1…