PaperlayerView exact revision on arXiv

Source-linked AI summary

HERO: Harnessing Evidence for Test-time Refinement and Prompt Optimization in Program Generation

Mingyu Lu, Timothy Ossowski, Tristan Naumann, Hoifung Poon, Peniel Argaw

arXiv:2610.19184v2cs.CL

TL;DR

Program-generation harness updates must improve recurring failures without disrupting successful behavior, while test instances may still need their own execution-based corrections. HERO combines offline prompt optimization grounded in execution evidence with test-time refinement of the current program. Across four SQL, Cypher, and Python benchmarks, it outperforms the initial prompt and existing prompt-optimization baselines, with further gains from test-time refinement.

  • Problem

    Reusable prompt revisions can fix observed failures yet regress previously successful cases, while instance-specific execution errors may remain after cross-task optimization.

    Source: §1 · pp. 1–2

  • Method

    HERO learns reusable prompt updates from execution evidence across training tasks, validates revisions, and refines each test program using instance-specific evidence.

    Source: §1 · p. 2

  • Results

    Across four SQL, Cypher, and Python benchmarks, HERO consistently outperforms the initial prompt and existing prompt-optimization baselines, with test-time refinement adding gains on every benchmark.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The results support combining reusable cross-task optimization with instance-specific refinement for program generation.

    Source: §7 · p. 9

    Evaluation uses fixed benchmark-specific budgets, and task-rollout counts do not fully capture differences in reflection, candidate-generation, and selection costs.

    Source: Limitations · pp. 9–10

Abstract

from arXiv · show

Harness optimization improves LLMs by refining reusable prompts and scaffolding, yet requires further adaptation for program generation tasks such as text-to-SQL and text-to-Python. Existing methods often rely on LLM reflectors to propose harness updates, which can fix observed failures but also disrupt previously correct behavior. Text-to-code tasks offer a natural way to mitigate this issue: we can execute generated outputs, allowing us to evaluate and verify the effects of proposed updates directly. Building on this signal, we introduce HERO Harnessing Evidence for Refinement and Optimization), which combines offline and test-time harness optimization. At the first stage, HERO learns reusable prompt updates from execution-verified successes and failures and validates candidate revisions before adoption. At test time, we treat the generated code as an instance-specific harness and refine it using execution feedback. Across four SQL, Cypher, and Python benchmarks, HERO achieves the best offline performance, improving pass@1 by 7.6 points over the initial prompt, while test-time refinement adds another 2.0 points and improves execution efficiency.

1 Introduction

Program generation needs reusable harness updates that correct failures without disrupting successful behavior, plus instance-specific corrections grounded in each program’s execution. HERO combines offline optimization and test-time refinement to address both needs.

1 Introduction

  • Reusable updates must correct recurring errors without degrading behavior that already works, while current programs may need instance-specific feedback.Persistent shared context can carry unreliable or overly specific test-time feedback into later tasks.Source: §1 · pp. 1–2
  • HERO formulates program generation as offline prompt optimization coupled with test-time refinement of instance-specific programs.Reusable guidance is learned across tasks, while test-time adaptation is confined to the current program.Source: §1 · p. 2
  • HERO grounds offline prompt revisions in execution evidence from successful and failed programs, and validates candidate revisions before adoption.At test time, execution feedback and retrieved verified training examples guide revisions.Source: §1 · p. 2
  • Across diverse query-generation and programming tasks, test-time refinement further improves results, and learned prompts remain compact across tasks.The introduction also reports evaluation of execution efficiency and computational cost.Source: §1 · p. 2

2 Related Works

Prior work optimizes reusable prompts and harnesses or refines outputs using test-time feedback. HERO combines these directions for program generation, keeping its optimized prompt fixed during inference while refining the current program.

2 Related Works

  • Reusable prompt and harness methods optimize instructions, demonstrations, strategies, skills, or harness implementations around fixed LLMs.The section names MIPRO, GEPA, STRAGO, ACE, SkillOpt, and Meta-Harness as examples.Source: §2 · p. 2
  • Test-time methods use feedback from prior attempts or environments, with some approaches accumulating strategies across queries in persistent memory.The section discusses ReAct, Reflexion, self-refinement, and Dynamic Cheatsheet.Source: §2 · p. 2
  • HERO complements test-time adaptation with offline-learned reusable guidance, keeping the optimized prompt fixed while execution feedback and retrieved verified examples refine the current program.Its offline reflection also uses execution feedback from successful and failed programs generated under prior prompt revisions.Source: §2 · p. 2

3 Harness Optimization for Program Generation

The paper frames program generation as producing executable programs from natural-language specifications and treats harness optimization at shared and instance-specific scopes. The shared scope targets expected reward across instances; the instance scope targets the current instance.

3.1 Problem Setup: Program Generation

  • Program generation maps natural-language specifications to programs such as SQL queries, Cypher queries, and Python functions, whose execution yields observations.The setup distinguishes execution observations from access to test reference solutions or hidden correctness labels during refinement.Source: §3.1 · pp. 2–3

3.2 A Unified View of Harness Optimization

  • A harness comprises external state conditioning a fixed LLM policy or mediating its interaction with the environment, separated into shared and instance-specific state.Shared-state optimization targets expected reward across instances, whereas instance-specific optimization targets the current instance.Source: §3.2 · p. 3

4 Method

HERO learns a shared prompt from execution evidence across training tasks, then holds it fixed while refining each test instance’s program. Offline revisions are evidence-grounded and validated; test-time revisions draw on execution observations and verified demonstrations and pass an evidence gate.

4 Method

  • HERO has two complementary stages: offline optimization learns a shared harness, then test-time refinement adapts instance-specific state while keeping the shared harness fixed.The stages optimize harnesses at different scopes while the underlying LLM remains fixed.Source: §4 · p. 3

4.1 HERO-O: Offline Prompt Optimization

  • Offline optimization learns a reusable prompt across training tasks and separates diagnosis, revision acceptance, and final selection across three data splits.Examples used to propose an update are not used to determine its acceptance; a separate selection set chooses the final prompt.Source: §4.1 · pp. 3–4

4.1.1 Textual Gradients from Execution-grounded Experience

  • Execution-grounded reflection uses successful and failed programs and their outcomes to identify behaviors to preserve or revise, then summarizes evidence into feedback for prompt updates.Candidates are instructed to express reusable rules rather than instance-specific fixes, with filters against answer memorization.Source: §4.1.1 · p. 4

4.1.2 Adaptive Paired Acceptance

  • Candidate prompts are compared with the current prompt on paired instances; uncertainty-guided evaluation allocates more budget selectively, and adoption requires improvement above a margin.The paired improvement distinguishes regressions, unchanged correctness, and fixes; the final archive is selected separately using a selection set.Source: §4.1.2 · pp. 4–5

4.2 HERO-T: Evidence-Grounded Test-Time Refinement

  • Test-time refinement keeps the learned shared prompt fixed and adapts the instance-specific harness using evidence available at test time.The method does not use the test solution or correctness label for these steps.Source: §4.2 · p. 5

4.2.1 Two Sources of Refinement Evidence

  • Refinement combines execution traces, runtime errors, outputs, and environment probes with retrieved execution-verified training examples that supply implementation patterns.Demonstrations are retrieved separately for each test instance; the refiner selects at most three relevant examples from those retrieved.Source: §4.2.1 · p. 5

4.2.2 Evidence-gated Refinement

  • A proposed program revision is accepted only after passing programmatic checks and an LLM judgment that its changes are supported by accumulated evidence.Rejected revisions leave the current program unchanged; accepted revisions seed the next step.Source: §4.2.2 · p. 5

5 Experimental Setup

The experiments cover four text-to-function benchmarks across SQL, Cypher, and Python, comparing HERO with offline, test-time, and end-to-end harness methods under shared evaluation conditions. Evaluation measures execution accuracy, runtime efficiency, and whether accepted updates fix more cases than they break.

5.1 Benchmarks

  • Four benchmarks span text-to-SQL, text-to-Cypher, and text-to-Python generation, including realistic SQLite queries, Wikidata-derived property graphs, and practical library-use and data-science tasks.The data-science benchmark spans ten widely used libraries.Source: §5.1 · pp. 5–6
  • Diagnostic examples and execution feedback support prompt optimization; candidate revisions are evaluated on update sets and selected using separate selection sets where dataset size permits.Otherwise, the update set is reused for final harness selection.Source: §5.1 · p. 6

5.2 Models and Baselines

  • Baselines include offline prompt optimizers, test-time refinement methods, and Meta-Harness; test-time methods lack reference solutions, outputs, and correctness labels.Offline optimizers share a seed prompt, optimization data, and execution-based correctness signal.Source: §5.2 · p. 6
  • All methods use a fixed rollout budget based on GEPA’s auto=heavy setting, with GPT-5 at medium reasoning effort and results averaged over three seeds unless stated otherwise.Rollout counts are matched as closely as each procedure permits.Source: §5.2 · p. 6

5.3 Evaluation Metrics

  • Pass@1 is the primary execution-accuracy metric, measuring the fraction of instances for which one generated solution meets benchmark correctness criteria..Source: §5.3 · p. 6
  • Valid Efficiency Score (VES) measures runtime efficiency: incorrect solutions receive zero credit, while faster correct solutions score higher.It supplements accuracy by distinguishing correct programs with different execution costs.Source: §5.3 · p. 6
  • The cumulative fix-to-break ratio tracks incorrect-to-correct versus correct-to-incorrect transitions caused by accepted updates; values above 1 indicate more cumulative fixes than disruptions.The counts are accumulated over optimization progress.Source: §5.3 · p. 6

6 Results

HERO-O improves pass@1 across four code-generation benchmarks while keeping prompts compact, and HERO-T adds further gains through test-time refinement. Results also show more fixes than regressions during prompt optimization, with concurrent correctness and efficiency gains on three of four benchmarks.

6.2 Benchmark Performance of Offline Prompt Optimization

  • HERO-O improves pass@1 over the initial prompt on all four benchmarks: 4.16 points on BIRD-SQL, 4.66 on BigCodeBench, 10.1 on DSCodeBench, and 11.5 on CypherBench.It also achieves the highest mean pass@1 among offline prompt optimizers on all four benchmarks.Source: Table 1 · p. 7, §6.2 · p. 7

6.3 Test-Time Refinement Complements Offline Prompt Optimization

  • HERO-T adds an average 2.02 pass@1 points across the four benchmarks, with gains of 1.26 on BIRD-SQL, 1.20 on BigCodeBench, 1.20 on DSCodeBench, and 4.43 on CypherBench.The resulting system has the highest mean pass@1 on three of four benchmarks; ReAct is marginally higher on DSCodeBench.Source: Table 1 · p. 7, §6.3 · p. 8

6.1 Fixes and Regressions During Prompt Optimization

  • HERO-O maintains more fixes than regressions throughout optimization, reaching cumulative fix-to-break ratios of about 3.0 on BIRD-SQL and 2.2 on BigCodeBench.Using only failed cases lowers the average ratio from 2.23 to 1.91, indicating that successful and failed cases together provide more informative feedback.Source: Figure 2 · p. 7, §6.1 · p. 7

6.2 Benchmark Performance of Offline Prompt Optimization

  • On challenging BIRD-SQL questions, HERO-O scores 51.4 pass@1, 4.6 points above GEPA (46.8), its largest advantage across the three difficulty levels.HERO-O scores 70.5 on simple and 53.6 on moderate questions.Source: Figure 3 · p. 7, §6.2 · p. 7
  • HERO-O produces the shortest final prompts on average (1,949 tokens) and averages 2.79K tokens per optimization call, compared with ACE's 35,881-token prompts and 31.89K tokens per call.The learned prompts primarily retain general task-level instructions rather than accumulating instance-specific details.Source: Table 2 · p. 8, §6.2 · p. 8

6.4 Execution Efficiency

  • HERO-O has the highest mean VES on BIRD-SQL (91.9), BigCodeBench (75.0), and DSCodeBench (57.8), while HERO-T raises VES on those three benchmarks.On CypherBench, HERO-T raises pass@1 from 85.5 to 89.93 while VES falls from 109.8 to 101.7.Source: Table 3 · p. 8, §6.4 · pp. 8–9

6.5 Ablation Study and Sensitivity Analysis

  • Removing rendered success and failure cases drops pass@1 from 63.55 to 59.15 on BIRD-SQL and from 52.5 to 50.08 on BigCodeBench.The results indicate validation signals alone do not provide sufficient guidance for prompt revision.Source: Figure 4 · p. 9, §6.5 · p. 9
  • HERO remains relatively stable across tested configurations, stays above the seed prompt, and shows limited additional gains from test-time budgets beyond B=3.Experiments with GPT-5.4 also show gains over the evaluated baselines.Source: §6.5 · p. 9

7 Conclusion

HERO combines grounded offline prompt optimization with evidence-grounded test-time refinement. Across four SQL, Cypher, and Python benchmarks, it outperforms the initial prompt and existing prompt-optimization baselines, with further gains from test-time refinement.

7 Conclusion

  • Across four SQL, Cypher, and Python benchmarks, HERO outperforms the initial prompt and existing prompt-optimization baselines, and test-time refinement adds gains on every benchmark.The analyses also report reduced regressions during optimization and compact prompts that remain robust across settings.Source: §7 · p. 9
  • Execution-grounded updates reduce regressions during optimization, while learned prompts remain compact and robust across optimization settings.Source: §7 · p. 9
  • The results support combining reusable cross-task optimization with instance-specific refinement for program generation.Source: §7 · p. 9

Limitations

The evaluation uses fixed, benchmark-specific budgets, and task-rollout counts do not fully represent differences in method costs. Reproducibility also depends on proprietary model versions, and prompts may need re-optimization when the model changes.

Limitations

  • Fixed benchmark-specific budgets may limit gains that some methods could achieve with longer searches.Expanding budgets substantially is computationally demanding; ACE and Meta-Harness take approximately 12 and 24 hours, respectively, on BigCodeBench in this setup.Source: Limitations · p. 9
  • Task-rollout counts support controlled effort comparisons but do not fully capture reflection, candidate-generation, and selection costs.The authors sought fair and transparent comparisons within computational and methodological constraints.Source: Limitations · pp. 9–10
  • Reliance on proprietary GPT-5 and GPT-5.4 models means future updates or deprecation may affect exact reproducibility, and learned prompts may need re-optimization when the backbone changes.HERO operates at the harness level and can be applied to other compatible backbones.Source: Limitations · p. 10

A Datasets Details

The appendix describes four program-generation benchmarks, their optimization and evaluation splits, and their execution environments. Dataset-specific preprocessing and evaluation protocols determine which instances are used and how generated programs are scored.

A Datasets Details

  • All datasets use fixed splits and pool membership across methods and run seeds, with disjoint diagnosis, update, and applicable selection pools; evaluation data are reserved for final evaluation.Source: A Datasets Details · p. 12
  • BIRD-SQL evaluates realistic text-to-SQL over 95 SQLite databases, and this study uses TRAIN for optimization and DEV for evaluation, scoring 1,530 examples after exclusions.The concealed test set is not used; pools are partitioned at the database level to assess cross-database transfer.Source: A Datasets Details · p. 12
  • BigCodeBench-Instruct evaluates practical Python function generation; after excluding three tasks, the study uses 500 optimization tasks and 637 for frozen evaluation.Optimization feedback contains pass/fail results and exception types, while assertion messages with expected outputs are withheld.Source: A Datasets Details · pp. 12–13
  • CypherBench covers natural-language queries over 11 property graphs and 12 query patterns; its fixed question-level evaluation split can include graphs seen during optimization.The evaluation pool contains 1,000 questions; execution errors count as incorrect, and outputs are compared as multisets while allowing column permutations.Source: A Datasets Details · p. 13
  • DSCodeBench contains Python function-generation problems across ten libraries; after excluding 18 problems, 500 form the optimization set and 482 the evaluation set.Correctness requires matching reference outputs on all 200 seeded inputs under the benchmark comparator.Source: A Datasets Details · pp. 13–14

Software Environment

  • The evaluation environments use SQLite 3.53.4 for BIRD-SQL, Neo4j 5.20.0 for CypherBench, the official Docker image for BigCodeBench, and the pinned CPU environment for DSCodeBench.Source: Software Environment · p. 14

B Baseline Methods

The appendix distinguishes offline methods that change reusable shared harness components, test-time methods that adapt state during inference, and broader harness search. It also specifies HERO's execution-grounded prompt updates and instance-level refinement procedures.

B Baseline Methods

  • Baselines span shared-prompt optimization, test-time external-state adaptation, and Meta-Harness search over prompts and executable control logic.Methods generally share the initial prompt, task interface, and execution-based evaluator.Source: B Baseline Methods · p. 14, §B.1 · p. 14

B.1 Offline Prompt Optimization

  • ACE incrementally curates reusable strategies from execution trajectories, GEPA proposes targeted system-prompt revisions using evaluator feedback, and MIPROv2 searches instruction–demonstration combinations with Bayesian optimization.Source: §B.1 · p. 15

B.2 Test-Time Adaptation

  • Dynamic Cheatsheet's cumulative mode updates textual memory from problems and generated outputs, and that memory persists across evaluation examples.Its updates do not use reference solutions, correctness labels, or evaluator verdicts.Source: §B.2 · pp. 15–16
  • ReAct keeps the shared harness fixed and iteratively revises the current program using execution feedback from that instance.The adapted framework allows up to three environment interactions before final submission.Source: §B.2 · p. 16

B.3 End-to-End Harness Optimization

  • Meta-Harness searches candidate shared-harness implementations using prior candidates' source code, execution traces, and validation performance.Its search space includes prompting, memory, multi-call control flow, and self-verification while the policy remains fixed.Source: §B.3 · p. 16

B.4 Software environment.

  • The implementation uses GPT-5 snapshot 2025-08-07 for generation, prompt optimization, and test-time refinement, with a maximum of 16,384 completion tokens and default deployment settings.Source: C Implementation Details of HERO · p. 17

C.1 Offline Prompt Optimization

  • HERO's offline reflector uses correct and incorrect examples plus execution feedback to propose prompt revisions, which are compared with the current prompt and accepted above a minimum improvement threshold.Five rounds are run without early stopping, followed by adaptive selection from the prompt archive.Source: §C.1 · p. 17

C.2 Test-Time Refinement

  • HERO refines each evaluation instance from the offline prompt's initial output, using execution evidence and retrieved verified examples; revised candidates must execute successfully and have episode-supported evidence.The refiner has three diagnostic, execution, or retrieval actions; unsupported revisions are rejected and the initial candidate is retained when no supported improvement is found.Source: §C.2 · pp. 17–18

C.3 VES Details

VES combines execution-based correctness with relative runtime to measure efficiency, while HERO’s prompts remain compact despite optimization costs. The evaluation estimates runtimes with repeated, interleaved timing and benchmark-specific workloads.

C.3 VES Details

  • VES gives incorrect predictions zero reward and reflects both execution correctness and efficiency through the reference-to-prediction runtime ratio.Runtime estimates use repeated measurements, with interleaved reference and prediction timings and benchmark-specific exclusions for unreliable cases.Source: §C.3 · p. 18
  • BIRD-SQL efficiency evaluation uses 1,530 scored DEV questions, while BigCodeBench includes 440 of 637 TEST tasks using docstring examples for timing.The excluded BigCodeBench tasks had examples that failed on the canonical solution.Source: §C.3 · p. 18
  • CypherBench timing covers all 1,000 frozen TEST questions; DSCodeBench includes 453 of 482 TEST problems in its VES evaluation.CypherBench wall time includes network and serialization; DSCodeBench uses a fixed 200-input workload, excluding problems whose references cannot be timed reliably.Source: §C.3 · p. 18

D.1 Token Consumption and prompt length

  • HERO’s final prompts are 1.4–2.5K tokens, with the shortest prompt on DSCodeBench and lengths within 2× of the shortest elsewhere.Prompt lengths were stable across seeds, and the authors report that these lengths avoid substantially increasing shared input-token overhead at inference.Source: Table 5 · p. 18, §D.1 · p. 19

D.2 Hyperparameter Sensitivity

Sensitivity analyses test diagnostic evidence, offline search settings, and test-time refinement budgets. HERO’s default refinement budget captures much of the measured gain, while larger budgets add little computation.

D.2 Hyperparameter Sensitivity

  • Removing rendered diagnostic cases lowers mean pass@1 by 4.40 points on BIRD-SQL and 2.51 points on BigCodeBench; 32 failed and 32 passing cases perform best among tested counts.Counts from 8–64 per outcome remain above the seed prompt, but gains do not increase monotonically with more cases.Source: §D.2 · p. 19, Table 6 · p. 19
  • Combined successful and failed examples raise the mean fix-to-break ratio from 1.81 to 2.10, with improvements on three of four benchmarks.CypherBench rises from 1.41 to 2.06, while BigCodeBench slightly favors failure-only feedback, 1.71 versus 1.61.Source: Table 7 · p. 19, §D.2 · p. 20
  • Search settings with K=4–12 candidates and T=3–7 rounds score 61.52–63.55 on BIRD-SQL and 51.38–52.33 on BigCodeBench.All tested configurations exceed the seed prompt, but larger search breadth or depth does not consistently improve accuracy.Source: Table 6 · p. 19, §D.2 · p. 20
  • At B=3, refinement raises pass@1 from 62.53 to 63.59 on BIRD-SQL and from 52.59 to 53.79 on BigCodeBench.The best scores at larger budgets add only 0.46 and 0.27 percentage points, respectively.Source: §D.2 · p. 20, Table 8 · p. 21
  • Raising the refinement cap from B=3 to B=8 adds just 0.19 environment actions per query on each benchmark, while LLM calls rise by 0.10 on BIRD-SQL and 0.19 on BigCodeBench.At B=3, HERO also exceeds Re-Act’s mean pass@1 by 0.47 and 1.15 points, respectively, while using more actions and calls.Source: §D.2 · p. 20, Figure 6 · p. 20, Figure 7 · p. 21

D.3 Model Generalization

This section examines HERO’s generalization with GPT-5.4 across four benchmarks, including benchmark-dependent effects of test-time refinement, and describes the authors’ AI use and ethical concerns. It also presents the BIRD-SQL optimized prompt and its components.

D.3 Model Generalization

  • Offline optimization raised GPT-5.4 pass@1 over the seed prompt by 4.75, 0.21, 4.56, and 10.54 points on BIRD-SQL, BigCodeBench, DSCodeBench, and CypherBench, respectively.It achieved the highest offline pass@1 on DSCodeBench (55.39) and CypherBench (83.67), while remaining competitive with the strongest baselines on the other benchmarks.Source: §D.3 · p. 21
  • Test-time refinement improved GPT-5.4 pass@1 on BIRD-SQL (63.66 to 65.38) and CypherBench (83.67 to 88.63), but yielded +0.16 on BigCodeBench and -0.76 on DSCodeBench.It achieved the highest overall pass@1 on BIRD-SQL and CypherBench; on BigCodeBench, the optimized prompt with ReAct reached the highest performance (53.48).Source: §D.3 · p. 21

E AI Use Statement

  • The authors used generative AI for figure generation, editing, code generation, experiments, and statistical analysis, reviewed and tested AI-assisted code, and retained responsibility for the final work.They did not use generative AI to generate research ideas, hypotheses, scientific claims, theoretical or conceptual frameworks, mathematical claims or proofs, or synthetic datasets.Source: E AI Use Statement · p. 21

F Ethical Statement

  • HERO’s code-generation improvements could be misused for malicious operations or sensitive-data access, and deployment may raise privacy and security concerns; the authors encourage sandboxing, access controls, and human oversight.Source: F Ethical Statement · p. 22

G Prompt Templates

  • All optimization methods use the same benchmark-specific seed prompt, with task-specific inputs supplied separately in the user message.Source: Figure 8 · p. 23

G.1 Optimized Prompt for BIRD-SQL

  • The BIRD-SQL optimized prompt emphasizes planning, schema-grounded query construction, and verification, including exact requested output shape and minimal, key-based joins.Source: §G.1 · pp. 24–26, Figure 9 · p. 26
  • Dataset-specific guidance maps phrases to schema fields and join patterns, including credited status, school attributes, resource types, and app-category popularity.For app-category popularity, it aggregates over app_labels joined to label_categories and orders categories by app counts.Source: §G.1 · p. 27
  • The prompt directs the model to return one SQLite query in a SQL code block, select requested columns in order, and consult its playbook.The playbook contains strategies and insights from previous attempts at similar questions.Source: §G.1 · p. 28
  • The playbook recommends minimal predicates, avoiding join-induced row multiplication, and choosing COUNT(*) or COUNT(column) according to the counting task.It also distinguishes COUNT(DISTINCT key) for unique-entity counts across joins.Source: §G.1 · p. 28
Loading 2610.19184v2…