Adaptive-GEPA: Make Your Harness Fit Heterogeneous Requests

1 INTRODUCTION

• Adaptive-GEPA learns the division of work and the programs together in one search, deciding when to create specialists, what they handle, and how to improve them.It matches specialists across branches by the validation requests they handled and inherits each description together with its program.Source: §1 · p. 2
• The method extends reflective GEPA search to a routed program library whose specialist responsibilities are described in short natural-language text alongside executable code.The reflection model identifies a needed change and edits the selected component.Source: §1 · p. 2
• At approximately 18,000 scored calls, the Qwen3-8B library scores 70.6 versus 62.5 for single-program evolution and 73.0 for independently optimized, label-routed programs.The four specialists learn different retrieval, reasoning, verification, and repair procedures.Source: §1 · pp. 2–3
• The experiments compare complete systems on a fixed mixture, without establishing seed robustness or isolating individual component effects.The work focuses on maintaining specialist identity during search and recombination amid other recent systems for joint optimization and heterogeneous routing.Source: §1 · p. 3

2 PROBLEM SETTING

• A compound AI system combines language-model modules with control flow and tool calls; full-program optimization can change these alongside modules and inference configuration.The paper represents the system as executable source while keeping underlying model weights fixed.Source: §2 · p. 3
• The routed program consists of a router and one or more complete specialist programs, with both program behavior and request allocation learned; the number of specialists need not equal the number of task distributions.An evaluator scores each output and may return textual feedback.Source: §2 · p. 3
• Requests come from a mixture of task distributions, but the system receives no task-family label for an incoming request.The request includes task instructions.Source: §2 · p. 3
• The search budget counts scored request executions and can be exceeded by the evaluations completed in the final started iteration.The budget constrains the search used to find the program, not internal model or tool calls in deployment.Source: §2 · p. 3

3 ADAPTIVE-GEPA

• Adaptive-GEPA edits the router, specialist descriptions and source code, and which specialists are active; local edits develop candidates while aligned merges combine specialists across branches.Its design makes these choices separate search targets even though full-program search could also implement internal dispatch.Source: §3 · p. 3

3.1 REPRESENTING THE DIVISION OF WORK

• The router reads the request and active specialist descriptions, not program source, and dispatches to an active specialist or fixed generic fallback.Specialist slots start inactive, can grow to five in the main experiments, and each specialist may run multiple model calls and tools.Source: §3.1 · pp. 3–4

3.2 CHOOSE AN ACTION, THEN EDIT ITS TARGET

• Reflection selects an edit based on the failure: routing or description changes redirect requests, code rewrites repair procedures, and activation supplies missing specialists.The reflection model sees usage, action history, and minibatch execution records but not specialist source; it reads target source only for program or description rewrites.Source: §3.2 · p. 4

3.3 ALIGN SPECIALISTS BEFORE MERGING BRANCHES

• Specialists are aligned across branches by overlap in the validation requests they handled, rather than by slot index or task labels.The local matching heuristic is neither a maximum-weight assignment nor a test of semantic equivalence; merges inherit descriptions with programs and choose between changed aligned pairs using archived mean scores on their shared requests.Source: §3.3 · pp. 4–5

3.4 SEARCH, ACCEPTANCE, AND FINAL SELECTION

• Candidate acceptance uses feedback-minibatch improvement for reflected children or a validation-subsample comparison for merged children, followed by full validation.After full evaluation of a merge, specialists receiving no routing traffic are deactivated; the implementation retains pre-cleanup scores.Source: §3.4 · p. 5

4.1 EVALUATION SETTING

• The primary score equally weights four task-family scores; the evaluation uses Qwen3-8B, GPT-5.5 reflection, and a nominal 18,000 scored-call budget.GEPA-FPA-ALL is the closest baseline; per-family GEPA uses ground-truth labels and is a reference, not a strict upper bound.Source: §4.1 · pp. 5–6

4.2 A JOINT SEARCH IMPROVES THE MIXED WORKLOAD

• 70.6 versus 62.5: Adaptive-GEPA’s family-mean test score exceeds GEPA-FPA-ALL’s, while the label-routed reference scores 73.0.At 5,018 scored calls, its validation mean is 67.5, above GEPA-FPA-ALL’s 65.6 and GRPO’s 60.0; these counts are not total compute.Source: Table 2 · p. 6, §4.2 · p. 6
• The advantage persists outside PUPA: the other three-task mean is 63.5 versus 60.7, and Adaptive-GEPA improves all four families over the common seed reference.PUPA contributes 6.0 of the 8.1-point Qwen advantage; excluded-family means are also higher for GPT-4.1, 65.0 versus 62.1.Source: §4.2 · pp. 6–7

4.3 RESULTS WITH GPT-4.1

• With GPT-4.1 execution and GPT-5.5 reflection, Adaptive-GEPA scores 71.5 versus 69.8 for GEPA-FPA-ALL and 74.9 for the label-routed reference.Its specialists match all 651 post-hoc task labels; resumptions and a judge outage limit this run’s evidential strength.Source: Table 3 · p. 7, §4.3 · p. 7

4.4 THE LEARNED DIVISION CHANGES HOW REQUESTS ARE EXECUTED

• The final router matches the post-hoc family mapping on all 651 test requests, but this measures partition agreement, not best-expert selection.Description and router edits changed request coverage while specialist programs stayed fixed; learned specialists also use distinct procedures.Source: Table 4 · p. 7, §4.4 · pp. 7–8, Figure 3 · p. 7

4.5 SEARCH COMBINES IMPROVEMENTS FROM DIFFERENT BRANCHES

• Search forms and recombines specialist roles: candidate 46 combines mathematics from 37 with routing, privacy, and rewriting from 31, reaching a validation mean of 73.05.In one accepted merge, request overlap aligned four roles and the child scored 72.0 on full validation versus 68.4 and 68.3 for its parents; accepted merges could still regress on individual families.Source: §4.5 · p. 8, Figure 4 · p. 8
• Across 26 accepted merges, ten retained active description–program pairs unique to each parent and all ten exceeded both parents’ full-validation means, with a median gain of 3.2 points.Fallback use fell from 36.7% to 25% after a router edit, then to zero after a slot-4 program edit; these rates measure allocation, not answer accuracy.Source: §4.5 · p. 8, Figure 5 · p. 8

4.6 HOW SEARCH DISTRIBUTES LOCAL EDITS

• 60 of 76 local proposals (78.9%) activated a specialist or rewrote a program, while the other 16 edited routing or descriptions.Rewriting slot 4 moved all 45 LiveBench-IF validation requests to it and raised its score from 61.9 to 76.4, but the changed procedure prevents attributing the gain to syntax repair alone.Source: Table 5 · p. 8, §4.6 · pp. 8–9

Reflective improvement

• GEPA edits prompts from execution traces and combines candidates successful on different examples; Adaptive-GEPA retains its reflection and per-request selection for routed libraries.Other cited approaches include Reflexion, Self-Refine, MIPRO, and TextGrad.Source: §5 · p. 9

Program and tool optimization

• ADAS, AFlow, AutoPDL, and JTPRO search programs, workflows, or tool schemas, while Adaptive-GEPA evolves specialist implementations and responsibilities.Its branch merges must establish specialist correspondence before inheriting programs.Source: §5 · p. 9

Heterogeneous requests and harnesses

• Prior heterogeneous-routing systems include Mixture-of-Prompts, Causal Prompt Optimization, and Adaptive Auto-Harness; this work studies a fixed mixture and aligns executed-request footprints across candidates.The authors identify component correspondence during search and recombination as their focus within joint optimization and routing.Source: §5 · p. 9

6 LIMITATIONS

• The final method is evaluated with one seed on four fixed task families, not online discovery, overlapping capabilities, or distribution shift.Some programs contain benchmark-specific branches, and the comparison changes both search representation and profile allocation, so component effects are not isolated.Source: §6 · p. 9
• Validation reuse for archive search and final selection may favor noisy high scores; its effect was not measured using a fresh selection set or repeated final-candidate evaluations.Researchers also saw periodic test results during development, although candidate selection used validation scores alone.Source: §6 · p. 9
• PUPA can reward empty delegated requests despite zero scores for runtime failures, and resumptions plus an evaluator outage make the GPT-4.1 run supporting evidence rather than a clean replication.The paper makes no claim about latency or total-compute savings.Source: §6 · p. 9

7 CONCLUSION

• Adaptive-GEPA jointly evolves a router, specialist descriptions, and programs, combining branches by aligning specialists on the requests they handled.It inherits each description together with its program.Source: §7 · p. 9

AI USE STATEMENT

• The revision used generative AI for method clarification, protocol review, result analysis, writing, literature checks, figures, tables, and analysis code; the authors take responsibility for the final paper.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

• The evaluation uses established benchmarks, but PUPA utility alone does not establish safe or useful delegation, and deployment needs separate privacy, usefulness, and security evaluation.Development-time test monitoring and other evaluation limitations are documented in Appendix L.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• Appendices A–D document search, information boundaries, data splits, evaluation rules, and baselines; Appendices I and L distinguish measured results from projections and record run limitations.Figures are generated from recorded experiment data.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 CANDIDATE SEARCH

• Candidate selection removes redundant candidates from per-example best sets, then samples in proportion to the number of sets each candidate remains in.Reflection uses feedback minibatches; if the parent scores perfectly on every minibatch example, reflection is skipped.Source: §A.1 · pp. 12–13

A.2 SPECIALIST ALIGNMENT AND INHERITANCE

• Specialists are aligned by overlap in the validation requests they actually executed; unmatched specialists may be omitted when capacity is full or no positive-overlap partner exists.Request overlap is an operational measure, not proof of semantic equivalence, and the method was not compared in a controlled full search with alternative matching rules.Source: §A.2 · p. 13
• Merging treats the router and each description–program pair as components, copying descriptions and programs together and resolving conflicts with shared-request scores when available.After validation, unused specialists are deactivated based on routing selections, without reevaluating the cleaned candidate; later routing could change when the description list changes.Source: §A.2 · pp. 13–14

B DETAILED OPTIMIZATION PROTOCOL

• Feedback and Pareto sets contain 45 examples per family, and each 24-example reflection minibatch contains six per family; family labels are removed from model inputs.Evaluator feedback differs by task, while invalid source is rejected or falls back and runtime exceptions receive zero before evaluator dispatch.Source: B DETAILED OPTIMIZATION PROTOCOL · p. 14

C DATASET DETAILS

• Training, evaluation, and test examples are mutually disjoint within each family, with HotpotQA and AIME using separate official or competition sources for their splits.HotpotQA uses 45 training questions and disjoint evaluation and test samples from the official dev split; AIME test problems come from AIME 2025.Source: C DATASET DETAILS · p. 14
• PUPA retains its positional 111/111/221 training, evaluation, and test partition, while LiveBench-IF uses 45 validation and 100 test prompts from November 2024 and feedback prompts from June 2024.LiveBench-IF feedback has no source-article overlap with validation or test; AIME's five attempts per test problem are averaged before aggregation.Source: C DATASET DETAILS · pp. 14–15

D BASELINE AND TRANSFER CONFIGURATIONS

• GEPA-FPA-ALL optimizes one full program on the mixed data, while the per-family reference separately optimizes four programs with oracle routing; both use a nominal aggregate 18,000-call budget.The oracle-routed comparison removes cross-family interference and routing error but is not a strict upper bound.Source: D BASELINE AND TRANSFER CONFIGURATIONS · p. 15
• MoP uses clustering, nearest-centroid routing, prompt induction, and assignment, but its 1,010 metric calls make it an algorithmic rather than compute-matched baseline.Its GPT-4.1 twin retains the Qwen clustering and router, with 606/651 routing accuracy.Source: D BASELINE AND TRANSFER CONFIGURATIONS · p. 15
• GRPO full-weight fine-tunes Qwen3-8B without ColBERT or a reflection model; its 18,000-call comparison matches scored-rollout opportunity, not GPU-hours or auxiliary-model calls.A runtime interface difference remains because DSPy may issue a second generation after a parse failure, unlike one-generation GRPO rollouts.Source: D BASELINE AND TRANSFER CONFIGURATIONS · p. 15
• The actor-transfer experiment repeats Adaptive-GEPA and both comparison systems with GPT-4.1 as actor, holding the GPT-5.5 reflection model fixed and comparing methods only within each actor.All methods use the same ChainOfThought seed and approximately 18,000-call aggregate budget in that setting.Source: D BASELINE AND TRANSFER CONFIGURATIONS · pp. 15–16

E REFLECTION INFORMATION BOUNDARIES

• The reflection information table concerns what each operation can see; the action selector receives no program source, and an edit exposes only its target slot's source when needed.Source: Table 6 · p. 16, E REFLECTION INFORMATION BOUNDARIES · p. 16

F FINAL CANDIDATE SUMMARY

• The archived final run selects candidate 74, whose four active programs occupy slots 1–4 and whose router distinguishes mathematics, privacy delegation, factual retrieval, and constrained transformation.Its priorities include privacy-preserving reformulation, supplied-text transformation, retrieval-grounded factual QA, exact quantitative reasoning, then generic fallback.Source: F FINAL CANDIDATE SUMMARY · p. 16

F.1 COMPARISON WITH THE SELECTED SINGLE PROGRAM

• The selected library and single program both implement multi-stage procedures, but the library's mathematics path can use multiple attempts and review while the single program uses one attempt after deterministic shortcuts.GEPA-FPA-ALL scores higher on LiveBench-IF; the complete-system comparison cannot isolate which design difference accounts for the score gap.Source: §F.1 · p. 16, Table 7 · p. 17

F.2 DESCRIPTION AND ROUTER EVOLUTION

• In one recorded trajectory, a retrieval description gained public-fact examples, then a router edit sent article-based constrained rewriting to the rewriting specialist, while all five programs stayed unchanged.After the router edit, two rewriting requests returned to fallback; the newly retrieved factual request scored lower, and both rewriting scores were unchanged.Source: §F.2 · p. 17

G ADDITIONAL TEST RESULTS

• The common Qwen3-8B seed family mean is 52.6081 and the selected GRPO checkpoint mean is 54.0367; an independent seed evaluation is 52.1220.The difference between seed evaluations reflects sampling variation for the same program.Source: G ADDITIONAL TEST RESULTS · p. 17

G.1 SENSITIVITY TO INDIVIDUAL TASK FAMILIES

• Table 8 reaggregates fixed selected candidates after excluding one family; excluding PUPA leaves 2.81 points for Qwen and 2.91 for GPT-4.1, while PUPA contributes 6.00 of Qwen's 8.11-point difference.These are reporting checks on fixed runs, not searches trained with fewer task families.Source: Table 8 · p. 18, §G.1 · p. 18
• The final-test exports lack per-request score vectors, so they cannot support paired confidence intervals or a breakdown of PUPA quality, leakage, and empty-request rates.The reaggregations also do not remove search randomness, development-time test exposure, profile differences, or GPT-4.1 run interruptions.Source: §G.1 · p. 18

G.2 GRPO WITH A LARGER TRAINING BUDGET

• Both the retrospective 90,000-call cutoff and full-run validation select GRPO step 180, but validation and test scores favor different checkpoints.The cutoff uses validation records through step 225 and reuses the existing test evaluation.Source: Table 9 · p. 19, §G.2 · p. 19

H RECORDED MERGES AND COMPONENT INHERITANCE

• Of 47 merge proposals, 26 pass the five-request check; ten accepted children retain an active description–program pair unique to each parent.Thirteen match a parent’s complete serialized fields, and three differ only in stored inactive-slot code; none differs only in the router prompt.Source: H RECORDED MERGES AND COMPONENT… · p. 19, Table 10 · p. 19
• Ten accepted combinations have higher recorded validation means than both parents, with a median difference of 3.20 points.They produce eight distinct serialized children; the observations show cross-branch component reuse, not that merging was counterfactually necessary.Source: H RECORDED MERGES AND COMPONENT… · p. 19, Table 10 · p. 19
• Among 12 repeated-source children without cleanup changes, six score above and six below the stronger parent, with a median difference of −0.05 points.Candidate 39 becomes identical only after cleanup, and some archived parent scores also precede cleanup.Source: H RECORDED MERGES AND COMPONENT… · pp. 20–21
• Footprint alignment matches specialists across different slot layouts, but the merged child’s PUPA score is lower than either parent’s.Candidate 64 scores 71.99 mean versus 68.43 and 68.29 for its parents, while PUPA is 80.74 versus 83.38 and 87.35.Source: Figure 7 · p. 20, Figure 8 · p. 20, H RECORDED MERGES AND COMPONENT… · p. 21

I.1 AN EARLIER VERSION WITH TWO SEEDS

• In both archived seed runs, predecessor v4 improves all three reported families over its own initial program, but PUPA is excluded from the reported mean.The earlier scoring rule could reward empty outputs after execution errors; the runs are not multi-seed validation of the final algorithm or its advantage over GEPA-FPA-ALL.Source: §I.1 · p. 21, Table 11 · p. 21

I.2 REPLAY OF HISTORICAL MERGES

• Of 120 historical merge events, only 21 are replayed; Jaccard alignment improves projected means in ten and restores one retrieval specialist lost by index matching.Among 20 events with complete projections under both strategies, none has a lower projected mean under Jaccard; these projections do not measure routing or stochastic execution changes.Source: §I.2 · pp. 21–22, Table 12 · p. 22
• The historical runs differ in seed or failure scoring, so their end-to-end differences cannot be attributed to merge strategy alone.The replay does not isolate alignment or conflict resolution contributions to final test scores.Source: §I.2 · pp. 21–22

J FULL OPTIMIZATION GRAPH

• The search graph contains 79 candidate evaluations but only 64 distinct serialized candidates; candidate 74 matches candidate 73’s components despite a higher measured validation score.The higher score reflects a new evaluation, not a structural change; a parent link records a search operation, not necessarily retained changed components.Source: J FULL OPTIMIZATION GRAPH · p. 22, Figure 9 · p. 23

• Adaptive-GEPA records 18,103 metric calls, 152 reflection completions, and 11,913,344 reflection tokens; these counters exclude failed attempts without reported usage.Complete actor-token totals are unavailable, so the recorded history counts cannot support a total-compute comparison.Source: K COMPUTE AND LOGGED USAGE · p. 23
• The shorter GRPO run uses 18,000 scored calls over 45 update steps and about five hours, while the longer run uses 93,240 calls over 235 updates and about 18 hours 46 minutes.The matched nominal call budget is not a match in FLOPs, tokens, or wall-clock time; GRPO makes no reflection-model calls.Source: K COMPUTE AND LOGGED USAGE · p. 23
• AIME test monitoring repeats all 30 questions five times with caching disabled, and the tables average attempts within each problem before reporting labeled means.The callback’s raw aggregate weights AIME as 150 evaluations.Source: K COMPUTE AND LOGGED USAGE · pp. 23–24
• In the GPT-4.1 actor-transfer run, Adaptive-GEPA has four active family-aligned slots and a 651/651 post-hoc routing audit, while four GEPA-FPA runs receive 4,500 calls each.The selected validation macros are 70.04 for Adaptive-GEPA and 70.86 for GEPA-FPA-ALL.Source: K COMPUTE AND LOGGED USAGE · p. 24

L VALIDITY AND EVALUATION LIMITATIONS

• The experiment keeps four task families and their proportions fixed, and does not test unsupervised task discovery, online adaptation, or unseen families.Source metadata helps the harness sample and choose evaluators but is withheld from the candidate, router, executor, and reflection model.Source: L VALIDITY AND EVALUATION LIMITATIONS · p. 24
• The final run uses one seed, and alignment and corrected failure scoring were introduced together, leaving their individual effects unisolated.Replay does not replace a prospective multi-seed ablation.Source: L VALIDITY AND EVALUATION LIMITATIONS · p. 24
• Final selection reuses the validation set after repeated candidate scoring, with no fresh selection set or repeated finalist evaluation to estimate selection optimism.The 180 validation requests were scored in 79 full-candidate evaluations for Adaptive-GEPA, plus 47 five-request merge checks; these are scoring invocations, not independent LM samples.Source: L VALIDITY AND EVALUATION LIMITATIONS · p. 24
• The PUPA metric can reward empty delegation requests, and judge overcounts can yield leakage ratios above one and negative utility scores.The final program implements explicit redaction, but the PUPA score alone has limited power to establish useful delegation.Source: L VALIDITY AND EVALUATION LIMITATIONS · p. 24
• The MoP reproduction tests transfer of its prompt-mixture recipe, not performance at matched compute, because it changes the embedding model and retains its native budget.It also applies to families without gold demonstration strings.Source: L VALIDITY AND EVALUATION LIMITATIONS · p. 24
• The GRPO comparison does not match wall-clock compute or auxiliary calls and does not establish that RL with an agentic tool interface is intrinsically weaker.GRPO full-weight trains without a reflection LM or retrieval tools, unlike Adaptive-GEPA’s remote reflection model and generated programs.Source: L VALIDITY AND EVALUATION LIMITATIONS · p. 24
• Test scores were not used by the optimizer, but test curves were visible during method development because a callback evaluated every fifth accepted candidate.Thus the test sets were not isolated from researcher decisions during development.Source: L VALIDITY AND EVALUATION LIMITATIONS · p. 25
• Programs include benchmark-specific templates and branches, and held-out evaluation tests new instances rather than new domains or problem structures.Both actors share the GPT-5.5 reflection endpoint; the results may depend on executor, tools, and reflection model, and profile choices alone establish no latency or deployment-cost reduction.Source: L VALIDITY AND EVALUATION LIMITATIONS · p. 25