Source-linked AI summary
PACEvolve++: Improving Test-time Learning for Evolutionary Search Agents
Minghao Yan, Bo Peng, Benjamin Coleman, Ziqi Chen, Zhouhang Xie, Shuo Chen, Zhankui He, Noveen Sachdeva, Weili Wang, Ed H. Chi, Shivaram Venkataraman, Wang-Cheng Kang, Derek Zhiyuan Cheng, Beidou Wang
TL;DR
PACEvolve++ targets the limited adaptation of fixed policies in evolutionary search, especially when evaluations are expensive. It trains an advisor to make strategic search decisions, delegates code realization to a stronger frontier model, and adapts credit assignment across search phases. Across multiple research and engineering tasks, it achieves stronger, faster, and more stable search performance, although the experiments could not be repeated or extended over longer horizons.
Problem
Fixed evolutionary-search policies do not internalize evaluation feedback into stable task-specific preferences, while expensive evaluations constrain feasible rollout groups.
Method
PACEvolve++ trains an advisor for hypothesis generation, novelty assessment, and selection, delegates implementation to a stronger frontier model, and shifts credit assignment from group-relative feedback to frontier contribution across search phases.
Results
Across expert-parallel load balancing, sequential recommendation, and protein fitness extrapolation, PACEvolve++ outperforms existing methods while converging faster and stabilizing test-time training.
Takeaways & Limitations
The results support improving the reasoning policy, rather than only the search scaffold, for scaling self-evolving agents to realistic machine-learning engineering settings.
Takeaways & Limitations
High RL-training and evolutionary-search costs prevented repeating the experiments or running them over a longer horizon.
Abstract
from arXiv · showhide
Large language models have become drivers of evolutionary search, but most systems rely on a fixed, prompt-elicited policy to sample next candidates. This limits adaptation in practical engineering and research tasks, where evaluations are expensive, and progress depends on learning task-specific search dynamics. We introduce PACEvolve++, an advisor-model reinforcement learning framework for test-time policy adaptation in evolutionary search agents. PACEvolve++ decouples strategic search decisions from implementation: a trainable advisor generates, assesses, and selects hypotheses, while a stronger frontier model translates selected hypotheses into executable candidates. To train the advisor under non-stationary feedback, we propose a phase-adaptive approach that adapts its optimization strategy to different phases of the evolutionary process. Early in evolution, it uses group-relative feedback to learn broad search preferences; later, as reward gaps compress, it emphasizes best-of-$k$ frontier contribution to support stable refinement. Across expert-parallel load balancing, sequential recommendation, and protein fitness extrapolation, PACEvolve++ outperforms the state-of-the-art evolutionary search framework with frontier models, achieving faster convergence and stabilizing test-time training during evolutionary search.
1 Introduction
PACEvolve++ addresses the limits of fixed evolutionary-search policies by separating strategic hypothesis decisions from code implementation and adapting reinforcement-learning credit assignment to search phases.
- Motivation: Fixed policies can use search history as context but do not internalize feedback into stable preferences for prioritizing promising directions.The challenge is adapting decisions to novelty, feasibility, and likely frontier improvement rather than merely generating plausible hypotheses.
- Advisor decomposition: PACEvolve++ trains an advisor for hypothesis generation, novelty assessment, and mutation selection while a stronger frontier model implements the selected hypothesis.This decouples search-specific decision learning from executable-code realization.
- Phase-adaptive learning: The framework adapts credit assignment from group-relative feedback during exploration to frontier contribution during refinement.The transition aligns policy learning with changing evolutionary search dynamics.
- Empirical results: PACEvolve++ demonstrates strong performance across expert-parallel load balancing, sequential recommendation, and protein fitness extrapolation, outperforming existing methods while converging faster.The comparison includes methods with and without reinforcement learning.
2 Background
Evolutionary search repeatedly proposes, evaluates, and selects program modifications, while recent systems improve either the search scaffold or the policy operating within that loop.
- Evolutionary search loop: An evolutionary search agent conditions on a current best program, evaluation metrics, and search history to generate candidate modifications and retain higher-scoring descendants.Candidates are added to the best-program set when they score highly.
- Search scaffolds: Search-scaffold methods improve context management, backtracking, collaboration, and persistent idea storage over long-horizon evolutionary trajectories.PACEvolve is described as combining hierarchical context management, momentum-based backtracking, island-style collaboration, and a persistent idea pool.
- Policy learning: ThetaEvolve trains a mutation policy with grouped candidate rewards, whereas TTT-Discover uses entropic reinforcement learning and state reuse to emphasize exceptional rollouts.The latter is suited to settings where one breakthrough branch matters more than average batch quality.
- Open setting: These methods show that evolutionary trajectories can provide test-time supervision, but expensive evaluators restrict feasible rollout groups and make sample efficiency important.The tasks may require domain-specific reasoning about architecture, optimization, and system trade-offs.
3 Method
PACEvolve++ combines an advisor-based search workflow with a phase-aware reinforcement-learning objective designed for expensive, non-stationary evolutionary optimization.
- Workflow: At each iteration, an advisor uses the parent program and search history to select a hypothesis, a frontier model converts it into code, and a scorer evaluates the result.Outcomes are incorporated into the evolutionary population before policy updates.
- Advisor decomposition: Separating reasoning from implementation addresses failures in which end-to-end rewards conflate idea quality with the ability to implement complex candidates.The advisor learns idea generation, novelty classification, and hypothesis selection while the frontier model writes code.
- Search dynamics: The adaptive reasoning layer learns dynamic search priors for exploration, architectural consolidation, and fine-grained refinement under fixed evaluation budgets.These priors concern which mutation families unlock new regions and which partially successful ideas merit revisiting.
- Evaluation constraints: Expensive evaluations make small rollout groups necessary, while prior methods often use 512 candidates per training step.The method targets robust test-time reinforcement learning while retaining evolutionary-search sample efficiency.
- Training dynamics: PACEvolve++ reaches the best final reward while avoiding instability patterns observed in baselines.Figure 2 reports training dynamics for DeepSeek-R1-0528-Qwen3-8B on Multi-Evolve.
- Exploration regime: Exploratory training uses group-relative advantages with asymmetric clipping so rare promising tokens can receive meaningful positive updates.This preserves dense within-group credit assignment without the late-stage variance blow-up described for compressed rewards.
- Phase-adaptive objective: Early search uses dense group-relative feedback, while late refinement emphasizes whether a candidate changes the best-of-k frontier as reward gaps compress.Frontier-contribution geometry remains informative and invariant to affine reward rescaling in late-stage refinement.
- Advantage computation: Each rollout group is standardized before mixing credit signals, making branches numerically comparable and allowing collapsed signals to be skipped.The mixture changes credit-assignment semantics rather than merely changing update scale.
4 Experiments
PACEvolve++ is evaluated across expert-parallel load balancing, sequential recommendation, and protein fitness extrapolation, with results showing faster convergence and stronger final rewards than comparison methods. Its phase-aware objective also produces smoother training dynamics.
- 4.1 Tasks and evaluation: PACEvolve++ is evaluated on model routing, sequential recommendation, and protein engineering tasks with task-specific evaluators and shared training settings.The tasks include expert-parallel load balancing, KuaiRec sequential recommendation, and protein fitness extrapolation.
- 4.1 Tasks and evaluation: The protein benchmark evaluates extrapolation from wild-type, single, and double mutants to mutants of order three or higher.Its combined score is 0.7×r+0.3×P@5, averaged across datasets.
- 4.2 Experimental comparison: The experiments compare advisor-training variants including ThetaEvolve, TTT-Discover, Max@k training, and a no-RL scaffold baseline.The setup isolates the effect of test-time advisor training within a long-horizon evolutionary search harness.
- 4.3 Results: PACEvolve++ consistently achieves the strongest final reward, smooth optimization, and fastest convergence across the three tasks.On EPLB it reaches a saturated near-optimal solution using half the evolution budget; on the other two tasks it reaches better solutions than baselines.
- 4.3 Results: PACEvolve++ avoids baseline instability by preserving exploration early and improving refinement later through phase-matched training signals.Its gradient norms remain comparatively narrow around 1 while entropy stays materially higher than Max@k.
5 Related work
Related work develops evolutionary search scaffolds, test-time training methods, policy-optimization objectives, and advisor-model separation. PACEvolve++ combines these directions by adapting a strategic advisor while delegating implementation to a stronger model.
- Evolutionary search agents: FunSearch, AlphaEvolve, and OpenEvolve established repeated proposal, evaluation, and selection as a foundation for LLM-driven evolutionary search.PACEvolve later emphasized long-horizon search organization through context and trajectory management.
- Test-time training: Test-time training methods have increasingly been integrated into evolutionary search because the process naturally generates on-policy reinforcement-learning data.ThetaEvolve is cited as training a mutation policy against the evolving program database.
- Policy optimization for LLMs: PPO, GRPO, DAPO, Dr. GRPO, Pass@k training, and PKPO represent related policy-optimization approaches for language-model reinforcement learning.These methods differ in baselines, clipping, normalization, entropy guidance, and pass@k objectives.
- Advisor models and small-model steering: Advisor-model approaches separate strategic guidance from execution, allowing a compact trained model to steer a stronger frozen model.PACEvolve++ adopts this separation for task-specific search priors and faithful implementation.
6 Conclusion
PACEvolve++ adapts advisor policies to task-specific search dynamics while separating high-level reasoning from implementation. Across machine-learning engineering tasks, it delivers stronger and more stable search performance.
- Conclusion: PACEvolve++ learns task-specific search priors under expensive evaluation regimes through advisor-style reinforcement learning.The framework is designed for self-evolving agents.
- Conclusion: Decoupling high-level reasoning from implementation and aligning optimization with search dynamics stabilizes training in practical research and engineering settings.The conclusion identifies reasoning-policy improvement as important for scaling self-evolving agents to realistic domains.
- Conclusion: PACEvolve++ achieves stronger and more stable search performance across diverse machine-learning engineering tasks.The conclusion frames this as evidence for improving the reasoning policy rather than only the search scaffold.
A Limitations
The experiments could not be repeated or extended to longer horizons because reinforcement-learning and evolutionary-search evaluations were costly and resources were limited.
- Limitations: The study could not repeat experiments or evaluate a longer evolutionary horizon because both RL training and candidate evaluation were expensive.Each candidate evaluation involved training a model, further increasing resource demands.
- Experimental setting: The experiments used default training configurations, prompt templates, and evolution setups from prior work on A2 instances in GCP.The reported setup was online and on-policy across n evolutionary search threads.
B.1 Task complexity
The selected tasks are complex to implement, creating a capacity challenge for small open-weight models and motivating separation of idea generation from code implementation.
- Complex implementation requirements make end-to-end evolution with 4B–8B open-weight models prone to low implementation correctness.The resulting reward can favor ideas that are valid when implemented correctly, and end-to-end ThetaEvolve-style RL becomes infeasible in this setting.
- PACEvolve++ addresses this capacity concern by separating idea generation from executable-code implementation.The study leaves broader improvements to small models’ general coding capability outside its scope.
C Task details
The evaluation spans expert-parallel load balancing, sequential recommendation, and combinatorial protein fitness prediction, with task-specific search spaces and diagnostics.
- EPLB: EPLB minimizes maximum per-device workload while keeping expert assignment fast from workload tensors.The evolvable code returns a device-assignment map, using workload profiles from a public expert-load dataset.
- Training diagnostics: The diagnostics include Multi-Evolve training dynamics for 8B models, where PACEvolve++ remains comparatively well conditioned on gradient norms and entropy.ThetaEvolve shows gradient-norm spikes, Max@k entropy collapse, and TTT-Discover repeated entropy collapses.
- KuaiRec: KuaiRec evolves sequence encoders and scoring logic for a sequential recommender trained and evaluated within a fixed scaffold.The dataset contains roughly 7,176 users, 10,728 items, and 12.5 million interactions.
- Multi-Evolve: Multi-Evolve predicts joint fitness effects for proteins with three or more substitutions using training data containing wild-type, single, and double mutants.The evolvable block includes mutation featurization, pairwise epistatic interaction terms, and regularization.
D.1 Training Diagnostics
Training diagnostics indicate that PACEvolve++ combines strong performance with more stable auxiliary dynamics than the compared RL variants.
- PACEvolve++ achieves the best performance among the evaluated RL algorithms while showing the most stable entropy and gradient-norm behavior.The diagnostics report the fewest unexpected increases or decreases in these auxiliary metrics.
- PACEvolve++ remains the most stable on auxiliary metrics in 4B Multi-Evolve training dynamics.
- PACEvolve++ remains comparatively well conditioned in 8B KuaiRec training, while ThetaEvolve spikes in gradient norm and Max@k collapses entropy.TTT-Discover training collapsed quickly in the reported dynamics.
- PACEvolve++ remains comparatively well conditioned in 4B EPLB training, unlike the gradient spikes, entropy collapse, and rapid training collapse reported for the baselines.
- Disaggregated metrics separate different Pareto fronts, so a higher combined score does not guarantee Pareto dominance.The reported trade-offs concern balance versus speed, ranking quality versus hit-rate coverage, and correlation versus top-ranked mutant precision.
F Training stability
The analysis characterizes how reward scaling affects group-relative and SLOO signals, showing why their credit-assignment roles differ across evolutionary phases. Standardized group-relative feedback is useful in diverse regimes, whereas SLOO preserves frontier ordering and becomes uninformative when reward variation collapses.
- Reward compression is modeled by positive scaling that brings rewards closer while preserving their ranking.
- Adding a constant shifts the mean without changing variance, while scaling by δ scales the standard deviation linearly.
- SLOO depends on winner-changing margins, so its raw signal scales linearly with δ while constant offsets cancel.
- When branch variation dominates ϵnum, scale conditioning bounds the signal; when variation falls below numerical noise, the branch becomes uninformative and updates are skipped.
- After scale conditioning, SLOO preserves candidate ordering by frontier contribution: a candidate receives positive credit exactly when it is the highest-reward element in a subset.
- The resulting objective uses dense centered group-relative credit for diverse early-stage groups and frontier-contribution credit for late-stage best-of-k survival.