PaperlayerView exact revision on arXiv

Source-linked AI summary

PACEvolve: Enabling Long-Horizon Progress-Aware Consistent Evolution

Minghao Yan, Bo Peng, Benjamin Coleman, Ziqi Chen, Zhouhang Xie, Shuo Chen, Zhankui He, Noveen Sachdeva, Isabella Ye, Weili Wang, Chi Wang, Ed H. Chi, Fernando Pereira, Wang-Cheng Kang, Derek Zhiyuan Cheng, Beidou Wang

arXiv:2601.10657v2cs.NEcs.LG

TL;DR

LLM-in-the-loop evolutionary search lacks a systematic way to manage context and search dynamics, leading to context pollution, mode collapse, and weak collaboration. PACEvolve addresses these problems with hierarchical context management, momentum-based backtracking, and self-adaptive collaborative sampling, and reports state-of-the-art performance across diverse benchmarks.

  • Problem

    LLM-in-the-loop evolutionary search lacks a systematic scaffold for managing its process, while existing systems suffer from instability and three identified failure modes.

  • Method

    PACEvolve combines hierarchical context management with pruning, momentum-based backtracking, and self-adaptive collaborative evolution to coordinate internal exploration and cross-trajectory knowledge transfer.

  • Results

    PACEvolve consistently achieves state-of-the-art performance across diverse benchmarks, including LLM-SR, KernelBench, and Modded NanoGPT.

  • Takeaways & Limitations

    The framework offers a principled approach for developing robust, long-horizon LLM-in-the-loop evolutionary agents.

Abstract

from arXiv · show

Large Language Models (LLMs) have emerged as powerful operators for evolutionary search, yet the design of efficient search scaffolds remains ad hoc. While promising, current LLM-in-the-loop systems lack a systematic approach to managing the evolutionary process. We identify three distinct failure modes: Context Pollution, where experiment history biases future candidate generation; Mode Collapse, where agents stagnate in local minima due to poor exploration-exploitation balance; and Weak Collaboration, where rigid crossover strategies fail to leverage parallel search trajectories effectively. We introduce Progress-Aware Consistent Evolution (PACEvolve), a framework designed to robustly govern the agent's context and search dynamics, to address these challenges. PACEvolve combines hierarchical context management (HCM) with pruning to address context pollution; momentum-based backtracking (MBB) to escape local minima; and a self-adaptive sampling policy that unifies backtracking and crossover for dynamic search coordination (CE), allowing agents to balance internal refinement with cross-trajectory collaboration. We demonstrate that PACEvolve provides a systematic path to consistent, long-horizon self-improvement, achieving state-of-the-art results on LLM-SR and KernelBench, while discovering solutions surpassing the record on Modded NanoGPT.

1. Introduction

PACEvolve addresses instability in LLM-driven evolutionary search with a systematic scaffold for managing context and search dynamics. It combines hierarchical context management, momentum-based backtracking, and adaptive collaboration, achieving state-of-the-art results across several benchmarks.

  • LLM-in-the-loop evolutionary systems often exhibit high variance and fail to produce reliable improvements because both the LLM and search process are stochastic.
  • The paper frames its central question as how to build an effective agent scaffold for LLM-driven evolutionary search.
  • PACEvolve uses Hierarchical Context Management to decouple idea generation from selection and prune ineffective histories, addressing context pollution.
  • Momentum-Based Backtracking combats mode collapse, while Self-adaptive Collaborative Evolution balances internal backtracking with external crossover across parallel processes.
  • PACEvolve achieves state-of-the-art performance, outperforming existing methods on LLM-SR and KernelBench and surpassing prior records on Modded NanoGPT.

2. Motivation

The motivation identifies three obstacles in LLM-assisted evolutionary search: polluted context, exploration–exploitation imbalance, and weak knowledge transfer among parallel processes. These issues can produce increasingly local candidates, stagnation, and inefficient collaboration.

  • LLM-based search replaces fixed evolutionary operators with context-aware rewriting driven by experimental history.
  • Existing evolutionary agents use an execution-and-reflection loop but remain sub-optimal on demanding symbolic-regression and kernel-design tasks.
  • Context Pollution: Context pollution saturates the agent’s history with failed attempts, reinforcing flawed hypotheses and discouraging diverse candidate generation.
  • Mode Collapse: Poor exploration–exploitation balance causes mode collapse, leaving agents in local minima and increasingly favoring candidates similar to their context.
  • Weak Collaboration: Static crossover patterns weaken collaboration because they cannot adaptively determine when parallel agents should transfer knowledge.

3. Progress-Aware Consistent Evolution

PACEvolve organizes evolutionary search around context control, progress-aware escape from stagnation, and adaptive coordination across parallel trajectories. Its components prune and structure search history, trigger backtracking from momentum signals, and choose between backtracking and crossover to support collaborative progress.

  • Framework overview: PACEvolve combines Hierarchical Context Management, Momentum-Based Backtracking, and Self-Adaptive Collaborative Evolution to address context pollution, mode collapse, and weak collaboration.The framework is presented as a systematic approach to governing context and search dynamics.
  • 3.1. Hierarchical Context Management: HCM separates abstract ideas from concrete hypotheses, decouples idea generation from selection, and maintains a persistent idea pool for structured context.Macro-level ideas capture global diversity, while micro-level hypotheses refine local details.
  • 3.1. Hierarchical Context Management: Bi-level pruning compresses histories for each idea and removes low-performing idea directions, while permanent memory records pruned failures and rejected hypotheses.These mechanisms reduce active context while helping the agent avoid rediscovering known failures.
  • 3.2. Momentum-Based Backtracking: Relative Progress measures the fraction of the previous performance gap closed by a new best score, making improvement measurement adaptive to optimization scale.When no improvement occurs, Relative Progress is set to 0.
  • 3.2. Momentum-Based Backtracking: MBB smooths Relative Progress with an exponentially weighted moving average and intervenes when momentum falls below a stagnation threshold.The agent then reverts to an earlier state sampled from a power-law distribution favoring earlier iterations.
  • 3.3. Self-Adaptive Collaborative Evolution: Self-Adaptive Collaborative Evolution unifies backtracking and crossover, selecting actions for stagnant islands according to their potential for global progress.The scheme compares island advancement using Absolute Progress and assigns action weights before sampling.

4. Experiments

PACEvolve is evaluated on established scientific and engineering benchmarks, open-ended engineering, and component ablations. It consistently outperforms comparison methods on LLM-SR and KernelBench, improves Modded NanoGPT, and shows complementary effects from its search components.

  • Evaluation setup: PACEvolve is evaluated against existing evolutionary frameworks on Symbolic Regression and KernelBench, then on complex engineering challenges and component ablations.The evaluation uses direct benchmark comparisons, an open-ended deployment, and an ablation study.
  • Symbolic Regression: PACEvolve-Single outperforms every baseline on LLM-SR for best, worst, mean log NMSE, and P75 log NMSE.PACEvolve-Multi discovers 3 solutions with log10 NMSE lower than -8 and improves P75 and mean NMSE.
  • KernelBench: PACEvolve improves kernels across difficulty levels and converges to higher speedup than other evolutionary search methods.The study evaluates 16 representative kernels spanning operators, layers, and models on a single A100 40GB GPU.
  • KernelBench: PACEvolve-Single and PACEvolve-Multi outperform the best existing kernels on KernelBench in all tested cases.PACEvolve-Multi further outperforms PACEvolve-Single in 81.25% (13/16) of kernels and matches or exceeds other frameworks in most comparisons.
  • Complex environments: PACEvolve reduces Modded NanoGPT training time from 142.8s to 140.2s through successive data-loading, initialization, hyperparameter, and context-scheduling changes.The benchmark was already reduced from approximately 2700 seconds to 142 seconds, so the reported additional gain is described as substantial.
  • Ablation studies: In ablations, hierarchical context erasure improves mean and best performance, momentum-based backtracking removes low-performing processes, and self-adaptive sampling improves mean and P75 performance.Backtracking slightly affects the best-performing processes by increasing exploration, while cross-island sampling preserves its process-level benefit.

5. Related Works

LLM-based evolutionary search replaces rigid classical operators with context-aware reasoning, but existing scaffolds remain ad hoc and often emphasize context aggregation rather than systematic context management and search control.

  • LLM-based evolutionary agents use background knowledge and past results to propose context-aware solution improvements instead of relying on fixed mutation and crossover operators.
  • Existing agents such as AlphaEvolve and ShinkaEvolve improve context quality through reflection, summarization, and additional in-context examples.
  • The broader literature also develops tree-based and rubric-based iterative search and fine-tuning methods for improving LLM-based agents.

6. Conclusion

PACEvolve presents a principled evolutionary recipe combining context management, backtracking, and adaptive multi-island coordination. Extensive evaluation reports consistent state-of-the-art performance across diverse, complex benchmarks.

  • PACEvolve combines HCM for promoting idea diversity and pruning ineffective histories, MBB for escaping local minima, and CE for self-adaptive multi-island coordination.
  • The framework is evaluated extensively on diverse and complex benchmarks to develop robust LLM-in-the-loop evolutionary agents.
  • PACEvolve consistently achieves state-of-the-art performance across the reported benchmark evaluation.

A. Failure Analysis

The failure analysis links stagnation in LLM-assisted evolutionary search to biased experimental context, weak exploration, and static coordination across islands. PACEvolve addresses these issues with persistent idea memory, backtracking, and progress-aware sampling.

  • Failure case: In the nonlinear oscillator task, the goal is to discover the differential equation governing motion by minimizing NMSE.
  • Context pollution: Vanilla searches that append summarized histories commonly become trapped in local minima, as LLMs may persist with flawed hypotheses despite negative results.
  • Context management: HCM separates idea generation from experiment selection and periodically summarizes or trims ineffective paths to preserve diversity and prevent context explosion.
  • Context pollution: Summarized histories can condition future iterations toward similar ideas, making trajectories unlikely to discover better solutions after initially poor progress.
  • Mode collapse: MBB provides a hard escape mechanism because context erasure alone cannot remove a best-performing idea that is a local minimum.
  • Weak collaboration: Static periodic crossover fails to balance internal search stability with external knowledge transfer across parallel islands.
  • Collaborative evolution: PACEvolve’s self-adaptive policy lets islands choose backtracking or crossover according to current progress and momentum.

B.1. Notation Table

The notation and implementation passages describe PACEvolve’s structured context as a persistent hierarchy of ideas and hypotheses, managed through generation, selection, summarization, pruning, and prompt-guided operations.

  • Notation: PACEvolve’s notation table summarizes the mathematical concepts used throughout the framework.
  • Idea memory: A persistent idea pool stores conceptual proposals, merges similar ideas, refines descriptions, and supports later selection of high-reward ideas.
  • Context pruning: Hypothesis histories are summarized after reaching a cap, while the idea pool is pruned to remove ineffective directions and encourage exploration.
  • Prompt templates: The prompt templates guide brainstorming, idea classification, experiment selection, code writing, history summarization, and idea dropping under an idea cap.

B.3. Action Weighting

The action-weighting mechanism assigns probabilities to backtracking and crossover using island progress, balancing exploration, exploitation, and stagnation avoidance.

  • Action weights balance backtracking against crossover according to the progress of the triggered island and its best available partner.The mechanism compares absolute progress across islands before sampling an action.
  • Crossover Weight: Crossover receives utility from the direct performance gain offered by a partner island, favoring islands with higher absolute progress.
  • Backtrack Weight: Backtracking combines dominance and low-progress stagnation components, becoming relevant when the current island dominates or similarly low progress persists.Similarity is defined as S = max(0, 1 −|Ai −Abest|).
  • The sampling policy selects among backtracking and crossover actions using weights proportional to their assigned utilities.
  • The resulting policy balances exploration through high-gain partners, exploitation through backtracking, and stagnation avoidance through progress-sensitive action selection.

C.1. Benchmark Task Selection

PACEvolve evaluates evolutionary search on tasks selected for properties that make progress measurable, comparisons robust, and repeated experiments feasible, while noting world-knowledge and solution-space caveats.

  • Benchmark tasks are selected using four desirable properties: complex solution spaces, limited LLM world knowledge, smooth reward landscapes, and low-cost evaluation.These properties support large-scale empirical studies of evolutionary search.
  • The supported task set spans algorithm design, combinatorial optimization, machine-learning kernel engineering, and deep-learning research.
  • The evaluated examples include feature-cross design, symbolic regression, KernelBench kernel optimization, and Modded NanoGPT training-latency optimization.
  • Caveats: KernelBench and Modded NanoGPT have questionable LLM World Knowledge because their optimization strategies may overlap with pretraining knowledge.
  • Caveats: KernelBench also has a questionable Solution Space because some speedups may derive from one innovation while others combine multiple innovations.

C.2. LLM-SR

The LLM-SR evaluation uses a nonlinear damped oscillator instance, with repeated trials to characterize variability in evolutionary search outcomes.

  • The study uses the first Nonlinear damped oscillator question from LLMSR Bench for symbolic regression.The task asks the LLM to recover an oscillator acceleration equation from synthetic data.
  • Ten trials per run are conducted on the instance to establish a systematic understanding of inter-run variability.
  • The ground-truth instance combines cubic, exponentially damped, and sinusoidal terms in x, t, and v.

C.3.1. KERNEL LIST

The KernelBench evaluation uses a sampled kernel list, with each kernel associated with a difficulty level and problem index.

  • Table 5 lists the kernels sampled for evaluation in KernelBench.
  • Each sampled kernel is associated with a difficulty level and problem index in KernelBench.
  • Idea generation and selection use the GEPA prompt as background and instruction context.

C.3.2. HEAD TO HEAD COMPARISON

PACEvolve’s single- and multi-island variants outperform existing KernelBench kernels across all tested cases. The multi-island variant also wins most head-to-head comparisons with other evolutionary frameworks.

  • PACEvolve-Single and PACEvolve-Multi outperform the best existing kernels on KernelBench in all tested cases.
  • PACEvolve-Multi further outperforms PACEvolve-Single in 81.25% (13/16) of tested kernels.
  • PACEvolve-Multi finds equivalent or better kernels than ShinkaEvolve and CodeEvolve in 14/16 cases and OpenEvolve in 15/16 cases.

D.1. BatchNorm

The BatchNorm implementation uses a custom CUDA extension for FP16-oriented, vectorized batch normalization. It loads channel parameters, processes four values per index, and exposes the kernel through a PyTorch module.

  • Each thread handles a four-element vector, derives the channel index, and loads gamma, beta, running mean, and running variance for that channel.
  • The kernel computes inv_std = rsqrtf(var + epsilon), then applies channel scaling and shifting before converting half-precision results back to float output.
  • The implementation defines a custom CUDA batch-normalization kernel and launches it through a PyTorch C++/CUDA extension.The extension exposes batch_norm_fp16_compute_cuda and returns the first tensor from the wrapper call.
  • The PyTorch module stores affine parameters and running statistics, sets eps to 1e-5, and delegates forward computation to the compiled CUDA implementation.

D.2. Conv3d Divide Max GlobalAvgPool BiasAdd Sum

This implementation fuses a 3D convolution and downstream reduction operations into custom CUDA kernels. It uses half precision, tiled execution, shared-memory reduction, and CUDA graph replay for repeated compatible inputs.

  • The kernel processes channel pairs with a 2D block, shared half-precision storage, and warp-level reduction support.
  • The launch configuration computes output dimensions from pooling factors, allocates shared memory for warp reductions, and dispatches one block per channel pair.
  • For unchanged input shapes, the module captures and replays a CUDA graph, while shape changes trigger graph-state reset and recapture.
  • The PyTorch wrapper computes an inverse scaling factor from the number of pooled outputs and divisor before invoking the fused operation.
  • The fused CUDA kernel combines convolution, max pooling, bias handling, log-sum-exp, and ReLU-related computation in one kernel.
Loading 2601.10657v2…