Source-linked AI summary
PACEvolve: Enabling Long-Horizon Progress-Aware Consistent Evolution
Minghao Yan, Bo Peng, Benjamin Coleman, Ziqi Chen, Zhouhang Xie, Shuo Chen, Zhankui He, Noveen Sachdeva, Isabella Ye, Weili Wang, Chi Wang, Ed H. Chi, Fernando Pereira, Wang-Cheng Kang, Derek Zhiyuan Cheng, Beidou Wang
TL;DR
LLM-in-the-loop evolutionary search lacks systematic process management and often suffers instability, stagnation, and weak collaboration. PACEvolve combines hierarchical context management, momentum-based backtracking, and self-adaptive collaborative sampling, achieving state-of-the-art results across diverse benchmarks and surpassing prior records on Modded NanoGPT.
Problem
LLM-in-the-loop evolutionary systems lack systematic process management and exhibit instability, stagnation, and ineffective knowledge transfer across parallel searches.
Source: §1 · pp. 1–2Method
Results
Takeaways & Limitations
PACEvolve offers a principled approach for developing robust LLM-in-the-loop evolutionary agents.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Large Language Models (LLMs) have emerged as powerful operators for evolutionary search, yet the design of efficient search scaffolds remains ad hoc. While promising, current LLM-in-the-loop systems lack a systematic approach to managing the evolutionary process. We identify three distinct failure modes: Context Pollution, where experiment history biases future candidate generation; Mode Collapse, where agents stagnate in local minima due to poor exploration-exploitation balance; and Weak Collaboration, where rigid crossover strategies fail to leverage parallel search trajectories effectively. We introduce Progress-Aware Consistent Evolution (PACEvolve), a framework designed to robustly govern the agent's context and search dynamics, to address these challenges. PACEvolve combines hierarchical context management (HCM) with pruning to address context pollution; momentum-based backtracking (MBB) to escape local minima; and a self-adaptive sampling policy that unifies backtracking and crossover for dynamic search coordination (CE), allowing agents to balance internal refinement with cross-trajectory collaboration. We demonstrate that PACEvolve provides a systematic path to consistent, long-horizon self-improvement, achieving state-of-the-art results on LLM-SR and KernelBench, while discovering solutions surpassing the record on Modded NanoGPT.
1. Introduction
PACEvolve is introduced as a systematic framework for stabilizing LLM-driven evolutionary search, which otherwise suffers from high variance and three core challenges: Context Pollution, Mode Collapse, and Weak Collaboration. It combines HCM, MBB, and self-adaptive CE, achieving state-of-the-art performance across LLM-SR, KernelBench, and Modded NanoGPT.
- LLM-in-the-loop evolutionary systems exhibit high variance because LLM and search stochasticity prevents reliable, consistent improvement.Although these systems can leverage evolution history for iterative refinement and knowledge-guided optimization, their scaffolds remain insufficiently systematic.Source: §1 · p. 1
- The identified failure modes are Context Pollution from failed histories, Mode Collapse from poor exploration–exploitation balance, and Weak Collaboration across parallel searches.These failures degrade idea quality, cause local-minimum stagnation, and limit adaptive knowledge transfer among concurrent processes.Source: §1 · pp. 1–2
- PACEvolve combines HCM for context pruning, MBB for escaping local minima, and self-adaptive CE for balancing backtracking with crossover.HCM decouples idea generation from selection; MBB provides a hard escape mechanism; CE coordinates internal exploration and external knowledge transfer.Source: §1 · p. 2
- PACEvolve achieves state-of-the-art performance on LLM-SR and KernelBench while surpassing prior records on Modded NanoGPT.The framework significantly outperforms existing methods across these diverse and complex benchmarks.Source: §1 · p. 2
2. Motivation
LLM-based evolutionary search can rewrite complete solutions using experimental context, but existing execution-and-reflection scaffolds remain sub-optimal because their histories bias generation, trap searches in local minima, and limit collaboration.
Motivation
- LLM-based search performs context-aware operations that can rewrite entire solutions from rich experimental histories, unlike traditional algorithms with fixed mutation and crossover operators.Source: §2 · p. 2
- Searches that fail to find low-NMSE solutions quickly often remain trapped in local minima, with substantial variance across independent runs.Source: §2 · pp. 2–3, Figure 3 · p. 3
Context Pollution
- Context Pollution saturates histories with failed trials, reinforcing flawed hypotheses, degrading signal-to-noise, and reducing diverse candidate generation.Source: §2 · p. 2
Mode Collapse
- Mode Collapse reflects poor exploration-exploitation balance: agents favor ideas resembling their context, exploit known directions, and insufficiently explore novel solutions.Source: §2 · p. 3
Weak Collaboration
- Weak Collaboration arises when multi-island searches use static periodic crossover that replaces weaker agents with top performers, preventing adaptive knowledge sharing.Source: §2 · p. 3
3. Progress-Aware Consistent Evolution
PACEvolve governs long-horizon evolutionary search through hierarchical context management, momentum-based backtracking, and self-adaptive collaborative evolution. Together, these components target context pollution, mode collapse, and weak collaboration while balancing internal refinement with cross-trajectory search.
3. Progress-Aware Consistent Evolution
- PACEvolve integrates HCM, MBB, and self-adaptive sampling to address context pollution, mode collapse, and weak collaboration in evolutionary search.The framework is designed to enable superior solution discovery and achieves state-of-the-art performance on challenging scientific and real-world engineering tasks.Source: §3 · p. 3
3.1. Hierarchical Context Management (HCM)
- HCM separates macro-level ideas from micro-level hypotheses, maintains a persistent idea pool, and uses bi-level pruning to preserve diversity while compressing or removing low-performing context.Pruned failures and rejected hypotheses remain in permanent memory so future candidates can avoid rediscovering known failures.Source: §3.1 · p. 4
3.2. Momentum-Based Backtracking (MBB)
- MBB measures scale-invariant Relative Progress and its momentum, triggering power-law backtracking and context reset when momentum falls below a stagnation threshold.Relative Progress tracks the fraction of the previous performance gap closed, while momentum provides an adaptive signal for escaping local minima.Source: §3.2 · pp. 4–5
3.3. Self-Adaptive Collaborative Evolution (CE)
- Self-adaptive collaborative evolution unifies backtracking and crossover, selecting actions probabilistically from island progress to favor promising knowledge transfer, dominant-agent backtracking, or crossover under shared high progress.The momentum-driven multi-island framework balances internal exploration with external exploitation to maximize overall search progress.Source: §3.3 · p. 5
4. Experiments
Experiments evaluate PACEvolve against evolutionary frameworks on Symbolic Regression and KernelBench, in complex Modded NanoGPT engineering, and through component ablations. Across these settings, PACEvolve improves search performance, discovers further NanoGPT optimizations, and shows complementary benefits from its components.
4.1. Evolutionary Framework Comparison
- The comparison evaluates scientific reasoning on synthetic oscillator equations and code optimization across diverse GPU kernels using established evolutionary-search benchmarks.The experiments compare PACEvolve with ShinkaEvolve, OpenEvolve, CodeEvolve, LLM-SR, and uDSR under repeated 1000-iteration searches for Symbolic Regression, while KernelBench tests 16 kernels across operators, layers, and models.Source: §4.1 · p. 5, §4.1.1 · p. 6, §4.1.2 · p. 6
4.1.1. SYMBOLIC REGRESSION
- PACEvolve-Single outperforms every baseline on best, worst, mean, and P75 Log10 NMSE for the Nonlinear Oscillator task.PACEvolve-Multi discovers three solutions with Log10 NMSE below -8 and significantly improves P75 and mean NMSE when at least one island performs well.Source: §4.1.1 · p. 6, Table 1 · p. 6
4.1.2. KERNEL BENCH
- PACEvolve-Multi outperforms PACEvolve-Single on 13/16 KernelBench kernels and matches or exceeds ShinkaEvolve, CodeEvolve, and OpenEvolve in 14/16, 14/16, and 15/16 cases, respectively.Both PACEvolve variants outperform the best existing kernels in all tested cases, while PACEvolve-Single beats ShinkaEvolve on every tested kernel.Source: §4.1.2 · p. 6, Table 2 · p. 7
4.2. Automated Engineering in Complex Environments
- PACEvolve reduces Modded NanoGPT training time from 142.8s to 140.2s through data sharding, skip-connection initialization, hyperparameter updates, and dynamic context scheduling.The benchmark is evaluated by reaching validation loss 3.28 on FineWeb using 8 H100 GPUs, extending optimization across architecture, kernels, data processing, and communication.Source: §4.2 · p. 7
4.3. Ablation studies
- Adding hierarchical context management improves mean and best performance, but momentum-based backtracking is needed to eliminate low-performing processes trapped by local minima.Context erasure removes low-performing ideas and histories but does not eliminate a best-performing idea when it is itself a local minimum.Source: §4.3 · p. 8
- Self-adaptive cross-island sampling preserves backtracking’s elimination of low-performing processes while significantly improving mean and P75 performance.It enables concurrent exploration of high-reward, low-NMSE regions when at least one island discovers a promising direction, though backtracking slightly reduces the best-process performance by increasing exploration.Source: §4.3 · p. 8
5. Related Works
LLM-driven evolutionary search replaces fixed mutation and crossover design with context-informed, flexible improvement proposals that have succeeded across mathematics, kernel optimization, and code optimization. Related agents improve context through reflection, summarization, or more examples, whereas PACEvolve emphasizes context management and dynamic search control, alongside broader iterative-search and long-context research.
- LLM-based evolutionary agents use background knowledge and past results to propose flexible solution improvements, enabling successes in discrete mathematics, kernel optimization, and code optimization.This paradigm moves beyond fixed mutation and crossover operators by giving LLMs a reasoning-driven role in each iteration.Source: §5 · p. 8
- AlphaEvolve and ShinkaEvolve improve evolutionary search by extracting insights from prior trials or summarizing attempts, while PACEvolve targets context management and dynamic search control.The paper characterizes PACEvolve as a principled response to current failure modes rather than merely another context-aggregation method.Source: §5 · p. 8
- Other related approaches improve LLM agents through iterative fine-tuning, tree- or rubric-based search, and methods for long-context reasoning, while PACEvolve identifies practices for evolutionary search design.These lines of work address diverse agent-improvement and reasoning settings and uncover phenomena related to those studied here.Source: §5 · p. 8
6. Conclusion
PACEvolve presents a principled evolutionary recipe for LLM-in-the-loop search, combining context management, backtracking, and collaborative sampling. Extensive evaluation shows consistent state-of-the-art performance across diverse, complex benchmarks.
- The framework is designed to address core challenges arising in LLM-assisted evolutionary search through a principled evolutionary recipe.Source: §6 · p. 8
- PACEvolve combines HCM for idea diversity and pruning ineffective histories, MBB for escaping local minima, and CE for self-adaptive multi-island coordination.Together, these modules address core challenges in managing evolutionary search dynamics.Source: §6 · pp. 8–9
- PACEvolve consistently achieves state-of-the-art performance across diverse, complex benchmarks.The evaluation supports its use as a principled approach for developing robust LLM-in-the-loop evolutionary agents.Source: §6 · p. 9
A. Failure Analysis … Insight 3: Dynamic cross-island synergy is key to fostering collaborative evolution
The paper diagnoses local-minimum trapping and context-induced repetition in LLM-SR evolution, then introduces context management, progress-aware backtracking, and adaptive cross-island coordination to promote diverse, collaborative search.
A. Failure Analysis
- In LLM-SR’s Nonlinear Oscillators benchmark, the agent searches for a symbolic f minimizing Normalized Mean Squared Error (NMSE).The target equation has the form ¨x + f(t, x, ˙x) = 0.Source: A. Failure Analysis · p. 13
- Vanilla evolution traps most searches in local minima because summarized histories reinforce flawed hypotheses despite negative results (Zhu et al., 2025).Source: A. Failure Analysis · p. 13
Insight 1: Hierarchical Context Management is key to enabling innovative thinking
- Hierarchical persistent idea memory separates idea generation from experiment selection and periodically summarizes or removes ineffective paths to preserve diversity and prevent context explosion.Source: Insight 1: Hierarchical Context… · p. 13
- Summarized experimental histories can condition later iterations toward similar ideas, making trajectories unlikely to discover better solutions when early progress is poor.Source: Insight 1: Hierarchical Context… · p. 13
Insight 2: Progress-aware regret is the key to escaping local minima
- Momentum-based backtracking prunes failed trajectories and reverts context to promising ancestors, removing detrimental conditioning to drive exploration beyond local minima.The paper reports that backtracking eliminates trajectories stuck in local minima throughout evolution.Source: Insight 2: Progress-aware regret is… · p. 13
Insight 3: Dynamic cross-island synergy is key to fostering collaborative evolution
- Static periodic crossover disrupts rapidly improving islands, while delayed crossover or backtracking wastes steps on stagnating islands, limiting coordinated progress.Source: Insight 3: Dynamic cross-island… · p. 13
- A self-adaptive sampling policy unifies backtracking and crossover, letting islands choose internal exploration or external knowledge transfer according to progress and momentum.This schedules knowledge exchange when it is most beneficial to collective search while preserving effective internal refinement.Source: Insight 3: Dynamic cross-island… · p. 13
B. Method Details … Idea Capping Prompt Template
PACEvolve’s method details organize persistent idea memory through hierarchical context management, separating conceptual generation from experimental selection while pruning histories and ideas to sustain exploration. Its prompt templates operationalize brainstorming, classification, selection, summarization, and idea capping throughout the evolutionary search.
B.1. Notation Table
- PACEvolve defines the mathematical concepts and notation used throughout its framework in a dedicated notation table.Table 3 summarizes all PACEvolve notations.Source: §B.1 · p. 13, Table 3 · p. 14
B.2.1. DECOUPLING GENERATION AND SELECTION
- The method decouples candidate generation into idea generation and idea selection, maintaining a persistent hierarchical pool that stores conceptual directions and experiment records.The LLM proposes and classifies ideas, refines matching entries or adds new ones, selects an idea, formulates a hypothesis, implements it, and appends summarized results to that idea.Source: §B.2.1 · p. 14
B.2.2. CONTEXT PRUNING
- Hierarchical Context Management caps hypotheses per idea and total ideas, summarizes older histories, prunes less promising concepts, and retains a separate full log to prevent repeated failed hypotheses.These operators constrain context growth while encouraging novel exploration; Algorithm 1 specifies the corresponding procedure.Source: §B.2.2 · p. 15
B.2.3. PROMPT TEMPLATES
- The prompt templates distinguish task-specific information from evolution-managed context and instruct the agent to use both during search.The templates mark replaceable task materials and managed evolutionary contexts.Source: §B.2.3 · p. 15, Idea Generation Prompt Template · p. 15
Idea Generation Prompt Template
- The idea-generation prompt requires three distinct options with rationale, expected impact, and experimental value while balancing exploration, tuning, compute limits, and prior results.It encourages high-risk candidates, avoids repeated configurations, and seeks a strong final candidate over roughly X attempts.Source: Idea Generation Prompt Template · p. 16
Idea Classfication Prompt Template
- The classification prompt merges ideas that share underlying concepts while preserving specific descriptions, and labels genuinely orthogonal or difficult-to-merge hypotheses as new ideas.Updated descriptions must remain concise and no more than two sentences.Source: Idea Classfication Prompt Template · pp. 16–17
Idea Selection Prompt Template
- The selection prompt requires choosing a valid idea-repository ID and stating a concrete experiment before proceeding to coding.The instructions prohibit hallucinated idea IDs while allowing experiment details to elaborate the selected idea.Source: Idea Selection Prompt Template · p. 17
History Summarization Prompt Template; Idea Capping Prompt Template
- The summarization and capping prompts retain the best trial and characterize successful or unsuccessful ideas, then drop ideas that are invalidated, exhausted, outdated, or unlikely to achieve breakthroughs.Idea removal is guided by experiment history, improvement potential, investigation depth, performance gaps, and the target idea cap.Source: History Summarization Prompt Template · p. 17, Idea Capping Prompt Template · p. 18
B.3. Action Weighting · 3. Synergy Bonus (wsyn
The self-adaptive crossover sampler assigns action weights from island progress, using crossover, backtracking, and synergy terms to balance exploration, exploitation, and stagnation avoidance. It combines these weights with momentum-based triggering and probabilistic action selection for collaborative evolution.
B.3. Action Weighting
- Action weighting balances competing search principles using the triggered island’s absolute progress A_i and the best partner’s progress A_best.A_best is the maximum progress among all partner islands.Source: §B.3 · p. 18
- Crossover weight w_Cj uses a partner’s direct performance gain, favoring islands with higher absolute progress.Source: §B.3 · p. 18
- Backtracking receives weight from dominance when the current island outperforms others and from stagnation when similarly progressing islands remain low-performing.Source: §B.3 · p. 18
- The stagnation component uses similarity S = max(0, 1 −|A_i − A_best|), becoming large when progress is both similar and low.Source: §B.3 · p. 18
3. Synergy Bonus (wsyn
- When two islands show similarly high progress, a synergy bonus increases the crossover weight of the best partner j_best.The bonus is high only when similarity and both islands’ progress are high.Source: §3 · p. 18
- Momentum-based backtracking and collaborative evolution trigger self-adaptive sampling when momentum falls below ϵ_rel, enabling exploration, exploitation, and stagnation avoidance.The procedure updates momentum from relative progress, computes island and neighbor progress, and applies the resulting action weights.Source: §3 · p. 19
- The sampler assembles backtracking and partner-specific crossover weights, then selects an action with probability proportional to those weights.Backtracking reverts to a previous state sampled by a power law, whereas crossover uses the partner selected by the sampled action.Source: §3 · p. 19
C. Experiment Details · C.1. Benchmark Task Selection
PACEvolve selects evolutionary-search benchmarks using four properties: complex solution spaces, limited reliance on LLM world knowledge, smooth reward landscapes, and low-cost evaluation. The supported tasks span algorithm design, combinatorial optimization, machine learning, kernel engineering, and deep learning, with caveats for KernelBench and Modded NanoGPT.
C.1. Benchmark Task Selection
- Benchmark tasks are selected for complex solution spaces, limited LLM world knowledge, smooth reward landscapes, and low-cost evaluation.Together, these properties support progressive search improvements, iterative exploration, effective use of evolutionary history, and repeated experiments for robust comparisons.Source: §C.1 · pp. 19–20
- The platform supports tasks spanning algorithm design, combinatorial optimization, machine learning, kernel engineering, and deep learning.Table 4 lists a subset of supported tasks and whether they satisfy the benchmark properties.Source: §C.1 · p. 20, Table 4 · p. 20
- The benchmark suite includes Consistent Hashing, Capset, DCN Optimizer Design, and Feature Cross tasks targeting algorithmic and machine-learning improvements.These tasks ask the LLM to improve consistent hashing and Capset constructions, design DCN optimizers, or create feature sets and crosses for Logistic Regression comparable to DCNv2.Source: §C.1 · p. 20
- The suite also includes Symbolic Regression, KernelBench, and Modded NanoGPT for equation fitting, deep-learning-kernel latency optimization, and distributed GPT2-style training latency.Modded NanoGPT targets validation loss in a setup using 8 H100 GPUs.Source: §C.1 · p. 20
- KernelBench and Modded NanoGPT have questionable LLM World Knowledge because their optimization strategies may overlap with LLM pretraining data.This remains a concern even though the benchmarks were created after the tested LLMs’ knowledge cutoff.Source: §C.1 · p. 21
- KernelBench also has a questionable Solution Space because some kernel speedups arise from one innovation while others combine multiple composable innovations.This makes the benchmark’s solution-space structure less consistent with the stated selection criteria.Source: §C.1 · p. 21
C.2. LLM-SR
The LLM-SR evaluation uses the first Nonlinear damped oscillator question from LLMSR Bench, with ten trials per run to characterize inter-run variability. Other benchmark problems share the format while varying primarily in perturbations to term combinations.
C.2. LLM-SR
- The evaluation uses the first Nonlinear damped oscillator question in LLMSR Bench.The ground-truth instance is −1.0267 ∗x ∗∗3 − 1.0267 ∗x ∗exp(−Abs(x)) + 0.9480 ∗sin(t) −0.7123 ∗sin(v).Source: §C.2 · p. 21
- Ten trials per run were conducted on this instance to establish a systematic understanding of inter-run variability.Source: §C.2 · p. 21
- Other LLMSR Bench problems use a similar format and primarily vary through perturbing term combinations.Source: §C.2 · p. 21
C.3. More KernelBench Results · C.3.1. KERNEL LIST
This section specifies the sampled KernelBench kernels, their difficulty levels, and problem indices, while describing the prompt used for idea generation and selection.
C.3.1. KERNEL LIST
- PACEvolve uses the GEPA prompt as the background and instruction section for idea generation and selection.This prompt is used during the procedure described in §B.2.3.Source: §C.3.1 · p. 21
- The evaluation uses a sampled list of KernelBench kernels with corresponding difficulty levels and problem indices.These kernels are catalogued in Table 5.Source: Table 5 · p. 21, §C.3.1 · p. 21
C.3.2. HEAD TO HEAD COMPARISON · D. Discovered Kernels
PACEvolve-Single and PACEvolve-Multi outperform the best existing KernelBench kernels in all tested cases, with Multi extending Single’s gains across most kernels. The following section also documents the kernels discovered by PACEvolve after removing comments and inline checks.
C.3.2. HEAD TO HEAD COMPARISON
- PACEvolve-Single and PACEvolve-Multi outperform the best existing kernels on KernelBench in all tested cases.Figure 5 presents the head-to-head win-rate comparison across methods.Source: §C.3.2 · p. 21, Figure 5 · p. 22
- PACEvolve-Single outperforms ShinkaEvolve in all tested kernels.Source: §C.3.2 · p. 21
- PACEvolve-Multi further outperforms PACEvolve-Single in 81.25% (13/16) of tested kernels.Source: §C.3.2 · p. 21
- Against ShinkaEvolve and CodeEvolve, PACEvolve-Multi finds equivalent or better kernels in 14/16 cases.Source: §C.3.2 · p. 21
- Against OpenEvolve, PACEvolve-Multi finds equivalent or better kernels in 15/16 cases.Source: §C.3.2 · p. 21
D. Discovered Kernels
- The discovered-kernels section lists kernels found by PACEvolve after removing PACEvolve-generated comments and inline checks.Source: D. Discovered Kernels · p. 22
D.1. BatchNorm
The BatchNorm implementation uses a custom CUDA extension with a vectorized FP16 computation kernel and exposes it through a PyTorch module. It applies per-channel affine normalization using running statistics, optimized memory loads, and fused half-precision arithmetic.
Kernel computation
- A custom CUDA kernel computes BatchNorm with per-channel gamma, beta, running mean, running variance, and epsilon parameters.The kernel derives the channel index from flattened spatial positions and computes inverse standard deviation, scale, and shift before transforming inputs.Source: §D.1 · pp. 22–23
Kernel optimization
- The kernel vectorizes four input values per thread, uses read-only cached parameter loads, and performs FP16 pairwise fused multiply-add operations before writing four outputs.Inputs are loaded as float4 values, converted to half2 pairs, processed with __hfma2, converted back, and stored as float4.Source: §D.1 · p. 23
PyTorch integration
- The PyTorch wrapper launches the kernel with 1024 threads per block, while ModelNew forwards inputs through the compiled extension using stored parameters and running-statistics buffers.The extension is compiled with -O3 and --use_fast_math, and the module initializes affine parameters, running statistics, and eps=1e-5.Source: §D.1 · pp. 23–24
D.2. Conv3d Divide Max GlobalAvgPool BiasAdd Sum … D.16. LayerNorm
This section presents a sequence of fused CUDA and PyTorch implementations for convolutional, pooling, normalization, neural-network, reduction, recurrent, and matrix operations. Across these components, the implementations use custom GPU kernels and execution strategies such as fusion, half-precision and vectorized computation, tiling, shared memory, warp-level reductions, and CUDA graphs.
D.2. Conv3d Divide Max GlobalAvgPool BiasAdd Sum
- D.2 implements a fused Conv3d–pooling–bias–sum operator using half-precision inputs, two-channel blocks, warp reductions, and CUDA graph replay for repeated shapes.The operator computes pooling-derived scaling and launches a monolithic 2D-block kernel with shared memory before returning a reduced output.Source: §D.2 · pp. 25, 26, 28, 29
D.3. Conv3d Max LogSumExp ReLU
- D.3 defines a fused Conv3d–max-pooling–bias–log-sum-exp–ReLU kernel that processes output tiles with shared-memory weights, inputs, and convolution results.The kernel uses fixed convolution and pooling parameters, maps blocks across spatial and depth dimensions, and handles padded input coordinates during tiled loading.Source: §D.3 · pp. 30–31
D.4. ConvTranspose2d BiasAdd Clamp Scaling Clamp Divide
- The kernel adds per-channel bias, clamps half-precision vector pairs between zero and an upper bound, uses channels-last data, and launches 256-thread blocks with fast-math compilation.Source: §D.4 · pp. 36–37
- ConvTranspose2d uses CUDA graph execution with static inputs and outputs for the fused bias-add, clamp, scaling, clamp, and divide operation.Source: §D.4 · p. 38
D.5. GELU
- GELU implements the approximation with fused multiply-add operations, tanh evaluation, float4 vectorization, grid-stride processing, and tail handling.Source: §D.5 · p. 39
D.6. Matmul with large K dimension
- Large-K matmul uses 64-element K tiling, shared-memory staging, float4 loads, register accumulation, and split-K atomic accumulation across 128 partitions.Source: §D.6 · pp. 41–42
D.7. Max pooling 2D
- The 2D max-pooling kernel processes 2×2 output tiles per thread, applies configured stride, padding, and dilation, and uses -FLT_MAX for invalid inputs.Source: §D.7 · pp. 44–45
- The max-pooling implementation exposes a grid-stride CUDA kernel through a compiled PyTorch extension and calls it from the module’s forward method.The extension is compiled with optimization and fast-math CUDA flags, and forward delegates directly to max_pool2d_grid_stride_cuda.Source: §D.7 · p. 46
D.8. MLP
- The MLP fuses linear computation and ReLU with float4 vectorization, four accumulators, and a 32-thread-by-warp launch configuration.The kernel handles vectorized input chunks, accumulates into sum_a through sum_d, adds bias, applies fmaxf, and launches over output tiles and batches.Source: §D.8 · pp. 47–48
D.9. RMSNorm
- RMSNorm dispatches to a cached hybrid CUDA extension with separate kernels for contiguous NCHW and channels-last NHWC inputs.The implementation validates CUDA float32 4D inputs, computes variance-based normalization, and rejects unsupported memory formats.Source: §D.9 · pp. 53–54
D.10. Softmax
- Softmax uses a single-pass online CUDA kernel with warp reductions, shared memory, float4 vectorization, and numerically stabilized max-sum accumulation.The kernel supports dimensions up to 8192 in a single pass and computes outputs as exponentials normalized by the block sum.Source: §D.10 · pp. 55–57
D.11. VGG16
- The VGG16 implementation defines half-precision float4 kernels for bias addition and fused bias-plus-ReLU operations.The kernels process vectorized input and bias data with unrolled half2 arithmetic, including a 4D NHWC fused-kernel variant.Source: §D.11 · pp. 59–60
- The kernels use half-precision float4 and half2 vectorization, unrolled channel processing, and integer indexing for NHWC tensor traversal and 2×2 pooling.Pooling computes four input locations, applies bias and ReLU to each, then reduces them with pairwise half2 maxima.Source: §D.11 · pp. 61–62
- The implementation provides CUDA operators for 4D NHWC tensors that fuse bias addition, ReLU, and max-pooling, with an optional flattening stage.Source: §D.11 · pp. 62–63
- PyTorch wrappers define fused linear, convolutional, convolution/ReLU, convolution/ReLU/max-pool, and convolution/ReLU/max-pool/flatten modules for the VGG16 architecture.The model assembles these modules across the VGG16 convolutional stages and classifier.Source: §D.11 · pp. 63–65
- The model runs in CUDA half precision with channels-last inputs and captures execution in a torch.cuda.CUDAGraph after warm-up iterations.The implementation initializes static inputs and outputs, converts inputs to channels_last, and uses a separate CUDA stream during graph capture.Source: §D.11 · pp. 65–66
- VGG16 uses CUDA stream synchronization, CUDA graphs, and a fixed-batch-size check in its forward path.Source: §D.11 · p. 66
D.12. Mean Reduction over a dimension
- The mean-reduction implementation supports contiguous and noncontiguous layouts with grid-stride kernels, warp or shared-memory aggregation, unrolling, and vectorized loads.It computes outer and inner sizes around the reduction dimension, handles singleton and empty outputs, and dispatches separate kernels by layout.Source: §D.12 · pp. 67–70
D.13. RNN
- The RNN CUDA kernel combines input and hidden features in shared memory, processes four hidden outputs per thread, reduces with warp shuffles, and applies approximate tanh.The implementation uses 128-element input-to-hidden and hidden-to-output tiles with 1024 threads per block.Source: §D.13 · pp. 71–72
- RNN uses separate feature-parallel CUDA kernels for input-to-hidden and hidden-to-output computation, with shared-memory staging, float2 vectorization, and warp shuffles.The implementation returns output and new_hidden.Source: §D.13 · pp. 73–75
D.14. BMM InstanceNorm Sum ResidualAdd Multiply
- D.14. BMM InstanceNorm Sum ResidualAdd Multiply fuses linear projection, InstanceNorm statistics and affine transformation, residual addition, and elementwise multiplication in one CUDA kernel.The implementation uses float4 vectorization, shared-memory intermediates, and warp reductions to compute the mean and inverse standard deviation.Source: §D.14 · pp. 77–79
- D.14. BMM InstanceNorm Sum ResidualAdd Multiply defines a fused linear-normalization-add-multiply CUDA operator with dimension checks and a divisibility requirement.The custom extension launches one block per batch element and requires out_features to be divisible by 4.Source: §D.14 · pp. 79–81
D.15. AlexNet
- D.15. AlexNet implements FP16 cuBLAS linear layers with fused bias, ReLU, dropout, and optional FP32 casting.The implementation uses half2 vectorization, unrolling, CUDA error checks, and tensor-core-compatible cuBLAS GEMM settings.Source: §D.15 · pp. 84–85
- D.15. AlexNet defines FP16 fused ReLU-max-pooling and linear/dropout operators, plus flatten-linear fusion and FP16-to-FP32 output conversion.The model uses channels-last FP16 inputs and combines these operators with AlexNet convolutional and classifier layers.Source: §D.15 · pp. 87–89
D.16. LayerNorm
- D.16. LayerNorm computes per-channel sums and squared sums with a 64-block-per-channel reduction, then applies normalized values using rsqrtf(var + 1e-5f).The implementation uses float4 vectorization, unrolling by 16, shared-memory reduction, atomic accumulation, and separate statistics and application kernels.Source: §D.16 · pp. 90–92
- LayerNorm requires 4D input in NCHW format and verifies that the channel dimension matches num_features.Invalid dimensionality or channel count raises a ValueError.Source: §D.16 · p. 93
- The input width must be divisible by 4 to satisfy the float4 optimization requirement.This constraint is checked before execution.Source: §D.16 · pp. 93–94
- After validation, LayerNorm executes the layer_norm_cuda kernel with the input, weight, and bias.The CUDA call returns the normalized output.Source: §D.16 · p. 94
Related papers
This paper cites
Cited by
- Trustworthy Recursive Self-Improvement in the Era of Large Models: Foundation, Risks, Evaluation and Open Challenges2026
- Verify Less, Evolve More: Training Idea-Level Critics for Verification-Efficient ML Evolving Agents2026
- AgentDiscover: Autonomous Discovery with Minimal Search Scaffolding2026
- Dream-RSI: Recursive Self-Improvement through Evolving Worlds2026
- AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces2026
- BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolution2026