ParEVO: Synthesizing Code for Irregular Data: High-Performance Parallelism through Agentic Evolution

Liu Yang; Zeyu Nie; Andrew Liu; Felix Zou; Deniz Altinbüken; Amir Yazdanbakhsh; Quanquan C. Liu

Source-linked AI summary

ParEVO: Synthesizing Code for Irregular Data: High-Performance Parallelism through Agentic Evolution

Liu Yang, Zeyu Nie, Andrew Liu, Felix Zou, Deniz Altinbüken, Amir Yazdanbakhsh, Quanquan C. Liu

arXiv:2603.02510v2Mar 3, 2026cs.LGcs.DCcs.NEcs.PF

TL;DR

Irregular parallel programming is difficult for LLMs because dynamic work and dependencies expose weaknesses in sequentially biased code generation. ParEVO addresses this with verified ParlayLib-centered training, specialized fine-tuned models, and an evolutionary coding agent that uses execution feedback. The framework achieves state-of-the-art ParEval results and surpasses commercial LLMs and traditional heuristics.

Problem
Irregular parallelism remains difficult because dynamic work and dependencies challenge current LLMs and make safe, efficient code synthesis difficult.
Method
ParEVO combines a verified parallel-code corpus, fine-tuned models aligned with ParlayLib, and evolutionary code refinement guided by compiler, test, and performance feedback.
Results
ParEVO achieves state-of-the-art results on ParEval, surpassing commercial LLMs and traditional heuristics.
Takeaways & Limitations
The results support moving beyond syntax generation toward systems that jointly reason about scalability, correctness, and hardware interactions.
Takeaways & Limitations
ParEVO is optimized for shared-memory multicore architectures and does not address distributed-memory systems such as MPI or PGAS.

Abstract

from arXiv · show

The transition from sequential to parallel computing is essential for modern high-performance applications but is hindered by the steep learning curve of concurrent programming. This challenge is magnified for irregular data structures (such as sparse graphs, unbalanced trees, and non-uniform meshes) where static scheduling fails and data dependencies are unpredictable. Current Large Language Models (LLMs) often fail catastrophically on these tasks, generating code plagued by subtle race conditions, deadlocks, and sub-optimal scaling. We bridge this gap with ParEVO, a framework designed to synthesize high-performance parallel algorithms for irregular data. Our contributions include: (1) The Parlay-Instruct Corpus, a curated dataset of 13,820 tasks synthesized via a "Critic-Refine" pipeline that explicitly filters for empirically performant algorithms that effectively utilize Work-Span parallel primitives; (2) specialized DeepSeek, Qwen, and Gemini models fine-tuned to align probabilistic generation with the rigorous semantics of the ParlayLib library; and (3) an Evolutionary Coding Agent (ECA) that improves the "last mile" of correctness by iteratively repairing code using feedback from compilers, dynamic race detectors, and performance profilers. On the ParEval benchmark, ParEVO achieves an average 106x speedup (with a maximum of 1103x) across the suite, and a robust 13.6x speedup specifically on complex irregular graph problems, outperforming state-of-the-art commercial models. Furthermore, our evolutionary approach matches state-of-the-art expert human baselines, achieving up to a 4.1x speedup on specific highly-irregular kernels. Source code and datasets are available at https://github.com/WildAlg/ParEVO.

1. Introduction

ParEVO targets irregular parallelism, where unpredictable work and dependencies make efficient, safe code difficult for current LLMs. It combines verified ParlayLib-centered data and models with evolutionary refinement to improve synthesized parallel algorithms.

Motivation: Irregular algorithms have unpredictable memory access and dynamic work distribution, making static load balancing ineffective.Efficient implementations require techniques such as work-stealing, dynamic scheduling, and lock-free synchronization.
Motivation: Current LLMs exhibit sequential bias, often introducing races through naive parallelization or serialization through coarse-grained locks.These failures can make generated parallel code incorrect or slower than its sequential counterpart.
Contributions: Parlay-Instruct contains 13,820 parallel coding tasks synthesized through a Teacher-Student pipeline and verified against a compiler oracle.The corpus also includes graph-focused performance data curated from DMOJ competitions.
Contributions: Fine-tuned DeepSeek, Qwen, Qwen-Rust, and Gemini models internalize ParlayLib semantics, primitives, algorithms, and safe parallel Rust patterns.The released models include a 6.7B-parameter DeepSeek model and two 30B-parameter Qwen3 models.
Contributions: ParEVO formalizes data synthesis and code generation as evolutionary searches over abstract syntax trees, using compilation, performance tests, logs, and profiles to guide mutation and crossover.The Evolutionary Coding Agent iteratively repairs candidate solutions using evaluation feedback.
Contributions: Fine-tuning creates an alignment tax: models achieve higher Pass@1 rates but lower Speedup@1 rates by favoring safer high-level primitives over risky atomics.The reported trade-off reflects conservative use of stable primitives such as parlay::unique.

2. Related Work

Prior work covers code generation, automated parallelization, evolutionary coding, and high-level parallel abstractions, but ParEVO combines execution-based correctness feedback with performance-guided synthesis for irregular data.

LLMs for Code Generation: Existing code-generation models perform well on sequential tasks but struggle with complex planning and reasoning, especially in HPC settings.Related evaluation has moved beyond surface overlap toward structure-aware measures such as CodeBLEU.
Automated Parallelization and HPC Translation: Automated parallelization has largely targeted serial-loop translation and pragma prediction for regular workloads, while some approaches lack correctness feedback.Examples include OpenMP- and CUDA-oriented systems such as BabelTower, OMPGPT, AutoPar, and UniPar.
Automated Parallelization and HPC Translation: MuSL co-evolves a translator and test generator to filter parallel translations, whereas ParEVO uses an evolutionary feedback loop for synthesized irregular parallel code.The comparison highlights correctness-oriented feedback as a differentiating design dimension.
Structured Reasoning and Agentic Coding: Evolutionary and reflective coding frameworks iteratively correct generation failures, with EvoTune additionally updating its model from search-derived reinforcement signals.These systems motivate ParEVO’s use of search beyond single-shot generation.
Novelty: ParEVO replaces ordinary unit-test fitness with rigorous hardware profiling and problem-specific tests designed to expose races and performance degradation.Its execution feedback uses scalable runtimes and human-expert tests as selection pressure.
Abstractions for Irregular Parallelism: ParlayLib exposes work-span theory through composable primitives such as scan, reduce, and filter, making efficient irregular algorithms more accessible.This abstraction contrasts with approaches focused on lower-level threading or regular matrices.

3. Methodology: The ParEVO System

ParEVO combines evolutionary data synthesis, supervised fine-tuning, and inference-time evolutionary search to generate and optimize parallel code using correctness and runtime feedback.

System Overview: ParEVO has three stages: evolutionary data synthesis, supervised fine-tuning, and inference-time evolutionary search.The stages connect training data construction with model alignment and final code optimization.
Data Synthesis: The synthetic corpus combines ParlayLib primitives, DMOJ slow-fast code, and algorithmic examples generated through a Teacher-Student-Critic pipeline.The process begins from 593 manually authored golden examples and applies type, constraint, and algorithmic mutations.
Data Synthesis: 29,700 initial candidates were generated through five passes of ten mutations per seed plus 50 complex-primitive tasks.Mutation operators include changing data types, adding logical predicates, and transforming algorithm structures.
Data Filtering: 13,820 verified instruction-tuning pairs remained after compilation and unit-test filtering discarded 15,880 candidates.The final dataset contains 13,120 training pairs and 700 held-out test pairs.
Performance Dataset: The performance dataset contains 20 challenging DMOJ graph problems with evolutionary solution trajectories generated using a dual objective for test failures and execution time.Optimization pairs were retained when the optimized solution achieved at least 1.2× runtime speedup over the baseline.
Model Training: DeepSeek-6.7B and Qwen3-Coder-30B were selected as open-source backbones, while Gemini-2.5-Pro supported long-context scenarios; LoRA reduced fine-tuning compute costs.The larger Qwen model used a two-stage SFT and DPO alignment pipeline.
Evolutionary Coding Agent: The evolutionary agent maintains diverse candidate populations annotated with performance metrics and diagnostic artifacts, then iteratively refines them in program space.Evaluation used a dual-socket 64-core node with 32 threads for OpenMP, ParlayLib, and Rust unless otherwise specified.
Evolutionary Coding Agent: The framework overview combines human expert context with evolutionary LLM refinement, correctness verification, stress testing, profiling, and MAP-Elites population selection.The fitness function assigns zero to candidates failing compilation or tests and otherwise incorporates runtime T(x).

4. Experimental Results

ParEVO is evaluated across ParEval, PBBSBench, RPB, and DMOJ using both generated-code performance and correctness measures. Results show strong speedups, improved correctness from fine-tuning, benefits from evolutionary refinement, and a correctness–speedup trade-off on graph problems.

ParEval Performance: 107.43× mean Speedup@1 was achieved by Gemini-2.5-Parlay across all 59 ParEval problems relative to a sequential baseline.The result is attributed to valid, compilable parallel code and heavy-tailed performance on specific irregular tasks.
Fine-tuning Effects: Fine-tuning made ParEVO faster in eleven of twelve categories, frequently by an order of magnitude.It also improved graph Build@1 from 0.62 to 0.97 and Pass@1 from 0.42 to 0.76, while histogram Pass@1 increased from 0.19 to 0.63.
Fine-tuning Effects: 17.5× speedup was obtained on the complex number sorting task after fine-tuning, whereas the base model had Build@1 = 0.The fine-tuned model achieved Build@1 = 1, indicating successful compilation on that task.
Strong Scaling: 40× speedup was reached by generated Discrete Fourier Transform code on 64 cores, demonstrating strong scaling for this regular parallel workload.At 64 cores, another workload experienced performance loss from parallel overhead and thread contention in a nested parallel BFS loop.
Expert Baselines: 4.1× speedup over the baseline was achieved by the generated Rust solution for Maximal Independent Set, matching or exceeding expert implementations.The comparison used expert-written C++ and Rust implementations from PBBSBench and RPB.
Evolutionary Agent: 30 ECA iterations produced a 2.2× performance multiplier over single-shot generation.The ablation used a reserved set of DMOJ training problems; speedups were averaged across datasets relative to the first solution passing all tests.

5. Discussion and Limitations

ParEVO’s performance depends on aligning LLM generation with high-level parallel abstractions and using execution feedback for refinement. Its scope is limited by shared-memory assumptions, inference-time cost, and weaker generalization to unfamiliar domains.

The Role of Abstraction in Parallelization: Table 2 compares ParEVO-generated code with expert human implementations at 32 threads and also reports speedup relative to one thread.The baseline is state-of-the-art human-written code, and results use the best speedup across test inputs.
The Role of Abstraction in Parallelization: 30 iterations of the Evolutionary Coding Agent produce a 2.2× performance multiplier over single-shot generation.The ablation defines the comparison against the first solution that passes all tests.
The Role of Abstraction in Parallelization: ParlayLib’s high-level primitives encapsulate scheduling logic and reduce parallelization to local transformations aligned with Transformer token-local prediction.These primitives support code that is described as both mathematically sound and highly performant.
Limitations and Future Directions: ParEVO currently targets shared-memory multicore systems and does not address distributed-memory settings such as MPI/P-GAS.Distributed systems introduce communication-latency and data-partitioning constraints that are outside the stated scope.
Limitations and Future Directions: The Evolutionary Coding Agent increases inference-time computation because it generates and compiles multiple candidates, though the authors consider this cost amortizable for repeatedly executed HPC kernels.The stated rationale concerns kernels that may run trillions of times over their lifecycle.
Limitations and Future Directions: On some benchmarks, the model produces confident hallucinations when applying learned parallel patterns to unfamiliar algorithmic domains.The paper identifies formal verification in the evolutionary loop as a direction for constraining these semantic errors.

6. Conclusion

ParEVO connects generative AI with high-performance computing through specialized parallel-code data and model fine-tuning. Its results on ParEval surpass commercial LLMs and traditional heuristics, while evolutionary refinement is presented as essential for optimization beyond syntax generation.

6. Conclusion: ParEVO combines a specialized dataset of parallel primitives with fine-tuned models that internalize the Work-Depth cost model.The framework is presented as bridging modern generative AI and high-performance computing.
6. Conclusion: The Evolutionary Coding Agent uses compiler and runtime-profiler feedback as adversarial criticism to optimize code beyond single-pass syntax generation.The authors connect this process to reasoning about scalability, correctness, and algorithm–hardware interactions.

Impact Statement

The paper aims to broaden access to efficient parallel programming while recognizing verification risks in critical applications. Its examples pair parallel code generation with executable testing and benchmarked performance metrics.

The work could make parallel computing more accessible to non-experts and potentially reduce the energy footprint of large-scale computations.
Figure 3 contrasts lock-free Map-Scan-Write event generation with sequential push back that prevents parallelization and incurs reallocation costs.
Each training sample combines a natural-language instruction, a ground-truth parallel implementation, and an executable unit test for verification.
Table 5 reports Build@1, Pass@1, and Speedup@1 across commercial and open-weight models, with shaded regions distinguishing language and library groups.

C.2. Metric Breakdown by Problem Type

Fine-tuning improves build and pass rates across problem types, while speedup gains are strongest for irregular graphs and complex arithmetic. The resulting code is often more consistent but can sacrifice peak performance or fail on specialized APIs.

Fine-tuning universally improves Build and Pass rates, with the largest Speedup gains in irregular graph and complex arithmetic problems.
ParEVO solutions frequently match or exceed expert human baselines on runtime and scalability profiles.
Fine-tuned models often produce safer, more consistent code but slightly lower peak speedups by preferring high-level primitives over risky atomics.
Fine-tuning can cause confident API hallucinations in specialized domains, including repeated calls to a nonexistent convex-hull function.
Under a fixed token budget, Parlay-Instruct yields higher speedups on successful runs, whereas OpenMP fine-tuning produces more compiling and passing solutions with limited parallel gains.

F. Ablation Study on Agent Architecture and Search Strategies

The architecture ablation compares independent sampling and refinement with evolutionary agents that vary model fine-tuning and structural diversity. Results attribute ParEVO’s advantage to escaping local optima through MAP-Elites-guided exploration.

ECA-ft combines fine-tuned Qwen-Parlay with the complete diversity-driven ECA pipeline, unlike Best-of-N, Chain-of-Thought, Self-Refine, and reduced ECA variants.
The ablation evaluates runtime performance in seconds across DMOJ challenges, treating failed unit-test solutions as having no valid runtime.
ParEVO’s primary advantage over standard iterative repair is its ability to escape local optima.
MAP-Elites enforces structural diversity across synchronization primitives and data structures, enabling discovery of optimized implementations where single-lineage refinement fails.

G. Ablation Study on Iteration Budgets and Fine-Tuning Synergy

This ablation studies how evolutionary iteration budgets and fine-tuning interact. The full combination is most robust, while longer searches and structural diversity improve the chance of valid, efficient solutions.

ECA-Finetuned pairs Gemini-2.5-Parlay with the complete ECA pipeline for 30 iterations, while comparison settings vary iteration limits and diversity.
Table 9 reports runtime in seconds on DMOJ problems, with iteration configurations specifying the maximum evolutionary cycles and dashes indicating failed tests.
On coci19c1p3, every base-model configuration failed, whereas only fine-tuning combined with evolutionary search produced a correct optimized implementation.
On cco08p4, 5 iterations failed, 15 iterations produced a valid solution, and 30 iterations found the most time-efficient implementation.
Disabling MAP-Elites diversity reliably traps the search in local optima and can produce invalid or degraded solutions.

H. Integrity of Unit Tests in Training Data Verification

The training-data verification pipeline reuses human-authored tests where possible and checks the full corpus with ThreadSanitizer to assess race and runtime-error risks.

Human-authored tests from 593 seed programs are reused after mutation operators preserve each seed’s deterministic input–output mapping.New tests are synthesized only when input or output types change.
Approximately 150 ThreadSanitizer flags were manually inspected, with no genuine data races found among 50 reviewed cases.The flagged cases were candidate races or runtime errors requiring manual inspection.
ParlayLib’s functional, lock-free data-parallel primitives make data races and deadlocks structurally unlikely by replacing explicit thread management.

I. Leakage and Near-Duplicate Overlap Analysis

The authors examine training–evaluation overlap and report minimal structural similarity, with matches attributed to standard or compilation-related boilerplate rather than solution content.

The evaluation benchmarks are described as disjoint from the training data, while base and fine-tuned models share identical pretraining data.C++ models use Parlay-Instruct, and Rust models use a Rust version of DMOJ for fine-tuning.
N-gram overlap was measured at N = 10 and 15 using Jaccard similarity across the fine-tuning corpora and evaluation suites.
ParEval and DMOJ showed negligible 10-gram overlap below 0.00002, with matches consisting of standard-library dependencies and generic variable assignments.
PBBSBench showed minimal 10-gram overlap below 0.6%, predominantly involving C++ headers and ParlayLib boilerplate required for compilation.
Rust Performance Benchmarks showed 0.0 15-gram overlap and 0.000004 10-gram overlap, limited to standard Rust comparison boilerplate.
Across evaluation suites, maximum structural overlap remained below 0.6%, and manual inspection found no evidence that evaluation solutions appeared in training data.