PaperlayerView exact revision on arXiv

Source-linked AI summary

EvoTrainer: Co-Evolving LLM Policies and Training Harnesses for Autonomous Agentic Reinforcement Learning

Guhong Chen, Yingcheng Shi, Yongbin Li, Binhua Li, Xander Xu, Hu Wei, Shiwen Ni, Min Yang, Jieping Ye

arXiv:2606.03108v2cs.AI

TL;DR

Autonomous LLM training usually searches recipes while keeping diagnostic infrastructure fixed, limiting evidence-guided steering in complex agentic RL. EvoTrainer co-evolves policies and training harnesses, matching or exceeding human-engineered RL references across three domains, with its largest gain on long-horizon SWE.

  • Problem

    Autonomous training systems typically keep diagnostics, memory, and intervention logic fixed despite changing bottlenecks and evidence in complex agentic RL.

  • Method

    EvoTrainer co-evolves LLM policies and training-side diagnostic harnesses through autonomous interventions, harness revision, persistent memory, and reusable skills.

  • Results

    Across Math, Coding, and SWE, EvoTrainer improves over no-RL baselines and matches or exceeds human-engineered RL references, with the largest gain on SWE-9B at 38.16 versus 33.77 BC%.

  • Takeaways & Limitations

    Autonomous LLM training can treat the trainer and its evidence-interpretation infrastructure as evolving objects rather than relying on repeated recipe edits and scalar-score comparison.

  • Takeaways & Limitations

    EvoTrainer is constrained by compute economics, consuming approximately 4.0 × 10^8 trainer-agent tokens per complete run and using a single training seed per version.

Abstract

from arXiv · show

Autonomous LLM training is often framed as recipe search, which leaves the training harness largely static. This limitation sharpens in agentic RL, where shifting bottlenecks and scalar rewards mask diverse failure modes. We introduce EvoTrainer, an autonomous training framework that co-evolves LLM policies and training-side harnesses through empirical feedback: it diagnoses rollout-level evidence, revises diagnostics, backtests interventions, and accumulates reusable skills. Evaluated on mathematical reasoning, competitive-programming code generation, and repository-level software engineering, EvoTrainer matches or exceeds the human-engineered RL references under the same data, codebase, and evaluation protocol, with the largest gain on long-horizon agentic SWE. Trajectory analyses show that retained strategies diverge across domains, evolving diagnostics prevent invalid high-scoring branches from being promoted, and reusable skills shape later search. Autonomous LLM RL should move beyond recipe search toward joint evolution of policies and the training harnesses that interpret them.

1 Introduction

The introduction argues that autonomous RL must improve not only model recipes but also the diagnostic infrastructure that interprets changing rollout evidence. EvoTrainer addresses this through coupled evolution of LLM policies and training-side harnesses, with evaluations spanning reasoning, coding, and software engineering.

  • Motivation: Fixed diagnostic infrastructure limits autonomous RL because shifting bottlenecks and scalar scores can obscure reward leakage, behavior collapse, or informative failures.Agentic RL produces long tool-using trajectories whose successful scores may hide unhealthy behaviors and whose failed branches may provide valuable negative evidence.
  • Framework: EvoTrainer treats the trainer itself as an object of improvement, while policies evolve within runs and trainers improve across runs by revising harnesses and reusing skills.The trainer observes completed versions, analyzes rollout evidence, proposes interventions, updates diagnostics, and selects what to test next.
  • Framework: EvoTrainer co-evolves policies through controlled version search and harnesses through self-reflection when metrics, analyzers, backtests, or search procedures are insufficient.Persistent memory and reusable skills support the coupled processes of policy self-evolution and trainer self-reflection.
  • Evaluation: EvoTrainer evaluates mathematical reasoning, competitive-programming code generation, and repository-level software engineering, matching or exceeding human-engineered RL references under the same protocols.The largest reported gain is on SWE-9B: 38.16 BC% versus 30.1…
  • Analysis: Analyses show that retained strategies diverge across domains, the evolving harness rejects invalid high-scoring branches, and reusable skills alter later search.These findings provide process-level evidence beyond score-driven iteration.

2 Related Work

Prior work increasingly automates scientific discovery, model development, and harness optimization, but inference-side harness evolution remains distinct from EvoTrainer’s training-side diagnostic adaptation. Specialized RL methods and agentic-RL studies expose sensitivity to trajectory, reward, and environment dynamics, motivating EvoTrainer’s evidence-based adaptation layer.

  • Autonomous scientific discovery: Recent systems automate scientific discovery and model development through propose–train–evaluate loops, meta-optimization, specialist-agent feedback, and end-to-end discovery.Examples include AutoResearch, Bilevel Autoresearch, specialist-agent frameworks, and the AI Scientist line.
  • Harness evolution: System performance depends on infrastructure around model weights, with Meta-Harness showing automatically discovered task-side harnesses can outperform hand-engineered designs.
  • Harness evolution: Inference-time harnesses such as AHE and TDScaling evolve coding-agent scaffolding, whereas EvoTrainer evolves a training-side diagnostic harness with metrics, analyzers, backtests, retrieved evidence, and reusable skills.The cited works target scaffolding around an LLM rather than the training process itself.
  • Agentic reinforcement learning: Agentic-RL research identifies trajectory-level instability, variance sensitivity, and dependence on reward shaping and environment stability.RAGEN identifies Echo Trap, RAGEN-2 proposes SNR-aware variance filtering, and long-horizon tool-using studies document sensitivity to reward shaping and environment stability.
  • Agentic reinforcement learning: EvoTrainer adapts to version-specific failures by retrieving or revising candidate mechanisms and retaining only interventions supported by cross-version evidence.

3 EvoTrainer: Co-Evolving LLM Policies and Training Harnesses

EvoTrainer treats autonomous training as evidence-conditioned version transitions in which LLM policies and the training-side diagnostic harness evolve jointly. Persistent memory and reusable skills make experimentation cumulative across versions and domains.

  • Core formulation: EvoTrainer makes the version transition the unit of autonomous improvement, preserving improvement, regression, mixed evidence, and insufficient evidence rather than binary keep-or-reject outcomes.Each completed state includes the policy version, harness, artifacts, diagnosis, proposed intervention, and observed outcome.
  • Policy self-evolution: The trainer constructs runnable versions, tests controlled candidate interventions, and promotes, prunes, reverts, or merges isolated branches using explicit experimental lineage.Single-factor interventions are preferred for clean attribution; ambiguous outcomes can trigger targeted ablations or backtests.
  • Training-side harness evolution: The diagnostic harness evolves when existing evidence cannot explain outcomes, distinguish hypotheses, or select the next intervention, expanding metrics, analyzers, procedures, and external retrieval.Its diagnostic layers cover score, signal, behavior, and version evidence.
  • Persistent memory and skills: Persistent memory stores lineage, recurring cases, reusable skills, and search knowledge, turning isolated experiments into cumulative processes with operational cross-domain reuse.A SWE filtering utility is later retrieved in Math and Coding when zero-variance groups recur.
  • Autonomy scope: The trainer autonomously runs the diagnostic loop and recommends interventions after human bootstrap, while humans gate costly or consequential execution.The workflow requires access to repository files, shell execution, experiment artifacts, and retrieval utilities.

4 Experiments

EvoTrainer is evaluated across mathematical reasoning, competitive-programming code generation, and repository-level software engineering under a common protocol, matching or exceeding human-engineered RL references. Results and trajectory analyses show domain-specific retained recipes, richer diagnostics, harness audits, and skill reuse shaping autonomous training.

  • Evaluation setup: EvoTrainer tests single-turn Math and Coding alongside long-horizon, executable-environment SWE interaction.All domains use Avg@8 with seed 42; correctness is judged by frozen-model evaluation, execution, or SWE BC%.
  • Main results: EvoTrainer achieves the strongest score in every reported Table 3 column and significantly improves over the no-RL base model in every domain.The no-RL comparisons have paired Wilcoxon p < 0.001 throughout.
  • Main results: On human-engineered RL, EvoTrainer improves SWE-9B by ∆= +4.39 and Math by ∆= +2.88, while matching human-engineered performance on SWE-4B and Coding.The SWE-9B improvement has 95% CI [+2.61, +6.34] and p < 0.001; the Math improvement has p < 0.001.
  • Domain-specific evolution: Retained recipes diverge by domain: Math emphasizes computation-aware tool augmentation, Coding execution-aligned reward shaping with variance-aware filtering, and SWE a behavior-sensitive pathway.RAGEN v2 SNR Filtering is the strongest algorithmic baseline in every column, but EvoTrainer improves on it throughout.
  • Trainer-level mechanisms: Richer diagnostics advance SWE-9B from v3 at 33.33 BC% to v4 at 36.30 BC% and v8 at 38.16 BC%, a +4.83 gain over v3.Harness audit changes an apparent 48.80 BC% Git-leak result to a legitimate 31.04 BC%, while skill reuse supplies a mechanism-matched Coding candidate.

5 Conclusion

EvoTrainer advances autonomous training by co-evolving policies, diagnostic harnesses, and reusable skills through evidence-conditioned experimentation. Across mathematical reasoning, coding, and software engineering, it matches or exceeds human-engineered RL references, with the largest gain on SWE-9B.

  • Contribution: EvoTrainer co-evolves runnable training versions, diagnostic harnesses, and reusable skills instead of repeatedly editing recipes based only on scalar scores.The trainer agent autonomously runs the diagnostic reasoning loop, while humans bootstrap the workspace and gate costly or consequential actions.
  • Results: Across Math, Coding, and SWE, EvoTrainer improves over no-RL baselines and matches or exceeds human-engineered RL references.Retained strategies diverge across domains, and component-level counterfactual evidence indicates the gains are not reducible to scalar-score iteration alone.
  • Results: 38.16 vs. 33.77 BC% is the largest reported gain, on SWE-9B.This result is reported alongside consistent improvements over no-RL baselines across Math, Coding, and SWE.

Limitations

EvoTrainer is primarily limited by compute economics, requiring substantial trainer-agent computation and using a single training seed per version.

  • 4.0 × 108 trainer-agent tokens are consumed on top of RL training, constraining EvoTrainer’s current realization.The compute constraint motivates using a single training seed per version, with stochasticity reported through per-task paired bootstrap.

A Training Objective Details · B Additional Domain-Evolution Details

The appendix expands the GRPO-style training objective and supplements the Math and Coding evolution traces with representative evidence supporting retained interventions.

  • A Training Objective Details: The training core uses a GRPO-style objective expanded from the main paper’s Section 3.6 summary.
  • A Training Objective Details: The token-level policy ratio is defined using the current and old policies for each token.
  • A Training Objective Details: The objective uses a clipped policy formulation with asymmetric clipping.
  • A Training Objective Details: Clip-Higher bounds the policy ratio with ϵℓ = 0.20 and ϵu = 0.27, alongside a weak KL term to a fixed reference policy.
  • A Training Objective Details: The full objective averages token-level losses within trajectories and trajectory-level losses within groups.
  • B Additional Domain-Evolution Details: The domain-evolution appendix adds evidence for the Math and Coding traces through representative version scores, diagnostic indicators, and supporting observations.It omits intermediate versions that were explored but not retained on the promoted path.

B.1 Math Evolution Trace … B.4 Diagnostic Metric Glossary

The retained Math and Coding trajectories evolve both policies and training-side diagnostics: they repair truncation and measurement artifacts, reshape reward signals, and introduce filtering or external computation for residual failures. Across the traces, diagnostic evidence redirects interventions from apparent model errors toward harness bottlenecks and more informative rollout signals.

  • B.1 Math Evolution Trace: Math evolution proceeds from an 18% truncation bottleneck through reward refinement and variance filtering to Code Interpreter support for computation-heavy residual errors.The retained sequence identifies response-length limits, improves reward-side signal quality, transfers a variance-aware filter, and adds external computation.
  • B.1 Math Evolution Trace: Approximately 18% of Math validation responses hit the maximum generation length, disproportionately affecting difficult competition problems whose reasoning chains terminate before completion.The diagnosis distinguishes incomplete reasoning caused by clipping from failures caused by an incorrect strategy.
  • B.1 Math Evolution Trace: Math v7 lowers DGR from approximately 0.21 to 0.13, while v8 has approximately 27% of validation samples invoke Code Interpreter, with gains concentrated on computation-intensive cases.The limited score gain over v5 indicates that low-variance optimization signal is not the sole residual bottleneck.
  • B.2 Coding Evolution Trace: Coding first repairs a format-gate measurement artifact, then replaces binary correctness with shaped continuous CR, and finally transfers SWE-derived filtering skills.The retained path omits intermediate explored versions, while v3 repairs format-gate false negatives and stable shaped-CR training begins at v8.
  • B.2 Coding Evolution Trace: Version v8 raises MidBand mass to approximately 0.34 and lowers DGR to approximately 0.31; v10 reaches 51.29 Avg@8 with DGR ≈0.18 and ADiv ≈0.63.The final Dual-Level Filter combines trajectory-level anomaly exclusion with group-level low-variance filtering and is adopted only in Coding.
  • B.3 Format-Gate Repair in Coding: Coding format-gate repair reveals that truncation, rather than semantic failure alone, drives many zero-reward outputs under the previous protocol.In 700 sampled responses, p50, p90, and p99 lengths all reach 20,480 tokens; 62.1% contain at least 18,000 tokens and 56.1% fail to close the reasoning block.
  • B.3 Format-Gate Repair in Coding: Under the stricter criterion, 352 samples qualify and all satisfy ntok ≥18,000; increasing max_new_tokens from 20,480 to at least 32,768 enables more complete outputs.The concentration of qualifying failures at high token counts shows that the format failures are tightly coupled to truncation.
  • B.4 Diagnostic Metric Glossary: Table 7 summarizes the diagnostic indicators used in the Math and Coding auxiliary analyses.The glossary section consolidates the diagnostic metrics appearing across the retained evolution traces.

C Cross-Domain Reuse of StdGroupFilter … D.4 Echo Trap in SWE-4B

EvoTrainer reuses validated group-level reward-dispersion filtering across SWE, Math, and Coding, while harness diagnostics reject misleading high scores and expose behavioral degeneration. Supplementary SWE analyses show that score improvements can conceal leakage, dead groups, or echo-trapped trajectories.

  • C Cross-Domain Reuse of StdGroupFilter: StdGroupFilter is developed in SWE as a reusable mechanism that filters low-information rollout groups by within-group reward variance rather than task-specific components.The filter tracks group-level reward statistics and is later adapted across domains.
  • C.2 Transfer to Math: The same validated filter transfers to Math because residual zero-variance groups share the SWE diagnosis, then to Coding alongside component-level reward shaping.Its transfer depends on within-group reward dispersion rather than task-specific reward semantics.
  • D.1 Score-Dominant SWE-9B Path versus Full EvoTrainer: 33.33 BC% is the saturation point of an early SWE-9B score-dominant path, while full EvoTrainer reaches 38.16 BC%.The early path improves over no-RL but saturates before stronger EvoTrainer regimes.
  • D.2 Harness Audit Prevents False Promotion under Git Leakage: 48.80 BC% is an invalid Git-leak score; harness inspection identifies anomalous Git usage and establishes 31.04 BC% as legitimate performance.Removing Git-history access prevents the contaminated branch from being promoted.
  • D.3 Dead-Group Rescue with the IF LLM Judge: At v4, 36.30 BC% coexists with a high dead-group ratio, motivating an instruction-following reward and retroactive backtesting on historical rollouts.The intervention targets residual groups with insufficient within-group reward variation.
  • D.3 Dead-Group Rescue with the IF LLM Judge: 45% of groups dead under correctness regain non-zero variance after adding a 0.1-weighted instruction-following signal.The backtest validates the mechanism before the signal becomes part of the retained recipe.
  • D.4 Echo Trap in SWE-4B: In SWE-4B, BC% rises from 29.71 to 33.28 while average turn count increases from 36–37 to 76.3, showing headline score gains can mask degeneration.The Echo Trap involves excessively long, repetitive trajectories with temporarily competitive validation scores.
  • D.4 Echo Trap in SWE-4B: Filtered degenerate trajectories increase eightfold, from 23 to 194, and behavior-level auditing rejects the echo-trapped branch in favor of a non-collapsing configuration.The filter distinguishes abandonment, idle failure, and Echo Trap modes.

D.5 Multiplicative Efficiency Factor Collapse (SWE-4B Branches)

The SWE-4B branch test isolates the effect of a multiplicative efficiency factor after Echo Trap degeneration. Both efficiency branches collapse into near-trivial behavior, while the continuous-CR control without efficiency remains viable, turning the failures into reusable negative evidence.

  • Branch design: After Echo Trap diagnosis, v9, v10, and v11 test responses to detected long-trajectory degeneration using different reward configurations.v9 applies efficiency to staircase correctness reward, v11 applies it to continuous CR, and v10 is the continuous-CR control without efficiency.
  • Branch outcomes: v9 initially reduces average turn count but collapses into near-trivial one-turn behavior, with BC% approximately zero, dead groups approaching 100%, and disappearing diversity.This branch uses the multiplicative efficiency factor with a staircase correctness reward.
  • Branch outcomes: v11 shows the same collapse under continuous CR plus efficiency, converging toward one-turn behavior and near-total dead-group saturation.Replacing the staircase reward with continuous CR does not prevent the failure when the multiplicative efficiency factor remains present.
  • Branch outcomes: v10 remains viable over hundreds of steps and avoids one-turn collapse when continuous CR is used without the multiplicative efficiency term.This counterfactual control isolates the efficiency multiplier as the differing component among the tested configurations.
  • Interpretation: Together, the branches show that a modest reward multiplier can create catastrophic behavioral incentives, so EvoTrainer retains these failures as reusable negative evidence.The conclusion relies on the contrasting v9 and v11 collapses against the viable v10 control.

E Search-Budget and Compute Accounting for Human Baselines … E.4 Trainer-Agent Inference Usage

EvoTrainer’s comparisons use shared data, code, hardware, evaluation, and seed conditions, while differing mainly in intervention-selection process. Despite greater human search compute in SWE and aligned iteration budgets elsewhere, EvoTrainer achieves stronger reported results, with trainer-agent inference accounted for separately.

  • E.1 Shared Experimental Conditions: Both conditions share the project codebase, official scripts, task assets, data splits, model families, hardware, evaluation protocol, validation setup, and seed convention.The principal difference is human RL engineering versus EvoTrainer’s trainer-agent workflow for selecting subsequent interventions.
  • E.1 Shared Experimental Conditions: The comparison uses the strongest human-engineered RL configuration observed under the shared stack, not an average or first attempt.The appendix characterizes the comparison as transparent but not perfectly compute-matched.
  • E.2 SWE Search Budget: SWE human baselines retain more versions, execute more training steps, and consume more GPU-hours, yet EvoTrainer achieves stronger final scores on SWE-4B and SWE-9B.The result indicates EvoTrainer’s SWE gains do not come from consuming a larger raw training budget, particularly for SWE-9B.
  • E.3 Math and Coding Budget Alignment: Math retains 8 major versions for both methods, while Coding retains 10, making iteration budgets and total training time broadly comparable within each domain.Table 17 summarizes this version-level alignment.
  • E.3 Math and Coding Budget Alignment: 84.17 / 73.33 / 81.94 on AIME 2024 / AIME 2025 / CNMO 2024 versus 80.83 / 71.67 / 77.78, while Coding Avg@8 reaches 51.29 versus 50.71.These results are reported under the aligned iteration budgets for Math and Coding.
  • E.3 Math and Coding Budget Alignment: The advantage is not confined to one favorable search budget, appearing under lower-budget SWE search and aligned-budget single-turn domains.This interpretation follows the budget comparisons across SWE, Math, and Coding.
  • E.4 Trainer-Agent Inference Usage: 4.0×108 trainer-agent tokens were consumed for diagnosis, evidence retrieval, backtesting, and intervention planning, separately from GPU-hour accounting.This accounting treats trainer-agent usage as LLM-mediated analysis and planning rather than policy-training compute.

F Head-to-Head Significance Analysis

EvoTrainer’s head-to-head significance analysis uses paired per-task Avg@8 comparisons with bootstrap confidence intervals and Wilcoxon tests. The headline results show significant gains on SWE-9B and aggregate Math, while SWE-4B and Coding match rather than significantly exceed the human-engineered RL reference.

  • Comparison families: The appendix compares EvoTrainer with human-engineered RL, the no-RL base model, and within-trajectory counterfactuals.These three comparison families support headline, seed-relative, and component-level analyses.
  • Statistical protocol: Each per-task observation is an Avg@8 score over 8 independent rollouts, analyzed with paired differences, 95% bootstrap intervals, and two-sided Wilcoxon tests.The paired bootstrap uses B = 10,000 task-level resamples; Math uses stratified aggregation across 78 problems.
  • Headline findings: p < 0.001 establishes significant gains for the headline SWE-9B result and aggregate Math improvement against human-engineered RL.The Math aggregate uses a stratified bootstrap over AIME 2024, AIME 2025, and CNMO 2024.
  • Headline findings: SWE-4B and Coding are not statistically distinguishable from human-engineered RL, so EvoTrainer is framed as matching rather than exceeding the reference.Their smaller margins sit within the bootstrap confidence interval.
  • Within-trajectory analysis: Within-trajectory paired statistics apply the same paired bootstrap and Wilcoxon protocol to the component-level counterfactual evidence.These comparisons are reported in Table 20 and referenced in Section 4.4.
Loading 2606.03108v2…