PaperlayerView exact revision on arXiv

Source-linked AI summary

Recursive Harness Self-Improvement for Frontier Reasoning Data Synthesis

Wenlong Zhang, Zhengbo Jiao, Chenxu Zhang, Lekang Jiang, SiYuan Ma, Qituan Zhang, Guo Chen, Linfeng Zhang

arXiv:2610.03548v1cs.AI

TL;DR

Static datasets and task-only recursion leave the synthesis procedure unchanged as reasoning tasks become harder. The paper co-evolves tasks and their harness, reporting difficulty growth across mathematics, coding, and science alongside downstream training value.

  • Problem

    As model capabilities improve, static datasets cannot adapt task difficulty, while task-level recursion reuses tasks without changing the construction skills and workflows.

    Source: §1 · p. 1

  • Method

    Task–harness co-evolution combines online skill updates from intermediate solver failures with post-task revisions to skills, prompts, and workflows, while keeping model weights and verification criteria fixed.

    Source: §1 · pp. 1–2

  • Results

    Across mathematics, coding, and science, fixed-solver accuracy fell from 100.0% to 54.8% over fourteen rounds, and the combined update schedules produced harder tasks than fixed-harness recursion or either schedule alone.

    Source: §1 · pp. 2–3, §5 · p. 11

  • Takeaways & Limitations

    The findings support adapting the synthesis harness alongside tasks to generate increasingly challenging data with downstream training value.

    Source: §5 · p. 11

    The downstream harness contribution needs matched-token training comparisons, while total evolution cost, candidate validity, fallback rates, and independent-run uncertainty remain unreported.

    Source: §4.5 · p. 10, §5 · p. 11

Abstract

from arXiv · show

Generating progressively harder reasoning problems requires synthesis procedures that adapt as the task distribution evolves. Existing task-level recursion reuses generated problems as seeds but leaves the construction harness unchanged. We present task-harness co-evolution, a framework for recursive harness self-improvement (RSI) in reasoning-data synthesis. Online self-improvement converts intermediate solver failures into reusable skills during generation. Post-task self-improvement revises skills, prompts, and workflows after each batch, adopting candidates only when they generate harder valid tasks within a bounded cost increase. Model weights and verification criteria remain fixed. Across mathematics, coding, and science, mean solver accuracy decreases from 100.0% to 54.8% over fourteen evolution rounds. Ablations show that combining both update schedules produces harder tasks than fixed-harness recursion or either schedule alone. The resulting data improves downstream SFT and GRPO performance. In particular, a 27B student fine-tuned on 10K synthesized mathematics examples achieves 62.5% mean-16 accuracy on APEX, competitive with selected frontier-model references. These results support adapting the synthesis harness alongside the tasks to generate increasingly challenging data with downstream training value.

1. Introduction

Because static datasets and task-only recursion leave construction fixed as solver capabilities change, the paper proposes co-evolving verified tasks and their synthesis harness. It reports difficulty growth across domains and downstream training value.

1. Introduction

  • Static datasets cannot adapt difficulty as model capabilities improve, and task-level recursion leaves reusable construction skills and workflows unchanged.Figure 1 contrasts fixed-pipeline synthesis, task-only recursion, and the approach's co-evolving tasks and harness.Source: §1 · p. 1, Figure 1 · p. 2
  • Task–harness co-evolution recursively updates verified tasks and the synthesis harness, using solver failures for online skill updates and batch feedback to revise skills, prompts, and workflows.Candidate harnesses are evaluated against the online snapshot and adopted under validity, difficulty, and cost criteria; model weights and verification logic remain fixed.Source: §1 · p. 2
  • Across mathematics, coding, and science, DeepSeek-V4-Pro accuracy fell from 100.0% to 54.8% over fourteen rounds; combined updates produced harder tasks than fixed-harness recursion or either schedule alone.Figure 2a reports the accuracy trend across 150 seed chains; the update-schedule comparison is reported in the introduction.Source: §1 · pp. 2–3, Figure 2 · p. 3
  • A 27B student fine-tuned on 10K synthesized mathematics examples reached 62.5% APEX accuracy, while the data improved downstream SFT and GRPO performance.The APEX comparison is described as competitive with selected frontier-model references.Source: §1 · pp. 2–3, Figure 2 · p. 3

2. Related Work

Prior work advances from synthetic instruction generation and recursive task construction to optimizing agents and their harnesses. This paper places its approach in the latter line while adapting construction alongside an evolving task distribution.

Frontier Reasoning in Science

  • Reasoning benchmarks and solver systems target increasingly difficult mathematical, scientific, and coding problems, motivating valid and informative challenges as the reasoning frontier moves.The cited examples span competition-level and graduate-level mathematics and science, broad academic knowledge, and research-oriented scientific coding.Source: §2 · p. 3

Reasoning Data Synthesis

  • Synthetic reasoning data spans instruction bootstrapping, predefined complexity operators, self-training and self-play, and recursive task or seed construction.Examples include Self-Instruct, WizardLM, OpenMathInstruct-2, ReSTEM, Absolute Zero, R-Zero, BenchEvolver, and RST.Source: §2 · pp. 3–4

Harness Evolution and RSI

  • Unlike task- or seed-recursion approaches, this work adapts construction skills and workflow alongside the evolving task distribution.The cited related work describes task or seed recursion as expanding the difficulty frontier, while leaving this paper's additional harness adaptation as the distinction.Source: §2 · p. 4
  • Agent and harness optimization work evolves prompts, workflows, code-defined designs, or data-preparation runtimes using execution or feedback signals.The passage distinguishes work such as ADAS, AFlow, Promptbreeder, STOP, the Darwin Gödel Machine, Autodata, and DataFoundry.Source: §2 · p. 4

3. Harness-Level Self-Evolution

The section defines recursive task synthesis with a fixed model and adaptive harness, pairing immediate skill updates during generation with batch-level harness revision and selection.

3. Harness-Level Self-Evolution

  • Each round recursively generates tasks with a fixed language model and adaptive harness, combining online skill updates with post-task revision, evaluation, and selection.The complete procedure is given in Algorithm 1.Source: §3 · p. 4

3.1. Problem Formulation and Initialization

  • Each original seed retains its latest valid completed task; failed attempts preserve the predecessor and remain in logs and cost accounting.The method does not retrospectively select the hardest ancestor.Source: §3.1 · p. 4
  • Difficulty is measured as a ratio of cohort means, with values above 1 indicating harder tasks; generation cost includes failed attempts and online-update calls.The ratio is not the mean of per-task ratios, and the method prioritizes difficulty subject to validity and bounded cost growth.Source: §3.1 · p. 5
  • Seed and source screening addresses duplicates and benchmark overlap before reusable skills, prompts, and workflows initialize the harness.Skill examples and training exports are checked again because input screening alone cannot rule out contamination introduced during synthesis.Source: §3.1 · pp. 5–6

3.2. Online self-improvement

  • A solver failure on a valid intermediate QA becomes a reusable construction transformation, not an answer to that QA.The updated library is available to unfinished and remaining tasks in the batch, though admitted skills need not increase difficulty.Source: §3.2 · p. 6

3.3. Post-task self-improvement

  • After batch completion or budget exhaustion, the Meta Agent uses task tests, execution logs, operational metrics, and human-specified goals and constraints to propose harness edits.Allowed edits include skills, prompts, workflows, and the placement of verification and revision; verifier logic cannot change.Source: §3.3 · p. 6
  • Candidates and the online baseline independently generate one further round from shared current tasks under matched settings, and candidates must preserve validity while reducing solver accuracy within a cost-growth bound.The cost allowance is 1.5 times the relative difficulty gain, capped at a 50% increase; the gate guarantees neither better cost efficiency nor gains on unseen tasks.Source: §3.3 · pp. 6–7
  • The first accepted candidate becomes the next harness; after three rejections, post-task edits are discarded and the online snapshot is retained.Online skills and rejection logs remain, and incomplete probes cannot pass.Source: §3.3 · p. 7

4. Experiments

Experiments assess difficulty growth across mathematics, coding, and science, compare harness-update schedules, and test transfer and downstream training utility. Solver accuracy falls over fourteen rounds, while both GRPO and SFT evaluations report gains for the synthesized data.

4.1. Experimental Setup

  • Synthesis uses DeepSeek-V4-Pro as a fixed solver and measures task difficulty by mean accuracy over four attempts.Downstream training evaluates Qwen3.5-4B Thinking with GRPO and Qwen3.5-35B-A3B and Qwen3.8-27B with SFT.Source: §4.1 · p. 7

4.2. Difficulty Progression

  • Solver accuracy falls from 100.0% to 54.8% by round 14, declining in every domain under a fixed cohort.The evaluation evolves 150 seed chains over fourteen rounds, with 50 seeds per domain.Source: §4.2 · p. 7

4.3. Online and Post-task Self-Improvement Ablations

  • Full reaches 50.0% accuracy, 9.0 points below Post-task only and 23.5 below Task-only, and 17.0 below Online only.Online only reaches 67.0% at round 14 versus Task-only’s 73.5%, after regressing from 64.5% at round 10; the contribution of candidate screening is not isolated.Source: §4.3 · pp. 7, 9, Figure 5 · p. 9

4.4. Harness Transfer and Generation Cost

  • On 50 held-out mathematics seeds, H14 yields 52.0% accuracy versus H0’s 70.0%; at round 14, Full costs $1.065 per seed versus Task-only’s $1.332.The reported operator fees exclude Director, skill-update, and search overhead, so they do not establish total savings.Source: Figure 6 · p. 9, §4.4 · p. 10

4.5. Downstream Training Utility

  • GRPO reaches 62.7% unweighted average versus 58.4% for its checkpoint, while SFT reaches 55.7%, 4.6 points above the checkpoint and MMIQC/IQC.The GRPO data comparison also reports 59.5% for IQC as the strongest alternative by overall accuracy.Source: §4.5 · p. 10
  • The downstream evidence does not isolate harness evolution’s contribution through matched-token training on fixed-harness and evolved-harness data.Total evolution cost, candidate validity, fallback rates, and independent-run uncertainty are also unreported.Source: §4.5 · p. 10

5. Conclusion

The paper concludes that co-evolving tasks and their synthesis harness supports progressively harder reasoning-data generation. It identifies matched-token comparisons, total generation cost, candidate validity, and difficulty transfer as priorities for future evaluation.

5. Conclusion

  • Task–harness co-evolution recursively improves reasoning tasks alongside the procedure that generates them.The conclusion presents the synthesis harness as a useful target for recursive improvement.Source: §5 · p. 11
  • Across the evaluated domains, fixed-solver accuracy falls from 100.0% to 54.8% over fourteen rounds, while synthesized data improves downstream SFT and GRPO performance.A 27B student trained on 10K mathematics examples reaches 62.5% mean-16 accuracy on APEX.Source: §5 · p. 11
  • Future work should compare fixed- and evolved-harness data under matched training tokens and measure total generation cost, candidate validity, and transfer to independent solvers.Source: §5 · p. 11

AI use statement

The statement discloses generative AI assistance and author responsibility for verification. The evolution procedure combines online skill acquisition with batch-level harness revision and retains the online snapshot when candidates are rejected.

AI use statement

  • Generative AI assisted with data generation, code, experiment-log organization and analysis, manuscript drafting, and language editing; authors report responsibility for independently verifying the work.Source: AI use statement · p. 11

A. Complete Evolution Procedure

  • After three rejections, the online snapshot remains the next-round harness, while the archive records versions, evaluations, and costs.Source: A. Complete Evolution Procedure · p. 16
  • During generation, attributed solver-miss traces can be converted into skills that the next Director call can use.Source: A. Complete Evolution Procedure · p. 16
  • After each batch, the procedure tests a revised harness against the online snapshot and adopts it only when the acceptance criteria are met.The candidate process allows an initial proposal and at most two retries on the same evaluation cohort.Source: A. Complete Evolution Procedure · p. 16

B. Method Protocol Details

The protocol screens seeds and skill sources for overlap, distinguishes solver misses from other failures, and tests candidate harnesses on a shared cohort under fixed evaluation settings. Adoption requires harder valid tasks within bounded cost, while archival and accounting rules support reproducibility and comparison.

B. Method Protocol Details

  • Curation removes exact duplicates and confirmed benchmark overlaps, flags near-duplicates for review, and keeps benchmark items and solutions isolated from generation agents.Shared concepts alone do not count as overlap; audit records are needed to establish contamination status.Source: B. Method Protocol Details · p. 17
  • Skills encode intent, construction method, intended difficulty effect, and tool hints; solver misses are distinguished from invalid problems, wrong answers, and tool failures.Source: B. Method Protocol Details · p. 17
  • Candidate evaluation uses a shared 150-task cohort across three domains, matched settings, and fixed inputs; probe-time skills and outputs do not feed back into production.The cohort is selected from current retained tasks, so this evaluation is not a held-out generalization test.Source: B. Method Protocol Details · p. 17
  • Acceptance requires strictly lower accuracy and no decrease in new-candidate validity, with a fixed 1.5 cost cap; the thresholds are engineering preferences, not statistically calibrated.In the stated example, the candidate may cost up to 1.30 times the baseline.Source: B. Method Protocol Details · p. 17
  • Without acceptance, the online snapshot is retained; the archive preserves snapshots and rejection records, while total evolution cost includes rejected proposals and post-batch evaluation.The relative cost cap does not establish lower absolute cost or lower cost per unit of difficulty.Source: B. Method Protocol Details · p. 18
  • Table 3 covers the 27B APEX training gain, while Table 4 covers selected APEX reference models; both report mean-16 accuracy.Source: Table 3 · p. 19, Table 4 · p. 19

C. Additional APEX Training and Reference Results

The section reports a within-checkpoint APEX gain after SFT on 10K synthesized mathematics examples and describes selected reference-model comparisons. It also identifies supplementary tables for difficulty progression and update ablations.

C. Additional APEX Training and Reference Results

  • 62.5% mean-16 accuracy on APEX follows SFT on 10K synthesized mathematics examples, compared with 56.3% for the Qwen3.8-27B starting checkpoint.The displayed-score gain is 6.2 percentage points; this comparison measures training gain, not the harness's contribution to it.Source: C. Additional APEX Training and… · p. 19
  • The reference comparison evaluates the listed models on APEX 2025 using MathArena, with scores averaging 16 attempts per problem rather than measuring pass@16.The stated 3.1-point gap to Kimi K3 (Think) concerns point estimates, not statistical equivalence; the ordering applies only to listed models.Source: C. Additional APEX Training and… · p. 19
  • Table 5 reports DeepSeek-V4-Pro accuracy progression over a fixed seed cohort, while Table 6 reports update ablations on 50 shared seeds.The supplied captions specify that lower accuracy is harder and that bold marks the lowest non-initial accuracy in each ablation column.Source: Table 5 · p. 20, Table 6 · p. 20

D. Numerical Difficulty and Ablation Results

The supplied passages define the scope of the difficulty and cost tables and clarify how overall accuracy and evaluation uncertainty are reported, but do not provide the tables’ numerical outcomes.

D. Numerical Difficulty and Ablation Results

  • Overall accuracy is recomputed across three equally weighted domains, and the four solver evaluations are not independent evolution runs.The aggregate values do not support an inference about run-level uncertainty.Source: D. Numerical Difficulty and Ablation… · p. 20
  • Table 7 reports DeepSeek-V4-Pro accuracy on tasks generated from 50 shared held-out mathematics seeds, with both harnesses frozen.Lower accuracy indicates harder tasks.Source: Table 7 · p. 21
  • Table 8 reports DeepSeek-V4-Pro accuracy and per-round generation-operator cost on 50 reasoning seeds, with lower values better for both measures.Cost is expressed in USD per original seed.Source: Table 8 · p. 21

E. Frozen-Harness and Cost Evaluation

The frozen-harness comparison uses shared held-out mathematics seeds, while the cost comparison reports per-round operator fees per original seed. At round 14, Full is both harder and lower-cost than Task-only on these measures.

E. Frozen-Harness and Cost Evaluation

  • The frozen-harness comparison starts both snapshots from the same 50 held-out mathematics seeds, with skills, prompts, workflows, and verification or revision placements frozen.Only tasks evolve, and the reported round index is distinct from the harness version.Source: E. Frozen-Harness and Cost Evaluation · p. 21
  • Operator-cost values divide reported round fees by the original 50 reasoning seeds, not by newly successful tasks.The values are per-round rather than cumulative.Source: E. Frozen-Harness and Cost Evaluation · p. 21
  • The operator-cost measure excludes Director, skill-update, post-task update, candidate-search, initialization, and standalone reporting costs, so it is not total search cost.The round-14 comparison does not quantify the cost of obtaining H14 or establish matched-total-budget superiority.Source: E. Frozen-Harness and Cost Evaluation · p. 21
  • At round 14, Full reduces solver accuracy by 23.5 percentage points and operator cost by $0.267 per seed (20.0%), with the cost difference computed from reported three-decimal fees.Source: E. Frozen-Harness and Cost Evaluation · p. 21

F.1. Models, Benchmarks, and Evaluation

The evaluation specifies its synthesis and student models, benchmarks, protocols, and baselines. Training configuration and implementation details are also reported for the downstream comparisons.

F.1. Models, Benchmarks, and Evaluation

  • DeepSeek-V4-Pro is the fixed difficulty-evaluation solver; synthesis roles use DeepSeek-V4-Pro, Qwen3.8-Max, and GPT-5.6 as specified.All listed synthesis and meta-agent models run with Thinking enabled.Source: §F.1 · p. 22
  • Evaluation covers APEX/Shortlist, HMMT, LiveCodeBench v6, and SciBench; difficulty averages four solver attempts, while GRPO benchmark scores use mean-16 accuracy.Lower solver accuracy indicates harder tasks; higher downstream accuracy is better.Source: §F.1 · p. 22
  • Comparisons include OpenMathInstruct-2, OpenCodeReasoning, recursive IQC/MMIQC, and, for GRPO, WildSci and MetaMath.The passages retain the cited works’ printed bibliography markers.Source: §F.1 · p. 22

F.2. Training Configuration

  • GRPO and SFT comparisons use shared hyperparameters within each dataset comparison; training sets contain 10K examples, except the 35B SFT comparison uses 12K.The 35B configuration uses 10K mathematics and 2K coding examples; the 27B SFT run uses 10K mathematics examples.Source: §F.2 · p. 22, Table 9 · p. 23
  • SFT uses padding removal, gradient checkpointing, activation offloading, and resharding; GRPO uses dynamic batching and applies KL regularization through the actor loss.The GRPO reference model offloads parameters, while the actor keeps parameters and optimizer states on device.Source: §F.2 · p. 22

Related papers

Loading 2610.03548v1…