PaperlayerView exact revision on arXiv

Source-linked AI summary

ParetoRSI-Gym: Can AI Agents Improve Language Models Across Multiple Domains?

Haojin Wang, Yangzhen Wu, Zeyi Chen, Jian Xie, Jiaying Ye, Lili Meng, Shuyun Tang, Shangding Gu, Dawn Song

arXiv:2610.34386v1cs.LGcs.AI

TL;DR

PARETORSI-GYM evaluates iterative post-training data curation across multiple domains, addressing the focus of prior evaluations on single benchmarks. Claude Code with Opus-5 achieves higher generalization scores at its proxy-best checkpoint, while the analysis finds that agents often fail to keep improving models across rounds.

  • Problem

    PARETORSI-GYM addresses the focus of post-training data curation evaluations on single benchmarks rather than multiple domains.

    Source: §6 · p. 11

  • Method

    PARETORSI-GYM evaluates agents that iteratively curate data across five domains, use proxy-benchmark feedback, and are assessed on separate generalization benchmarks.

    Source: §1 · pp. 2–4

  • Results

    9.2% higher generalization scores than the starting model (+2.3 absolute points) for Claude Code with Opus-5 at its proxy-best checkpoint, averaged equally over three base models and five domains.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Agents form curation strategies from feedback but often do not continuously improve the model or return the best checkpoint, while harness design and agent-specific data preferences matter.

    Source: §1 · p. 4, §6 · p. 12

    Evaluation covers only three agent systems, three base models, five domains, and English text, and reflects system versions available when the study was written.

    Source: Limitations · p. 12

Abstract

from arXiv · show

Recursive self-improvement (RSI) requires an agentic system to design, train, and evaluate its own successor. Data-driven post-training is a central part of that loop, yet existing evaluations measure it on a single benchmark, where an agent selects from a static pool or distills a stronger model to push one score up in one run. Real post-training is multi-domain, and a gain in one domain has to be weighed against a regression in another. We propose ParetoRSI-Gym, in which an agent searches for data, curates it, trains, reads back per-domain results, and repeats, with everything but the data held fixed: the initial LLM, the five domains (math, code, finance, medicine, science), the toolset, and the SFT pipeline. Generalization benchmarks in each domain separate capability gains from overfitting to the scores the agent optimizes against. Across three agent systems, three base models, five rounds and multiple seeds, the strongest system, Claude Code with Opus-5, raises the generalization scores of its proxy-best checkpoint by 9.2% relative to the starting model (+2.3 absolute points), averaged with equal weight over three base models and five domains, with a single-domain relative gain of up to 63% (medicine on Ministral-3-3B). Further analysis shows that the harness matters as much as the model it drives, and that each agent keeps a unique data selection strategy. We see our work as a step towards evaluation of RSI systems in real-world machine learning engineering.

1 Introduction

ParetoRSI-Gym evaluates whether agents can iteratively curate post-training data across multiple domains, addressing the gap between single-benchmark evaluations and real post-training’s multi-capability objectives. Its proxy-feedback loop is paired with held-out generalization evaluation; across experiments, Claude Code with Opus-5 achieves the strongest reported gain, while outcomes also vary with harness and agent-specific data selection.

1 Introduction

  • ParetoRSI-Gym evaluates iterative data curation across five domains, with agents searching, retrieving, selecting data, designing training recipes, and revising them using per-domain proxy feedback.The benchmark uses LoRA SFT and holds the training pipeline fixed while agents choose round-specific hyperparameters.Source: §1 · pp. 2–4
  • Held-out generalization benchmarks and rejection of data close to evaluation items distinguish capability gains from overfitting to proxy feedback.The benchmark reports results on the same domains but different benchmarks from those used for feedback.Source: §1 · pp. 3–4
  • 9.2% relative (+2.3 absolute points): Claude Code with Opus-5 improves proxy-best generalization scores over the starting model, averaged equally across three base models and five domains.The largest proxy-best-round absolute gains concentrate in medicine and finance, reaching 14.8 points.Source: §1 · p. 3
  • In seven of nine configurations, the proxy-best checkpoint is not the final-round model, indicating that current agents do not robustly continue improving models.The reported comparison uses nine independent curation runs across three agent systems and three target models.Source: §1 · p. 3
  • With the backbone fixed, harness changes correspond to outcomes ranging from +4.3 to +0.2 points, while same-agent dataset overlap is 0.24 Jaccard versus 0.07 across different agents.The authors interpret the overlap pattern as evidence that agents retain distinct data-selection strategies.Source: §1 · pp. 3–4

2 PARETORSI-GYM

ParetoRSI-Gym runs a feedback-driven, multi-domain data-curation loop and evaluates outcomes on benchmarks withheld from agents. Its design combines persistent state, live data sourcing, iterative revision, proxy feedback, and separate generalization evaluation.

2.1 Design Principles

  • Agents pursue a multi-domain Pareto frontier, receive the full per-domain score vector each round, and carry models, training data, and written principles across rounds.The written-principles budget bounds how much evidence from one round can be carried into the next.Source: §2.1 · p. 4, §2.2 · p. 4

2.2 Setup

  • The setup distinguishes proxy benchmarks returned during the run from generalization benchmarks that cover trained capabilities but remain unseen by the agent.The agent’s objective is to advance the attainable proxy frontier relative to the initial model rather than raise a single score.Source: §2.2 · p. 4

2.3 Data Curation Loop

  • Agents search Hugging Face datasets, select datasets or subsets, and pull entries into a training pool before fine-tuning a model and updating the pool and principles from proxy results.The loop repeats for T rounds and reports both the final-round model and the checkpoint with the best proxy score on generalization benchmarks.Source: §2.3 · p. 5

2.4 Evaluation

  • Evaluation uses separate proxy and generalization suites targeting the same capabilities without shared items; generalization scores are reported as accuracy.The generalization suite is scored only after the final round, while the agent receives proxy scores and evaluation traces during the run.Source: §2.4 · p. 5

3 Experiments

The experiments evaluate autonomous post-training across three model families and five domains, using agent systems that curate data and iterate from proxy feedback under a largely fixed training pipeline. Claude Code has the strongest reported proxy-best average, but gains vary by domain and model, and later rounds or proxy-based checkpoint selection can miss better per-domain outcomes.

3.1 Model

  • The experiments use three base models from different pretraining families and include varied model sizes to examine model-dependent improvement patterns.The models are Qwen3-4B-Base, Ministral-3-3B-Base-2512, and Olmo-3-1025-7B.Source: §3.1 · pp. 5–6

3.2 Research Agents

  • Three research-agent systems pair different harnesses with backbone models and run autonomously in a sandbox restricted to Hugging Face Hub access.Generalization benchmarks are absent from the agents’ environment, and each run continues in one session with prior interaction history.Source: §3.2 · p. 6

3.3 Training Details

  • Each run lasts five rounds, with each round capped at 15M training tokens and 40k entries; the remaining training pipeline is frozen.Agents choose learning rate, LoRA rank, batch size, sequence length, and warmup within fixed ranges; Claude Code and Codex configurations use three seeds, while CheetahClaw runs once per base model.Source: §3.3 · p. 6

3.4 Main Results

  • +2.3 points with Claude Code, +0.6 with Codex, and −0.3 with CheetahClaw are the proxy-best checkpoints’ average gains over starting models across five domains and three base models.The corresponding relative gains are +9.2%, +3.1%, and +2.9%; the relative average weights low-baseline cells heavily.Source: Table 3 · p. 7, §3.4 · p. 7
  • The largest relative gain is +63% on medicine with Ministral-3-3B, but part of it reflects learning answer format rather than medical knowledge.Agents tend to improve medical and code scores, while math performance behaves differently and can decline; gains also vary by base model.Source: §3.4 · p. 8
  • In 24 of 45 settings, a later round performs worse than the round with the highest proxy gain, so continued curation can undo earlier gains.The checkpoint returned is often not the best one produced during the run.Source: §3.4 · p. 8
  • In 43 of 45 model–domain cells, the domain’s best round differs from the agent-selected proxy-best round, at an average cost of 2.0 points and up to 6.9.Domain Best combines different rounds across domains, so it is an oracle rather than a deliverable checkpoint; the findings suggest agents struggle to promote a Pareto frontier that preserves existing capabilities while exploring.Source: §3.4 · pp. 8–9

4 Analysis

The analysis examines how proxy feedback relates to generalization, how harness choice affects results with a fixed backbone, and how agents select data across rounds.

4.1 Trend Analysis

  • Proxy gains move in the same direction as generalization gains, but their magnitudes differ; on code, proxy gains stay within 3 points while generalization gains reach 11.Claude Code and Codex continue from an earlier checkpoint in 64% and 53% of rounds, respectively.Source: §4.1 · pp. 9–10, Figure 4 · p. 9

4.2 Harness Performance

  • With the backbone fixed, Codex gains 4.3 generalization points on average, versus 2.6 for Claude Code and 0.2 for CheetahClaw.The comparison uses DeepSeek V4.1 Flash Max on Qwen3-4B-Base, with one run per harness.Source: §4.2 · p. 10, Figure 5 · p. 10

4.3 Agent Decision-Making

  • Runs by the same agent have 0.24 Jaccard overlap in selected datasets, versus 0.07 across agents; overlap does not rise when runs share a base model.Average domain shares barely change across rounds, even though individual runs can heavily favor one domain.Source: Figure 3 · p. 9, §4.3 · p. 10
  • Fewer new datasets enter the pool as runs progress, suggesting agents become less willing to explore new sources later in the task horizon.The paper describes this as static behavior over time.Source: Figure 3 · p. 9, §4.3 · p. 11

5 Related Work

Related benchmarks have studied data curation and recursive self-improvement from different angles; PARETORSI-GYM focuses on iterative post-training curation across multiple domains.

5.1 Data Curation Benchmarks.

  • Earlier curation benchmarks evaluate downstream performance, while newer agent-based settings vary from fixed-pool selection to benchmark-specific post-training and evolving curation policies.PARETORSI-GYM instead lets agents select dynamic data and evaluates generalization across multiple domains.Source: §5.1 · p. 11

5.2 Recursive Self-Improvement Systems.

  • Prior RSI systems empirically improve scaffolds, edit their own editing mechanisms, share experience, co-evolve with evaluators, or update model weights; PARETORSI-GYM benchmarks multi-domain post-training curation.The related work also notes that harnesses tuned on an evolution split transfer only partly to unseen tasks.Source: §5.2 · p. 11

6 Conclusion

PARETORSI-GYM evaluates agent-driven post-training data curation across multiple domains, using proxy and generalization benchmarks. Experiments find that agents form strategies from feedback but may fail to improve models continuously or return the best checkpoint.

6 Conclusion

  • PARETORSI-GYM evaluates post-training data curation across multiple domains rather than focusing on one benchmark, with proxy and generalization benchmarks intended to mitigate contamination.The generalization benchmarks are intended to avoid overfitting to domains encountered during curation.Source: §6 · pp. 11–12
  • Across experiments, agents form curation strategies from feedback but can lack continuous model improvement and often return suboptimal checkpoints.The analysis also finds distinct data-selection preferences across agent systems and differences associated with harnesses.Source: §6 · p. 12
  • The authors present the benchmark as a first step toward more realistic and comprehensive evaluation of post-training automation.Source: §6 · p. 12

Limitations

The evaluation is limited to three agent systems, three base models, five domains, and English text; the agent systems are moving targets.

Limitations

  • The study covers only three agent systems and three base models, so broader scenarios and larger scales remain for future work.Source: Limitations · p. 12
  • Results are a snapshot of the harness and backbone versions available when the study was written.Source: Limitations · p. 12
  • The benchmark covers five domains and English text, leaving other languages and domains for future work.Source: Limitations · p. 12

Ethics Statement

The ethics statement outlines a five-domain data-curation workflow that optimizes a Pareto front using proxy scores and evaluates results on held-out sets, while cautioning that autonomous data selection can introduce bias. It also describes the fixed-pool decision process, which usually occurs without the agent reading entries, and a check that score changes cited in rewritten beliefs match the record.

Ethics Statement

  • Real-world use should include bias audits, content filtering, and strict data-retention controls because agents inherit source biases and add selection bias of their own.The study uses public datasets and published benchmarks, collects no user data, and sandboxes agents; entries are screened against evaluation items.Source: Ethics Statement · p. 12

B Prompts

  • The prompt makes the objective the Pareto front across five domain scores, treating the proxies as feedback rather than the final exam on held-out sets.Evaluation items, mirrors, and solutions are prohibited from training; checks reject overlapping pages, with gold answers unavailable and proxy-text searches recorded as violations.Source: B Prompts · pp. 18–19
  • The prompt requires agents to inspect available data, plan actions tied to skill principles, acquire and select data, revise the skill from checked score evidence, and submit.Acquisition is blocked until the plan exists, and the skill records principles and their supporting round-by-round evidence.Source: B Prompts · p. 19
  • Training is one SFT pass over a pool capped at 40,000 entries and 15 million tokens, from a chosen checkpoint whose lineage carries forward prior gains or damage.The prompt fixes epochs at one and the seed; branching from an earlier checkpoint is the stated way to escape a regression, at the cost of discarding later learning.Source: B Prompts · p. 20

C A Round in Practice

  • The agent made four pulls totaling 17,100 rows, then chose 25,908 entries with 10.0M scheduled tokens against a 15M cap, without opening any row.It selected a recipe with learning rate 2 × 10−6, rank 32, batch 64, length 4096, and init_from: base, discarding the round-1 checkpoint.Source: C A Round in Practice · p. 23
  • The harness rejects a round if the score change cited by a rewritten belief disagrees with the record.In the described round, ten new beliefs were added and two were retired.Source: C A Round in Practice · p. 23
  • The pool is fixed in Step 6 without reading any entries in 49 of the 50 logged rounds.The decision determines the fate of tens of thousands of entries.Source: C A Round in Practice · p. 23

D.1 Failure Case on Mathematical Reasoning

Most agents fail to improve mathematical reasoning, and the case study links a sharp first-round decline followed by recovery to changes in training-data answer length. The figure tracks answer length and math accuracy across rounds.

D.1 Failure Case on Mathematical Reasoning

  • Most agents fail to improve the original model on mathematical reasoning tasks.Source: §D.1 · p. 23
  • Figure 6 plots answer length and math accuracy over the rounds, visualizing the two measures discussed in the case study.Source: Figure 6 · p. 24, §D.1 · p. 24
  • Math falls 20.1 points in round one, then recovers above M0 in round two for Claude Code with Opus-5 on Olmo-3-1025-7B.Both rounds train from the base model under the same recipe.Source: §D.1 · p. 24
  • Short-answer NuminaMath data coincides with the early decline, while later selection favors longer reasoning chains that match the original model’s answer lengths.The authors state that the model gains accuracy by generating longer answers; the first-round selection record cites a token budget as the reason for preferring concise traces.Source: §D.1 · p. 24

D.2 Forming Experience While Curating

The finance case follows an agent forming beliefs from proxy feedback and using them to adjust its data selection. Filtering the selected economic traces is followed by a best-in-run finance score and reduced truncation.

D.2 Forming Experience While Curating

  • Round one raises finance from 0.188 to 0.224 while answers remain 2,279 tokens; the other four domains lose their reasoning style.Source: §D.2 · p. 25
  • Round two uses 10-option economic traces and reaches 0.244, while truncation rises to 29% at 3,356 tokens.Source: §D.2 · p. 25
  • In round 3, operating on fin3, the system filters the same source to traces of at most 8k characters; finance reaches 0.260, the best score in the run.Truncation falls to 21% at 2,726 tokens.Source: §D.2 · p. 25
  • The authors interpret the agent’s proxy-derived beliefs as useful experience that guides curation and contributes to later improvement.Source: §D.2 · p. 25
  • After round one, the agent associates preserved long reasoning in finance with 10-option OpenScience economic traces, then uses that belief to guide later curation.Finance answers averaged 2,279 tokens, compared with 65–718 tokens in the other domains.Source: §D.2 · p. 25

D.3 A Round That Improves Every Domain

On Ministral-3-3B, the agent reaches a checkpoint that is at least as good as the starting model on every generalization benchmark. Later rounds lose the Pareto improvement, illustrating the difficulty of retaining it.

D.3 A Round That Improves Every Domain

  • Round three is at least as good as M0 on every generalization benchmark and significantly better on medicine and code, but rounds four and five give back math.The round-three checkpoint also has the highest proxy score; the final checkpoint no longer dominates M0.Source: §D.3 · p. 25
  • Round two gains on math and code but loses 7.8 points on medicine; the agent discards it and branches round three from round one.Source: §D.3 · p. 25
  • The case demonstrates that the loop can reach a Pareto improvement over the starting model, but maintaining it is harder than finding it.Source: §D.3 · p. 25

E Datasets the Agents Select

Human review of agent-proposed data selections matches or exceeds the strongest agent on every base model. Dataset preferences differ by agent, while seed-to-seed variability is usually one to three points.

E Datasets the Agents Select

F Human Baseline

  • Human supervision matches or exceeds the strongest agent on every base model, with especially clear gains in math and finance.The clearest cases are math on Qwen3-4B-Base and Ministral-3-3B, and finance on Olmo-3-1025-7B.Source: F Human Baseline · p. 28, Table 7 · p. 28

G Variance Across Seeds

  • Most seed standard deviations are one to three points; the largest are ±5.6 on Claude Code math at Final and ±6.9 on Codex code.Reported means and sample standard deviations are across three training seeds on Qwen3-4B-Base.Source: G Variance Across Seeds · p. 28, Table 8 · p. 29

Related papers

Loading 2610.34386v1…