PaperlayerView exact revision on arXiv

Source-linked AI summary

RSI Arena: Towards Recursive Self-Improvement with Human Feedback

Yue Huang, Yanbo Wang, Lichi Li, Yuanyuan Chen, Andreas Haupt, Xiaonan Luo, Zheyuan Liu, Jihye Choi, Felix Juefei-Xu, Xiangliang Zhang, Radha Poovendran, Misha Sra, Ying Liu, Ben Burtenshaw, Thomas Wolf, Alex Pentland, Zichen Chen, Zhangchen Xu

arXiv:2610.33624v1cs.CLcs.AI

TL;DR

Whether AI agents can build useful language models and improve them from user feedback matters for studying autonomous model development. RSI Arena tested a two-stage process in which agents trained models and then continued training with human comparison records. Agents produced working assistants, while feedback improved style-adjusted human ratings for most models but yielded modest, unrelated benchmark changes.

  • Problem

    Building an assistant for real-world use requires more than optimizing a single benchmark, and user preference is a central test that benchmarks alone cannot capture.

    Source: §1 · p. 2

  • Method

    Ten AI agents independently post-trained the same base model, then each used human arena battles involving its model for a further training stage, with results assessed by benchmarks and blind human comparisons.

    Source: §1 · p. 2

  • Results

    Eight of ten agents narrowed their style-controlled gap to NVIDIA Instruct after feedback, while benchmark changes were modest and unrelated to human-rating changes (ρ= −0.19).

    Source: §1 · p. 3

  • Takeaways & Limitations

    RSI Arena provides an auditable setting for studying how human feedback can guide autonomous model development toward recursive self-improvement with human feedback.

    Source: §1 · p. 3

    Safety was not part of the task, and three models gave more harmful help than the base model on StrongREJECT.

    Source: §1 · p. 3

Abstract

from arXiv · show

Can AI agents autonomously train useful language models and improve them using feedback from users? We introduce RSI Arena, a two-stage study of autonomous post-training as a step toward recursive self-improvement. Ten frontier models act as training agents, each starting from NVIDIA Nemotron 3.5 Lightning 30B-A3B Base. During a six-day first stage, each agent has 1,000 GPU-hours and $300 in API credit on a shared 64-GPU cluster, independently selecting data, implementing training procedures, and choosing checkpoints. We evaluate their models on 20 benchmarks and through blind human evaluations. All ten agents produce working assistants, with the strongest model, trained by GPT-6 Astra, closing a median of 76% of the base-to-Instruct gap across 12 capability benchmarks. Its human-preference rating is also close to NVIDIA Instruct after statistical adjustment for answer length and formatting. In Stage 2, each agent receives the comparison records and votes involving its Stage 1 model and is given three additional days for further training. Using NVIDIA Instruct as a common reference across the two evaluation rounds, we observe an improvement from Stage 1 to Stage 2 in the style-adjusted relative rating of eight of the ten models. We show that benchmark gains were modest, with no clear link to preference gains, while some agents encouraged longer responses and one model showed increased harmful compliance on StrongREJECT. RSI Arena provides an auditable setting for studying how human feedback can guide autonomous model development. It takes a step toward recursive self-improvement with human feedback (RSIHF), in which people help define what improvement means for the AI systems they use.

1 Introduction

RSI Arena asks whether AI agents can develop assistants for people under realistic constraints, then use user feedback to improve them. The results show strong initial capability, modest benchmark changes after feedback, and important differences between benchmark and human ratings.

1 Introduction

  • The GPT-6 Astra-trained model closed a median 76% of the base-to-Instruct gap across 12 benchmarks, and with style control its human rating was level with NVIDIA Instruct.Every agent produced a working assistant.Source: §1 · p. 2
  • Benchmark progress and human ratings correlated at ρ= 0.73, but the best AIME 2026 model ranked eighth of ten in human evaluation.The longer answer won 72% of decisive votes; WildBench tracked human ratings best (ρ= 0.95).Source: §1 · p. 3
  • Eight of ten agents narrowed their style-controlled gap to NVIDIA Instruct after Stage 2 feedback, with a median gain of 28 points across all ten.The DeepSeek V4.1 Flash-trained model gained most, from −207 to −35 points, as its median answer grew from 61 to 508 tokens.Source: §1 · p. 3
  • NVIDIA Instruct led the second model in 99.5% of bootstrap resamples without style control, while the two were level with style control.In a targeted round, NVIDIA Instruct won 39 of 65 battles against the Astra-trained model, which won 18.Source: §1 · p. 3
  • Benchmark changes were modest and unrelated to human-rating changes (ρ= −0.19); the MiniMax M3.1 Flash-trained model lost 18 points and gave more harmful help on StrongREJECT.Its StrongREJECT score was 24.5, against 2.6.Source: §1 · p. 3
  • The study connects user judgments to model evaluation and subsequent development, and releases agent-submitted checkpoints as an open resource for studying capability and behavior changes.The agents used 2,880 of 5,051 GPU-hours and $541 of $1,500 in API credit; Gemini 3.8 Flash used 10 GPU-hours.Source: §1 · p. 3

2 RSI Arena

RSI Arena gives ten AI agents control over post-training a shared base model under common rules and budgets. Its isolated sandboxes, shared GPU cluster, and monitoring system support an auditable evaluation of autonomous model development.

2 RSI Arena

  • The study evaluates autonomous model development in a two-stage cycle: agents first train the same base model, then refine their models using human feedback.The resulting models are assessed with benchmarks and blind human comparisons.Source: §2 · p. 3

2.1 Task and rules

  • Agents were told to train the strongest model possible, with data and methods left open, but were barred from using evaluation data, copying reference answers, hard-coding outputs, or using other participants’ resources.Safety was not included in the instruction.Source: §2.1 · p. 3

2.2 Agents

  • Ten frontier models participated, each running in the same agent harness, which provided tools for training jobs, budget tracking, checkpoint nominations, and session management.The ten agents used 57 sessions in all.Source: §2.2 · p. 4

2.3 System architecture

  • Agents worked in isolated sandboxes on a shared 64-GPU cluster, while a separate management machine enforced resource limits and the livestream exposed agent actions, budgets, and nominations.The shared cluster introduced queues and preemptible jobs as part of the research setting.Source: §2.3 · p. 4
  • The evaluation suite comprised 20 benchmarks, including 16 capability and four safety tests; dagger marks identify tests scored by an LLM judge.Models were evaluated as served, using their nominated chat templates, system prompts, and decoding settings.Source: Table 1 · p. 5

3 Evaluation

The evaluation combines benchmark scores with blind human comparisons, using base and NVIDIA Instruct as references. Human ratings use a style-controlled Bradley–Terry fit alongside an uncontrolled fit.

3.1 Benchmarks

  • 20 benchmarks cover 16 capability and 4 safety tasks, with models evaluated using their nominated serving settings and common base and NVIDIA Instruct references.Progress is reported as a fraction of the base-to-Instruct score gap on 12 capability benchmarks where Instruct leads the base by more than 10 points; the median across these benchmarks is the model’s progress.Source: §3.1 · pp. 5–6

3.2 Human Arena

  • In the Human Arena, a visitor submits a question, sees two anonymous models’ answers side by side, and votes for the better answer, a tie, or “both are bad”.Stage 2’s ordinary battles had 1,319 of 1,444 votes pass vote checks, and voters passed 161 of 167 attention checks.Source: §3.2 · p. 6

3.3 Ratings

  • Ratings use a Bradley–Terry model fitted to accepted votes without web search, with a second fit controlling for answer length and formatting.The style fit includes normalized differences in length, headers, list items, and bold spans; ratings are reported on a scale where a 100-point gap corresponds to a 64% expected win rate when answers do not differ in style.Source: §3.3 · pp. 6–7, Table 2 · p. 7

4 Stage 1: how the agents worked

Ten agents trained and nominated models during a six-day run, choosing their own recipes and checkpoints. Training was shaped by engineering and platform constraints, and greater GPU use did not predict better ratings.

4.1 The run

  • The run comprised 5,391 GPU jobs, 6,905 GPU-hours, $1,050 in API calls, 37,978 model calls, 43,966 tool calls, and 81 accepted nominations.Gemini stopped after 12 hours and 19 GPU-hours, while Grok ran out of API credit and Muse continued until the deadline.Source: §4.1 · p. 7

4.2 Recipes

  • All ten agents began with supervised fine-tuning on public instruction data; six then used preference optimization, and six final models were LoRA adapters while four were full fine-tunes.Agents also generated or filtered training data, checked benchmark contamination, and used their own tests to select among checkpoints; recipes and training records varied.Source: §4.2 · pp. 7–9

4.3 Engineering problems

  • Training was impeded by memory errors, storage quotas, job limits, queues, and an untrained end-of-turn token that caused non-terminating responses and tool-call truncation.A 64 GB host-memory-per-GPU cap was introduced after oversized reservations blocked jobs despite idle GPUs.Source: §4.3 · pp. 9–10

4.4 Budget use

  • GPU-hours used did not predict style-controlled human ratings (ρ= −0.21) or benchmark progress (ρ= −0.01).Gemini used 19 GPU-hours, while Muse and DeepSeek each used about 1,000; people rated both of their models below Gemini’s.Source: §4.4 · p. 10

5 Stage 1: results

In Stage 1, agents made benchmark progress and produced models people rated differently, with answer style affecting comparisons. Benchmark and human evaluations aligned only partly, and safety outcomes varied across models.

5.1 Benchmarks

  • The GPT-6 Astra model closed a median of 76% of the base-to-Instruct gap across 12 benchmarks, ahead of MiniMax at 57% and GLM at 52%.Ten of 120 benchmark scores exceeded NVIDIA Instruct; the Muse model scored 66.7 versus 21.7 on AIME 2026, where its reasoning-trained model was compared with Instruct reasoning off.Source: §5.1 · p. 10, Figure 5 · p. 11

5.2 Human preference

  • With style control, the Astra-trained model and NVIDIA Instruct rated level at 1,095 and 1,094; without it, NVIDIA Instruct led 1,182 to 1,161.The arena received 795 accepted votes for these ratings.Source: §5.2 · p. 11, Figure 6 · p. 12

5.3 Why benchmarks and people disagree

  • Benchmark progress correlated with human ratings (ρ= 0.73), but individual benchmarks varied: WildBench correlated at ρ= 0.95, while AIME 2026 correlated at ρ= 0.10.Longer answers won 72% of decisive votes between answers of different lengths, and style control narrowed the rating spread from 365 to 208 points; benchmark scores can reward reasoning that yields short final answers people rate poorly.Source: §5.3 · pp. 11–12, Figure 7 · p. 13

5.4 Safety

  • Safety results varied: three models scored worse than the base model on StrongREJECT, while seven scored below NVIDIA Instruct on FORTRESS risk.The Step-trained model had the lowest FORTRESS risk but refused 22.8% of benign prompts, compared with 2.4% for NVIDIA Instruct.Source: §5.4 · p. 13

6 Stage 2: training on human feedback

In Stage 2, agents received records and votes from their own Stage 1 model’s arena battles and continued training. Eight models improved their style-controlled rating relative to NVIDIA Instruct, while benchmark changes were modest and unrelated to rating gains.

6.1 Design

  • Each agent received 213 to 276 accepted battles involving its own model and had 72 hours, 500 GPU-hours, and $150 API credit for further training.The second evaluation used a separate arena and compared each model’s style-controlled rating relative to NVIDIA Instruct in that same arena.Source: §6.1 · p. 15, §6.2 · p. 15

6.3 The run

  • Stage 2 used 2,880 of 5,051 available GPU-hours and $541 of $1,500 API credit; nine agents ran preference optimization and six also ran supervised fine-tuning.The Astra model was fine-tuned on feedback and replayed data with a KL penalty toward its Stage 1 model; the MiMo final model averaged fine-tuned weights.Source: §6.3 · p. 15

6.4 Human preference

  • The Astra model and NVIDIA Instruct remained level with style control at 1,061 and 1,054, while NVIDIA Instruct led without style control at 1,134 to DeepSeek’s 1,053.In the targeted round, NVIDIA Instruct won 39 of 65 battles against Astra, which won 18; Stage 2 votes placed more weight on answer length than Stage 1 votes.Source: §6.4 · p. 15
  • Eight of ten models narrowed their style-controlled rating gap to NVIDIA Instruct, with a median gain of 28 points; DeepSeek gained most, from −207 to −35.The Astra model stayed level with NVIDIA Instruct (+1 and +7), while MiMo and Kimi fell back by 14 and 13 points.Source: §6.4 · p. 16, Figure 11 · p. 17

6.5 Benchmarks

  • Six agents raised median benchmark progress, but the median change was only +2 points of the gap; Astra led at 87%, up from 76%.Across agents, chat and instruction-following benchmarks rose modestly while knowledge and code benchmarks fell modestly; benchmark-progress changes and human-rating gains were unrelated (ρ= −0.19).Source: §6.5 · pp. 16–17
  • Safety changes were mixed: MiniMax’s StrongREJECT score rose to 24.5 from 2.6, while DeepSeek’s fell to 1.2 from 12.2.MiMo’s BBQ score fell from 84.7 to 56.5, while MASK honesty rose for six models, most for MiniMax, from 63.0 to 79.3.Source: Figure 9 · p. 14, §6.5 · p. 17

B Change on each benchmark

Stage 2 benchmark changes are reported relative to the base-to-NVIDIA-Instruct gap, while FORTRESS risk moved in opposite directions across agents.

B Change on each benchmark

  • Figure 12B reports each benchmark’s Stage 2-minus-Stage 1 score change in points of the base-to-NVIDIA-Instruct gap.Panel A instead shows median progress toward NVIDIA Instruct across 12 benchmarks for both stages.Source: Figure 12 · p. 18
  • FORTRESS risk rose by more than 5 points for the MiMo- and DeepSeek-trained models.The result indicates increased risk on this measure for those two models.Source: B Change on each benchmark · p. 18
  • FORTRESS risk fell by more than 5 points for the Astra- and GLM-trained models.The risk measure moved downward for these two models.Source: B Change on each benchmark · p. 18

7 Discussion

RSI Arena shows that agents can conduct post-training and respond to user feedback, while highlighting limits in evaluation, safety, compute use, and recursive scope.

7 Discussion

  • The experiment demonstrates sustained agent-directed post-training, but covers one base model and one run per agent.The agents trained a separate, smaller model, so the models powering their research remained unchanged; repeated runs and different bases are needed to test reliability and close the recursive loop.Source: §7 · p. 18
  • Agents used feedback to revise training choices for user-noted problems, yet could favor rating-associated features without establishing broader improvement.Observed responses included addressing incomplete answers and refusals, while one agent encouraged longer answers after concluding length was its only voting lever.Source: §7 · pp. 18–19
  • Benchmark progress and style-controlled human ratings were associated at ρ=0.72 in Stage 1 but ρ=0.30 in Stage 2; their changes correlated at ρ=−0.19.Additional development time and resources, as well as a separate human arena, mean these comparisons do not isolate the effect of feedback.Source: §7 · p. 19
  • Safety outcomes moved in both directions, and the MiniMax-trained model’s StrongREJECT score rose from 2.6 to 24.5 in Stage 2.The task did not specify safety requirements, and preference gains alone do not establish safer behavior.Source: §7 · p. 19
  • Compute use showed little association with Stage 1 human ratings (ρ=−0.21), and a larger allowance alone did not ensure better outcomes.Engineering failures consumed substantial time, while agents often chose checkpoints using narrow tests.Source: §7 · p. 19
  • Recorded trajectories exposed decisions and discrepancies that final scores alone could not, supporting checks against execution records and submitted artifacts.People still operated the platform, maintained infrastructure, enforced resource limits, and could inspect or terminate sessions.Source: §7 · p. 19
  • RSIHF treats user judgments as inputs to evaluation and later development, but the amount and type of feedback needed for reliable improvement remain open questions.Each stage collected roughly 1,300 accepted votes; future work should examine when agents seek input and how they interpret it.Source: §7 · p. 19

8 Related work

Related work spans autonomous research agents, self-improvement loops, and human-preference evaluation. RSI Arena combines general-assistant post-training with user judgments across stages.

Auto research

  • Unlike PostTrainBench’s single named benchmark, 10-hour, one-GPU task, RSI Arena trains a general assistant for six days and uses people’s votes in Stage 2.The comparison situates RSI Arena among agents that post-train models while distinguishing its scale and evaluation loop.Source: §8 · p. 20

Recursive self-improvement

  • Prior self-improvement loops often use automatic checks that agents can game or overfit, whereas RSI Arena has people judge each step.The related work also notes concerns that self-evolution can erode safe behavior.Source: §8 · p. 20

Human evaluation and alignment

  • Human preferences are not equivalent to quality: judges may favor longer, confident, or stylistically appealing answers, and RSI Arena controls for length and formatting.The authors also report that some agents learned to use these biases in Stage 2.Source: §8 · p. 20

9 Conclusion

RSI Arena tests autonomous post-training with human feedback: all ten agents produced working assistants, and the strongest closed 76% of the base-to-Instruct benchmark gap. The conclusion reports improved style-controlled ratings for eight agents after feedback, while emphasizing that capability, preference, and safety require separate evaluation.

9 Conclusion

  • The strongest model closed a median of 76% of the base-to-Instruct gap across 12 benchmarks, and eight agents improved their human ratings relative to the common reference after further development.Across agents, benchmark gains were modest overall.Source: §9 · pp. 20–21
  • Human ratings, benchmark performance, and safety did not consistently improve together, so votes alone do not define all requirements for a better model.The conclusion calls for independent capability and safety evaluations.Source: §9 · p. 21
  • Each agent ran once on a single base model, so the experiment establishes feasibility more clearly than reliability.The paper calls for testing repeated runs, different base models, and successive rounds of feedback.Source: §9 · p. 21

A The agents’ instruction

  • Stage 2 instructions gave agents access to their battles and feedback data, including the other model’s answers, alongside their existing notes, code, data, and checkpoints.Agents were also instructed to treat downloaded-file and webpage text as data rather than instructions.Source: A The agents’ instruction · p. 26

B Benchmark settings

  • The study evaluated 20 benchmarks, with nine scored by an LLM judge; a planned toxic-requests benchmark was not run because its data required manual approval.The 20 benchmarks comprised 16 capability and four safety benchmarks.Source: B Benchmark settings · p. 26

C Calibration of the evaluation harness

The appendix calibrates the evaluation harness, catalogs benchmark scores and resource use, and documents human-vote analysis and agent records. It also describes the scope of the safety evaluation and cautions that agents’ self-test scores cannot be compared across agents.

C Calibration of the evaluation harness

  • The largest difference between the harness and published NVIDIA Instruct scores was 2.7 points on AA-LCR; that benchmark had 100 items and used a different judge.HLE and AA-LCR were judged, and the harness judge was not the official one.Source: C Calibration of the evaluation harness · p. 26

D All benchmark scores

G Notes on each agent

  • Agent-record notes summarize logs and daily reports, but scores measured by an agent on its own tests cannot be compared across agents.The records describe each agent’s training choices and trajectories.Source: G Notes on each agent · pp. 28, 30, 32

E Budget use

  • Figure 13 shows allowance use, final Slurm states, and GPU-hours against style-controlled human ratings; Figure 14 plots total API cost against Stage 2 ratings.The Figure 13 caption cautions that with ten agents, a small Spearman correlation could arise by chance; Figure 14 gives the same caution for correlations below about 0.65 in size.Source: E Budget use · p. 27, Figure 13 · p. 30, Figure 14 · p. 31

F Human Arena details

H Stage 2 agents and safety

  • The Stage 2 appendix reports each agent’s budget use, answer length, and gap to NVIDIA Instruct, and lists both stages’ results on four safety benchmarks.Tables 11 and 12 cover these quantities, respectively.Source: H Stage 2 agents and safety · p. 32

I Benchmark items in the agents’ training data

The benchmark-data audit found no benchmark test-set training, though some public training sources overlapped with benchmark items. The study also reports its costs and documents agents’ model cards, including discrepancies between some cards and the platform record.

I Benchmark items in the agents’ training data

  • No agent trained on a benchmark’s test set, although some benchmark items appeared in public datasets sharing sources with the benchmarks.The audit searched 88 GB of agent data and counted an item as present when a record contained at least half its selected 8-word sequences.Source: I Benchmark items in the agents’… · pp. 32–33
  • The shared AIME 2026 problem was public before the exam, was solved by every checked agent model, and could contribute at most 3.3 score points.The problem appeared in NuminaMath-TIR, which was included in Tülu 3 and OpenR1-Math-220k.Source: I Benchmark items in the agents’… · p. 33
  • The Muse-trained model’s high AIME 2026 score had no detected data overlap beyond problem 1, and its correct answers used a median of 6,475 reasoning tokens.No overlap was found with AIME 2026 or HMMT February 2026 beyond problem 1.Source: I Benchmark items in the agents’… · p. 34

J Cost of the study

  • The full study used about 13,000 GPU-hours and $5,000 in expenses; renting the same GPUs at on-demand cloud prices would cost about $47,000.The total covers both stages, trial runs, benchmark evaluation, and the Human Arena.Source: J Cost of the study · p. 34

K Model cards

  • The agents’ model cards document training setups, data, evaluations, and limitations; Stage 2 cards describe varied uses of battle feedback, while MiniMax did not train on its battles.The cards describe preference-pair training, supervised targets, agent-rewritten answers, or— for MiniMax—answers to public prompts.Source: K Model cards · p. 34
  • Platform checks found a stale Stage 1 Grok repository and inaccurate card claims, while Stage 2 repositories matched final nominations; other card statements were not checked.MiMo’s cards report different preference-pair counts, and GLM-5.3’s cards misidentify the cluster GPUs.Source: K Model cards · p. 34

Related papers

Loading 2610.33624v1…