PaperlayerView exact revision on arXiv

Source-linked AI summary

From RL Papers to Framework Code: Inclusion and Review in Seven Open-Source Training Frameworks

Taiheng Pan

arXiv:2610.28027v1cs.LGcs.CLcs.SE

TL;DR

Framework inclusion is an unmeasured step between publishing RL methods for language models and practitioners encountering them as framework options. The paper traces seven frameworks against a screened arXiv pool and finds that few general methods reach their main branches. Inclusion remains sparse as the paper supply grows, and the authors release a tracker connecting failures, implementations, and contribution records.

  • Problem

    Framework inclusion had not been measured, although practitioners encounter RL methods through the frameworks they train with.

    Source: §1 · p. 1

  • Method

    The authors trace framework main-branch implementations against a screened arXiv pool and define inclusion as an implementation reaching a tracked framework’s main branch.

    Source: §4 · p. 4, §11 · p. 13

  • Results

    37 of 1,598 general methods (2.3%) reached a framework; 180-day inclusions stayed at 5–7 as quarterly cohorts grew 5.4-fold.

    Source: §1 · p. 2, §13 · p. 14

  • Takeaways & Limitations

    The released tracker links training failures to framework implementations and settings and records contribution rules, reviewer concerns, and earlier proposal outcomes.

    Source: §1 · p. 2

    Main-branch inclusion does not show how often methods are used or their user results, and the screened pool and seven-framework sample are not comprehensive.

    Source: §11 · p. 13

Abstract

from arXiv · show

Practitioners use reinforcement learning methods for language models through training frameworks, so a method that a framework includes is one configuration change away for its users. We measure how often this happens. We trace seven open-source frameworks for RL post-training through their commit histories and link them to 6,939 screened arXiv papers from January 2025 to June 2026. Of 1,598 papers that propose a general RL training method, 37 reached at least one framework. Across five quarterly cohorts of papers, the number included within 180 days stayed between 5 and 7 while the cohorts grew from 72 to 386 papers. Compared with papers from the same quarters, included papers more often trained policies of 30B parameters or more, at 41 against 10 percent, and more often changed a coefficient, clip, mask or weight in the existing loss, at 29 against 9 percent. Both gaps hold without papers from the companies that maintain the frameworks, while code release was equally common in both groups. The frameworks' issues and pull requests show that few methods are turned away in public. More than nine in ten of the general methods no framework included have no thread there that cites or names them, and many proposals that do appear stay open without a reply. Two further records, the frameworks' issues and the technical reports of 13 large open-weight models, show that users rarely report training failures in public and that the reports state 8 of the 13 widely included methods. We release the data as a tracker that links training failures to framework implementations and lists each framework's contribution rules and earlier proposals.

1 Introduction

The section treats framework inclusion as a distinct step between publishing a method and practitioners having it as a configuration option. It reports rare inclusion and examines its associations, public review record, and relation to users’ training reports.

1 Introduction

  • The motivating example contrasts GSPO’s rapid uptake across five frameworks with only 2 of 43 other general-method papers that month reaching any framework.GSPO reached TRL three days after its first arXiv version, with four more frameworks following within a week.Source: §1 · p. 1
  • 37 of 1,598 general methods (2.3%) reached a framework; included papers more often used policies of 30B or more and modified existing losses, while code release was equally common.The size comparison was 41% versus 10%, and the loss-change comparison was 29% versus 9%; both gaps persisted in the reported robustness checks and remain associations.Source: §1 · p. 2
  • Across five quarterly cohorts, 180-day inclusions stayed at 5–7 as cohort size rose from 72 to 386 papers, a 5.4-fold increase.First landings by landing quarter rose from 3 to 10 per quarter between the start and end of 2025.Source: §1 · p. 2, Figure 1 · p. 2
  • Maintainers declined or redirected only 12 of 1,561 methods not included; 96% had no thread citing them, and many proposals that appeared stayed unanswered.Merged proposals drew design and testing concerns, while demand and maintenance concerns appeared mostly in unmerged proposals.Source: §1 · p. 2, Figure 1 · p. 2
  • Of 48 training-failure reports among 7,075 framework threads, the 13 large-model reports stated 8 of the 13 widely included methods and were the source for 2.The named failures in framework threads were distributed differently from failures addressed in paper abstracts.Source: §1 · p. 2
  • The released tracker links training failures to framework implementations and settings, and records contribution rules, reviewer concerns, and earlier proposal outcomes.The authors present it as a starting point for authors seeking framework inclusion.Source: §1 · p. 2

2 Related work

  • Prior work tracks industry and academic patterns, code availability, paper-repository forks, method performance, or open-source contribution decisions; this study traces paper methods into framework code and proposal records.The related work distinguishes whether a method works from whether a framework includes it.Source: §2 · p. 3

3 Measuring inclusion

The authors measure inclusion by tracing algorithm options and cited implementations in framework histories, then screen and label arXiv papers to define the method supply. They check their paper classifications and bound the possible effect of methods described only in model reports.

3 Measuring inclusion

  • The study tracks seven frameworks’ algorithm locations and inclusion tiers, from defaults and selectable algorithms to switches, experimental code, and example recipes.The seven frameworks were selected using criteria including public development and RL post-training support for language models.Source: §3 · p. 3
  • A catalog maps framework option names to 65 methods, and parsing their histories yields 426 events involving option additions, removals, or default changes.Unmapped detected options halt the build until they are assigned to a catalog entry.Source: §3 · p. 3
  • A full-history citation audit found framework files citing 51 general methods and added 9 cited implementations that parsing had missed.Two added implementations introduced no option: a NeMo-RL KL-gradient fix and a ROLL trajectory-level agent pipeline.Source: §3 · p. 4
  • Landing dates use main-branch commits, with dates from bulk releases, root commits, or renamings treated as upper bounds; lags start at the first public source date.For each landing, the study also compares commit authors with source authors and uses prior pull-request mergers as a proxy for maintainers.Source: §3 · p. 4

4 Measuring the supply of methods

  • The screened pool contains 6,939 records from three arXiv categories dated January 2025 to June 2026, selected when titles or abstracts mention language models and RL.The authors caution that this is a screened set, not a complete census of RL papers.Source: §4 · p. 4
  • A general method must make a new or modified training-time RL component its main contribution and present it for language-model RL broadly, rather than one task or application.The denominator is the set of papers meeting that general-method rule.Source: §4 · p. 4
  • Manual review agreed with labels on 188 of 200 sampled papers for the general-method split, while repeated labels matched on 129 of 140 papers.Recognition checks labeled 29 of 30 non-example included sources as methods.Source: §4 · p. 4
  • Counting every report with a transferable RL component as a general method would raise the denominator by at most about 13% and lower inclusion from 2.3% to 2.1%.The report sample included 30 papers labeled methods and 40 labeled otherwise, excluding reports with included methods.Source: §4 · pp. 4–5

5 Inclusion rate

Framework inclusion is rare, though included methods can reach main branches quickly. Uptake grew in absolute terms, but the share of each cohort included within 180 days fell as the paper pool expanded.

5 Inclusion rate

  • 37 of 1,598 general methods reached a framework; 7 reached at least three frameworks, and 1 became an unconditional default.Including 8 papers outside the abstract-based general-method rule gives 45 of 1,606, with 9 in three or more frameworks.Source: §5 · p. 5
  • 18 included papers had at least one landing by a named commit author outside the paper’s author list.Another 12 reached frameworks only through author commits, while 7 landed only through unresolved handles or sources without author lists.Source: §5 · p. 5
  • With a common window, 30 of 1,155 general methods with at least 180 days of exposure were included within 180 days; cohort inclusion shares fell from 6.9% to 1.3% as intake grew.First landings of general methods rose from 3 in the first quarter of 2025 to 10 in the fourth quarter.Source: §5 · p. 5
  • 30 days was the median time from first arXiv version to a main branch among 49 recent catalog methods with exact first landings.Across 42 source papers the median was 30.5 days; these medians describe included methods only.Source: §5 · pp. 5–6
  • 40 of 70 landings for methods included in three or more frameworks came from named people outside the paper authors.For methods in exactly one framework, 13 of 38 first landed through a paper author, most often as example recipes.Source: §5 · p. 6
  • TRL makes token-level loss aggregation a default and truncated importance sampling conditional on enabling vLLM generation; several KL corrections apply when the KL coefficient is non-zero.NeMo-RL applies Tang and Munos’s corrected KL gradient when its KL penalty is non-zero.Source: §5 · p. 6, Table 1 · p. 6

6 Features of included methods

Included methods were associated with larger policies and loss-level changes, while the public proposal record shows that most methods never surfaced for review. Among proposals that did appear, many remained open without a maintainer reply, and framework practices differed.

6 Features of included methods

  • Policies of 30B parameters or more appeared in 41% of included papers versus 10% of other papers; this scale comparison was descriptive, not a controlled causal comparison.The gap persisted after excluding model reports (12 of 29 versus 13 of 139), among papers with an industry author (10 of 25 versus 11 of 90), and when every unreported size was counted against the gap (12 of 31 versus 23 of 150).Source: §6 · pp. 6–7
  • Without papers from framework-maintaining companies, large policies appeared in 6 of 19 included papers versus 9 of 119 others, and coefficient changes in 6 of 21 versus 13 of 128.Code release was equally common in both groups, at 44% each; the descriptive comparison excluded papers that did not report a property from the relevant row.Source: §6 · p. 7, Figure 3 · p. 7
  • Methods changing clipping or the trust region, off-policy correction, or loss aggregation had 10 inclusions among 187 methods; rollout sampling and credit assignment had 4 among 554.The comparison covers framework options only.Source: §6 · p. 7, Figure 4 · p. 8

7 Proposals and merges

  • Across seven frameworks, 1,500 of 1,561 general methods included in no framework had no public record; maintainers turned away 12 of 1,561, because few were ever proposed.A missing public record does not show that maintainers never saw a paper.Source: §7 · pp. 8–9
  • Of 28 unmerged pull requests, 17 remained open after a median of 196 days, and 13 of those had no reply from a prior merger.Among 17 open threads without a prior-merger reply, 16 had no response beyond the proposer and automated reviewers.Source: §7 · p. 9
  • Someone other than the proposer raised a concern in 60 of 88 merged pull requests, and 41 still had an open concern when merged.Demand or maintenance-burden concerns appeared in 6 of 88 merged pull requests, 5 of 28 unmerged pull requests, and 6 of 20 requests; all unmerged pull requests came from TRL.Source: §7 · p. 9
  • Frameworks differed in how they handled proposals: prior mergers opened 13 of TRL’s 23 landing pull requests, while 12 of verl’s 20 proposals for other methods remained open and 7 lapsed.slime and OpenRLHF added methods mainly through prior mergers, and NeMo-RL reviewers requested training results in 8 of 13 merged pull requests.Source: §7 · pp. 9–10
  • Only 4 of 114 recent-method landings were later removed; a KL-gradient correction was first withdrawn after experiments showed no gain, then landed as a conditional default after a backward-pass fix.Pull requests for coefficient changes were about as large as those for new loss terms, despite coefficient changes often appearing as compact configuration fields.Source: §6 · p. 7, §7 · p. 10

8 Failures that users report

The study screened framework issues and discussions for user-reported training failures, distinguishing observed failures from bugs, usage questions, and feature requests. Public reports were few and did not mirror the failures most frequently addressed by papers.

8 Failures that users report

  • Of 7,075 screened framework threads, 48 reported a training failure, including 23 collapses or divergences without a named cause and 12 naming rollout-training probability mismatch.A random sample of 250 unscreened threads contained no failure report.Source: §8 · p. 10
  • Sparse reward and loss of diversity appeared in 218 and 156 general methods, respectively, but neither was named in a screened thread.The rollout-training probability mismatch was named by 12 threads and addressed by 20 general methods and 8 catalog methods.Source: §8 · p. 10
  • These counts measure failures users name publicly, not what they need; zero mentions do not show that nobody encountered a failure.The random-thread sample supports the screen, and the counts are lower bounds.Source: §8 · p. 10

9 Methods in large model reports

The paper compares methods stated in 13 large open-weight model reports with widely included framework methods, then draws implications for authors and maintainers from the observed record. The reports state few methods, and their overlap with framework code does not establish the direction of influence.

9 Methods in large model reports

  • The report pool covers 13 open-weight models with at least 100B parameters, requires RL in the title or abstract, and counts named methods or cited sources as stated.A change described under another name counts as a variant.Source: §9 · pp. 10–11
  • 6 of 13 reports state at least one of the 13 methods, 2 are a method’s source, and 3 describe a variant under another name.Token-level loss aggregation is stated most often, by 3 reports; the KL term is removed in 4.Source: §9 · p. 11
  • The reports’ overlap with framework code is strongest for stabilisers shipped by six or seven frameworks, but does not show the direction of influence.The reports state what their authors chose to write, and the comparison uses the latest report versions.Source: §9 · pp. 10, 12

10 Implications for researchers and maintainers

  • A public issue naming the training failure, proposed change, and evidence gives maintainers something to find and answer, though a record does not guarantee attention.22 of 94 landings followed an earlier issue or discussion that cited the paper; verl closed 7 of 20 proposals for other methods in scheduled cleanups.Source: §10 · p. 12
  • Frameworks set different contribution rules, and their written rules and review do the selecting; proposals that follow those rules can address concerns in advance.TRL asks for tests, documentation, and a paper-index entry; slime accepts changes verifiable in routine runs, while verl collects many methods as recipes.Source: §10 · p. 12
  • A configuration, seed, and short run on a public model let maintainers check that a method runs and shows its signal, while matching paper gains requires its baselines, data, and budget.NeMo-RL reviewers requested results in 8 of 13 merged pull requests, and TRL asked for reproduced results before adding a method to its main trainer.Source: §10 · p. 12
  • Frameworks included 10 of 187 methods on clipping, off-policy correction, and loss aggregation, versus 4 of 554 on rollout sampling and credit assignment.The latter methods can also live in user code outside the tracked framework options; rollout/training probability mismatch appeared in 12 failure reports and 6 method requests.Source: §10 · p. 12
  • A stated route for methods and a proposal template can help maintainers answer proposals and surface reviewer-raised items, while demand measured by requests is thin.A template covering failure, change, evidence, and configuration field would put in the first message items raised in 60 of 88 merged pull requests.Source: §10 · p. 13

11 Limitations

The study measures implementations on seven frameworks’ main branches, not use or user outcomes, and its comparisons and public records have additional scope limits. It releases a maintained tracker and concludes that the measurement could also be reused in other fields with shared training or evaluation code.

11 Limitations

  • Framework inclusion records implementations on main branches, not how often users employ them or what results they get.The numerator is a lower bound based on options the researchers can parse and implementations framework files cite; sampling and reward logic may sit outside tracked options.Source: §11 · p. 13
  • The comparison is observational and small, while public records miss private channels and bulk code releases and the failure counts are lower bounds.One reader coded review fields and failure reports; the latter cover titles and opening posts after keyword screening.Source: §11 · p. 13

12 The tracker

  • The maintained tracker links methods to included frameworks, implementation locations, contribution rules, review concerns, and earlier proposal outcomes.It refreshes records weekly, validates additions for human review, and includes inputs for reproducing the report’s numbers, tables, and figures without network access.Source: §12 · p. 13

13 Conclusion

  • The study concludes that other fields with shared training or evaluation code could reuse this measurement, which needs paper versions, implementation histories, and public proposals.The tracker lists framework algorithm locations, reviewer concerns, and earlier proposal outcomes.Source: §13 · p. 14

A Labelling rules

The appendix defines a paper-level labeling scheme for RL methods and related work, checks those labels against manual review, and describes the fields and exclusions used in full-text coding.

A Labelling rules

  • Each paper receives exactly one label based on its title and abstract.The general-method boundary concerns whether the component is presented for RL with language models at large rather than only a specific task or domain.Source: A Labelling rules · p. 19
  • A general RL method contributes a component for RL with language models broadly, while a method tied to one task, domain, or application is embedded or an application.Labels also separate analysis, systems, offline preference, distillation, and other work from RL methods.Source: A Labelling rules · pp. 19–20

B Manual review

  • Manual review found general methods in 68 of 76 papers labelled general, 3 of 16 labelled embedded, and 1 of 108 labelled otherwise.Weighting the stratum rates gives about 1,590 general methods; the main disagreement is between embedded methods and applications, outside the denominator.Source: B Manual review · p. 20

C Full-text coding

  • Full-text coding compared every included general method from 2025 with a quarter-stratified random sample of non-included methods, using blind recodes for the comparison.Coded fields include affiliations, code links, policy size, mixture-of-experts use, change form, hyperparameters, extra models, failure motivation, evaluation domains, and compute reporting.Source: C Full-text coding · pp. 20–21
  • Fields a paper did not state were left empty and excluded from the corresponding Figure 3 row, and papers without empirical evaluation were excluded from its evaluation-breadth row.The sensitivity table compares included with non-included papers and reports two versions excluding company-affiliated papers.Source: C Full-text coding · p. 21, Table 7 · p. 21

D Included papers outside the general rule

The appendix identifies included papers that fail the abstract-based general-method rule and explains why they are excluded from the main rate.

D Included papers outside the general rule

  • Eight included papers do not meet the general rule from their abstracts, so the main rate leaves them out to keep eligibility consistent with the comparison.Most are model reports or system papers that describe an included component only in the body; the attribution of masked importance sampling remains open among near-simultaneous sources.Source: D Included papers outside the general… · p. 21, Table 8 · p. 21

E The public record

The public-record analysis audits framework code, proposal threads, failure reports, and model reports using separate collection and coding procedures. Its thread records include proposals that were declined, redirected, withdrawn, left open, or closed through stale cleanup.

E The public record

  • The framework discussion record contains 26,960 issues, pull requests, and discussions downloaded through the GitHub API on 18 October 2026.A paper-thread match required a pool-verified identifier, link, or full title; formal review texts were read for coded threads and landing pull requests.Source: E The public record · p. 21
  • Nine methods missed by parsing appeared next to an implementation in framework files, while 10 appeared in project or news lists, two in report-PDF reference lists, and two in configuration notes or background references.The passage identifies a NeMo-RL KL-gradient fix and ROLL’s trajectory-level agent pipeline among the nine.Source: E The public record · p. 22
  • Proposal threads were coded for outcomes and explicit maintainer reasons, with a second reader coding blind to the first; the analysis uses the blind codes.The two codings agreed on 40 of 48 thread outcomes as raw agreement, with most differences involving redirects and withdrawals.Source: E The public record · p. 22
  • The failure screen searched issue and discussion openings since 1 January 2025 for keyword patterns covering the catalog’s 13 failures, then coded 648 matches.The model-report reading covered RL sections of reports in the pool for open-weight models with at least 100B total parameters; six named reports fell outside the pool through a non-systematic check.Source: E The public record · p. 22
  • The proposal table covers requests or proposals for general methods that no framework included, with outcomes and reasons coded blind from each thread’s own text.Examples include declined proposals for VeriFree and RLPR, and open proposals without a prior-merger reply for T-PPO and ESSA.Source: Table 9 · p. 22, E The public record · p. 23

F Framework selection

The framework selection applies Section 3’s criteria to open RL frameworks with at least 2,000 GitHub stars and records the commits analysed. The comparison also identifies qualifying frameworks outside the tracked set.

F Framework selection

  • The selection table gives the analysed commit for each tracked framework and applies the selection criteria to open RL frameworks with at least 2,000 GitHub stars.The verl recipe repository is analysed at commit 81d782765e from 2026-10-13.Source: F Framework selection · p. 24
  • The framework comparison distinguishes the seven tracked frameworks from other repositories meeting the same selection criteria.For frameworks outside the study, its count records general methods implemented at the listed commit that none of the seven include.Source: Table 10 · p. 24
  • The comparison’s snapshots are the tracked frameworks’ study commits and the latest commits of qualifying frameworks outside the study.The table caption defines this basis for the commit comparison.Source: Table 10 · p. 24, F Framework selection · p. 24

G Supplementary tables and figures

The supplementary materials specify how recent-method option names map to catalog methods and how the all-method timing figure is scoped. The figure measures time from first public version to main branch.

G Supplementary tables and figures

  • The option-name table covers methods included in three or more frameworks and maps registry keys, loss types, configuration fields, or recipe folders to catalog methods.The parser uses those names to identify each method’s option at the snapshot.Source: Table 11 · p. 25
  • The option-name table is a snapshot mapping, whereas Figure 5’s stated unit is a catalog method with a dated source from 2025 or later.The captions define these distinct scopes for reading the supplementary materials.Source: Table 11 · p. 25, Figure 5 · p. 26
  • Figure 5 measures time from a method’s first public version to a main-branch landing for catalog methods with a dated source from 2025 or later.Its marks follow the convention used in Figure 2.Source: Figure 5 · p. 26

H Method catalog

The method catalog records first public dates, earliest framework landings, and where methods are offered at the snapshot. Its entries include named methods and framework-originated changes as well as paper-sourced proposals.

H Method catalog

  • The catalog defines first landing as the earliest main-branch commit in any framework and lists where each method is offered at the snapshot.For dates marked with ≤, the landing date is an upper bound.Source: Table 12 · p. 27
  • The catalog includes framework-originated entries, including the KL gradient with the score-function term attributed to Tang and Munos and dated 2025-06-11 for first public and 2026-06-24 for first landing.NeMo-RL is listed as the framework for this entry.Source: H Method catalog · p. 28
  • GSPO, described as a sequence-level ratio, was first public on 2025-07-24 and first landed on 2025-07-27, and the catalog lists it in all seven frameworks.The catalog attributes it to Zheng et al. [88].Source: H Method catalog · p. 28
  • SAO is listed as direct double-sided importance sampling associated with ROLL, with Fu et al. [20] and dates of 2025-05-30 first public and 2025-05-26 first landing.The catalog records AReaL as its framework.Source: H Method catalog · pp. 28–29
Loading 2610.28027v1…