PaperlayerView exact revision on arXiv

Source-linked AI summary

DataFlex-RL: An Evaluation Platform for RLVR Data Policies

Hao Liang, Mingrui Chen, Hengyi Feng, Meiyi Qiang, Wentao Zhang

arXiv:2609.06107v1cs.LGcs.CL

TL;DR

Whether RLVR data policies provide reproducible gains over uniform sampling remains unclear when methods are embedded in changing training stacks and evaluation choices. DataFlex-RL compares policy families under a common GRPO recipe and matched evaluation. The experiments find no clear policy winner at the measured precision, and rankings can change when evaluation coverage changes.

  • Problem

    Existing evidence does not establish whether directing RLVR training toward selected data yields gains robust to changes in model and training conditions.

    Source: §1 · p. 3

  • Method

    DataFlex-RL separates selection, reweighting, and mixture adaptation from their signals and evaluates them under a shared GRPO stack with matched seeds and 12-benchmark scoring.

    Source: §1 · pp. 3–4

  • Results

    None of eight Qwen2.5-7B-base selection or reweighting methods or three adaptive mixtures clears paired-interval evidence against its control; uniform GRPO improves Overall accuracy by 7.76 points.

    Source: §1 · p. 4, §7 · p. 12

  • Takeaways & Limitations

    Under the controlled recipes tested, the evaluated policies do not provide a reproducible improvement over uniform training, and benchmark coverage can change the apparent winner.

    Source: §7 · p. 12

    The results do not establish equivalence or rule out gains under longer training, other hyperparameters, or different policy families.

    Source: §7 · p. 12

Abstract

from arXiv · show

Data policies for reinforcement learning with verifiable rewards (RLVR) determine which rollouts are used, how strongly they are weighted, and which domains contribute to subsequent training batches. We introduce DataFlex-RL, an evaluation platform for comparing these choices under a common GRPO recipe. Our primary experiment evaluates 13 configurations across 12 matched seeds using Qwen2.5-7B-Base and 12 mathematics, logic, and science benchmarks. Uniform GRPO improves the domain-balanced average accuracy by 7.76 percentage points over the untrained checkpoint. None of the eight rollout-selection or reweighting methods achieves a paired 95% confidence interval that excludes zero relative to uniform sampling, and none of the three adaptive mixtures outperforms a fixed equal mixture at the same level of precision. A corrected 12-seed extension on Llama-3.1-8B-Base places the additional methods on the same score scale as the original controls, but does not reveal a consistent winner in terms of observed mean performance. We also quantify evaluation sensitivity by rescoring nine Qwen2.5-7B-Instruct runs using a math-heavy six-benchmark summary, consisting of five mathematics benchmarks and GPQA-Diamond but no logic benchmark, and comparing it with the domain-balanced 12-benchmark summary. The resulting rankings are negatively correlated, with a correlation coefficient of -0.33, whereas summaries that retain all 12 benchmarks largely agree. Across the controlled settings studied here, changing the data policy measurably changes the training process but does not produce a reproducible improvement over uniform training.

1 Introduction

The paper asks whether data policies yield reproducible gains when the GRPO recipe is held fixed, and introduces DataFlex-RL to compare them. Its controlled results show no clear policy winner and show that evaluation coverage can change the apparent winner.

1 Introduction

  • Prior evidence does not establish that directing training toward supposedly more useful data yields gains robust to changes in models and training conditions.Data policies are often bundled with changes to the model, prompt, verifier, rollout budget, or optimization settings, complicating attribution.Source: §1 · p. 3
  • DataFlex-RL separates intervention families from their signals and compares them with a shared GRPO stack, matched seeds, and a common 12-benchmark score format.Its taxonomy distinguishes selection, reweighting, and mixture adaptation as changes to different parts of training.Source: §1 · pp. 3–4
  • On Qwen2.5-7B-base, none of eight selection or reweighting methods or three adaptive mixtures clears paired-interval evidence; their mean spreads are 0.97 and 0.61 point versus GRPO’s 7.76-point gain.The extension to Llama-3.1-8B-base also finds no common winner across the expanded methods.Source: §1 · p. 4
  • Math-Heavy-6 and domain-balanced DB-12 rankings correlate at ρ = −0.33, while summaries retaining all 12 benchmarks largely agree.Method spread changes from 0.90 under Math-Heavy-6 to 3.31 under DB-12.Source: §1 · p. 4

2 Related Work

Related work spans on-policy RLVR data interventions, data-centric training policies, and evaluation infrastructure; DataFlex-RL brings these intervention comparisons into a matched protocol.

2.1 Data Processing for RLVR

  • RLVR selection and reweighting methods use related signals but make distinct interventions: selection zeros a response’s contribution, while reweighting changes it continuously.Examples include DAPO [31], PODS [28], GFPO [21], Advantage Reweighting [30], and prioritized experience replay [19].Source: §2.1 · pp. 4–5

2.2 Data-Centric Training

  • Offline policies such as DSIR [27] and DoReMi [26] set or optimize data distributions before training, while Skill-It [3] and DUMP [24] adapt curricula from observed progress.DataFlex-RL transfers the selection–mixture–reweighting abstraction from supervised LLM training to on-policy RL [11].Source: §2.2 · p. 5

2.3 Evaluation and Reproducibility

  • Evaluation harnesses standardize prompts and scoring, while reproducible RL comparisons also require matched seeds, intervals, robust aggregation, and complete procedures.The paper retains task-specific harnesses including GPQA [18], MMLU-Pro [23], and ZebraLogic [13].Source: §2.3 · p. 5

3 Three Data-Policy Interventions

The paper classifies data policies by where they intervene in GRPO and separately identifies the signals used to choose those interventions.

3 Three Data-Policy Interventions

  • RLVR data policies intervene by removing responses from the current update, changing their loss contribution, or changing the domain distribution for future prompts.These are the selection, reweighting, and mixture-adaptation families, respectively.Source: §3 · p. 5

3.1 Selection, reweighting, and mixture adaptation

  • Selection and reweighting act after rollout, whereas mixture adaptation sets domain probabilities before the next rollout and leaves the per-response loss unchanged.The fixed mixture assigns probability 1/3 to each of the three domains; selection can reduce update tokens, while reweighting preserves mean weight one.Source: §3.1 · pp. 5–6

3.2 Implemented methods and controlled comparisons

  • Signals include group solve rate, response advantage, token probability, and domain-level reward, advantage, or reward-slope summaries over rolling windows.Per, softmax, and topk all rank by response advantage, allowing comparison of continuous weighting against keep-or-discard selection.Source: §3.2 · p. 6
  • The 13 logged configurations are grouped by intervention; several adapt published proposals, while canonical mixture implementations use distinct signals.Twelve earlier proxy runs are retained for provenance but excluded from canonical means.Source: §3.2 · p. 6, Table 1 · p. 7

4 Experimental Setup

The experiments use a shared training and evaluation protocol, with a balanced math, logic, and science corpus and a run matrix spanning primary comparisons and extensions.

4 Experimental Setup

  • Comparisons hold the model, corpus, verifier, optimizer, rollout budget, and evaluation protocol fixed, using a shared driver and standardized run records.A campaign generator expands model–method–seed combinations, and the same records feed reconstruction and statistical-analysis scripts.Source: §4 · p. 7

4.1 Data, training, and evaluation

  • The 15,000-prompt corpus is split equally among math, logic, and science; training uses GRPO with five rollouts per prompt and 300 optimizer steps unless specified otherwise.Evaluation covers 12 benchmarks across the three domains; calibration checks answer parsing, and an audit finds no exact or normalized training–evaluation matches.Source: §4.1 · pp. 7–8

4.2 Run coverage

  • The experimental matrix contains 591 runs, including 108 additional Qwen2.5-7B-base runs; its primary block completes all 13 configurations with 12 seeds each, including a campaign-local adaptive-mixture comparison.The matrix also has a 171-run breadth study, 108 runs extending three baseline–selector comparisons from 3 to 12 seeds, 33 mechanism and strength controls, and 171 base-model scale and family runs from Revision Plan 6.Source: §4.2 · p. 8

5 Main Result on a Base Model with Training Headroom

The Qwen2.5-7B-base comparison evaluates 13 configurations with 12 matched seeds, measuring selection and reweighting against uniform sampling and adaptive mixtures against a fixed equal mixture. Uniform GRPO yields substantial gains, but the tested data policies do not show reproducible improvements over their controls.

5.1 Training Headroom

  • +7.76 points on Qwen-base and +10.25 points on Llama-base are the uniform-GRPO gains; the Qwen2.5-7B-base block combines training headroom, complete method coverage, and 12 matched seeds.The Qwen comparison is the primary inferential setting.Source: §5.1 · p. 8
  • Table 3’s Overall gives equal weight to MATH, LOGIC, and SCIENCE; methods relatively strong on LOGIC are not uniformly strong on MATH or SCIENCE.The table averages each benchmark across 12 matched seeds.Source: Table 3 · p. 9, §5.1 · p. 9

5.2 Selection and Reweighting Add No Reproducible Gain

  • None of the eight selection or reweighting methods has a paired interval excluding zero versus uniform sampling; the nine policy means span 0.97 point, compared with GRPO’s 7.76-point gain.Table 4 reports paired differences and intervals relative to uniform sampling.Source: Table 4 · p. 9, §5.2 · p. 9

5.3 Adaptive Mixtures Show No Reproducible Gain

  • All three adaptive mixtures have positive point estimates versus the fixed mixture, but every paired interval includes zero and their means span only 0.61 point.Table 5 reports differences and paired intervals relative to the fixed, equal static mixture.Source: Table 5 · p. 10, §5.3 · p. 10

6 Cross-Model and Evaluation Checks

Corrected cross-model checks show no common method winner, while omitting a domain can change the apparent leader. Summaries retaining all 12 benchmarks largely agree.

6.1 Llama-3.1-8B-base Evaluation

  • 10.25 points: uniform GRPO raises Llama-3.1-8B-base Overall from 10.87 to 21.12 across 12 matched seeds.The 95% CI for the gain is [7.75, 12.33].Source: §6.1 · p. 10, Table 6 · p. 10
  • Table 6 reports twelve-benchmark means for Llama-3.1-8B-base, averaging 12 matched seeds per row; Overall equally weights MATH, LOGIC, and SCIENCE domain means, and the final column gives the 95% t interval for the seed-level Overall mean.Source: Table 6 · p. 10
  • The nine additional corrected Llama methods range from 18.66 Overall for softmax to 21.98 for diffband, against a 21.12 reusable baseline, without a universal winner.The reported intervals describe each method’s seed-level mean, not paired method comparisons.Source: §6.1 · p. 11

6.2 Base-Model Scale and Family Coverage

  • Across the three-seed auxiliary model sweep, difffilter and tscl change direction by model, while maxvar remains below baseline in every row.These rows describe coverage and direction rather than standalone significance tests.Source: §6.2 · p. 11

6.3 Method Rankings Depend on Evaluation-Domain Coverage

  • Math-Heavy-6 ranks topk and diffband first, whereas DB-12 ranks gfpo and difffilter first; their rankings correlate at ρ = −0.33.Math-Heavy-6 omits logic, while Macro-12 and Item-12 correlate with DB-12 at ρ = 0.88 when all 12 benchmarks are retained.Source: §6.3 · pp. 11–12, Table 8 · p. 12

6 Cross-Model and Evaluation Checks

  • Corrected cross-model checks find no common winner, and omitting a domain can change the apparent leader even though full-12-benchmark summaries largely agree.The corrected Llama evaluation and base-model sweep show no common winner in observed means; the coverage analysis changes the apparent winner on the same runs.Source: §6.3 · p. 12

7 Conclusion

DataFlex-RL compares selection, reweighting, and mixture policies against uniform training under a shared GRPO recipe. Across the controlled settings studied, it finds no reproducible improvement over uniform training at the achieved precision.

7 Conclusion

  • On Qwen2.5-7B-base, uniform GRPO improves Overall accuracy by 7.76 points, but tested selection, reweighting, and adaptive-mixture comparisons show no reproducible policy advantage.None of eight selection or reweighting methods has a paired interval excluding zero versus uniform sampling, and none of three adaptive mixtures does versus a fixed equal mixture.Source: §7 · p. 12
  • On Llama-3.1-8B-base, GRPO improves by 10.25 points (95% CI [7.75, 12.33]), while corrected additional-method means reveal no clear common winner.The corrected extension puts nine additional methods on the same score scale as the original controls.Source: §7 · p. 12
  • The conclusion is limited to the controlled recipes tested and does not establish equivalence or rule out gains under longer training, other hyperparameters, or different policy families.Released records, logs, configurations, and reconstruction scripts support auditing and reuse.Source: §7 · p. 12

A.1 Sensitivity to the Evaluation Summary

Evaluation-summary choices can change method rankings when a domain is omitted, while alternative aggregations retaining all 12 benchmarks largely agree.

A.1 Sensitivity to the Evaluation Summary

  • DB-12 averages the MATH, LOGIC, and SCIENCE domain scores, giving each domain equal weight.Math-Heavy-6 instead averages five math benchmarks and GPQA-Diamond while omitting logic.Source: §A.1 · p. 15
  • Math-Heavy-6 and DB-12 rankings are negatively correlated (Spearman ρ = −0.33), while Macro-12 and Item-12 each correlate with DB-12 at ρ = 0.88.Macro-12 weights benchmarks equally; Item-12 weights them by evaluation-set size.Source: §A.1 · p. 15
  • The ranking change is driven primarily by omitting logic, rather than by weighting choices among summaries that retain all 12 benchmarks.Leave-one-benchmark-out correlations range from 0.78 to 0.98.Source: §A.1 · p. 15

A.2 Selector Diagnostics

Three-seed selector controls produce method-dependent shifts rather than a common direction. A separate gfpo analysis reports nearly unchanged accuracy alongside shorter responses and more correct answers per thousand characters.

A.2 Selector Diagnostics

  • Matched random preserves the number of groups or responses while choosing randomly; alternative strengths widen difffilter’s band or change maxvar and topk keep fractions.The diagnostics use the full nonzero solve-rate band for difffilter and keep fractions of 0.25 or 0.75 for maxvar and topk.Source: §A.2 · p. 15, Table 9 · p. 15
  • Targeted selection exceeds matched random for maxvar and topk, but not difffilter; matching update tokens raises difffilter and maxvar, but lowers topk.Table 9 compares selectors with controls for retention count, update volume, and intervention strength.Source: §A.2 · p. 15, Table 9 · p. 15
  • 72.21% for gfpo versus 72.19% for baseline on five math benchmarks; gfpo also reduces mean response length from 1568 to 1497 characters and raises correct answers per thousand characters from 0.461 to 0.482.Source: §A.2 · p. 15
Loading 2609.06107v1…