PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Beyond What You Sample: Off-Policy-Aware Cross-Model Trajectory Exchange for RLVR

Doohyuk Jang, Yoonsik Park, Gyouk Chu, Sihwan Park, Eunho Yang

arXiv:2609.37868v1cs.LGcs.AIcs.CL

TL;DR

Finite rollout budgets can leave RLVR methods without a reward-based signal, even when another heterogeneous model has found a successful trajectory. GRAFT replaces all-fail groups with peer trajectories while controlling cross-model mismatch. Across three model pairs and five math benchmarks, it improves both models over GRPO, and most gains persist with stored peer trajectories.

  • Problem

    Finite rollout budgets can produce all-fail groups with no reward-based policy-gradient signal, while successful trajectories missing from one model’s rollouts may already have been found by another.

    Source: §1 · p. 1

  • Method

    GRAFT replaces receiver all-fail groups with peer groups, retains their source-computed advantages, and controls cross-model influence with compatibility gating and token-level importance-ratio clipping.

    Source: §1 · p. 2

  • Results

    Across three heterogeneous model pairs and five mathematical reasoning benchmarks, GRAFT improves both models in every pair over GRPO (n=8), by 2.1 points on average and up to 4.5 points in model-level average score.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Stored peer trajectories preserve most gains, improving over GRPO by 1.8 points on average without simultaneous co-training or additional peer rollouts.

    Source: §1 · p. 2

    The study covers only two-model pairs, math, and base models of at most 3B parameters; across tokenizers, compatibility is a proxy rather than a density ratio.

    Source: §7 · p. 10

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) methods such as GRPO rely on successful self-generated trajectories, but finite rollout budgets can produce all-fail groups with no reward-based policy-gradient signal. While additional rollouts improve the chance of success at higher cost, successful trajectories missing from one model's rollouts may already have been discovered by another. Indeed, we observe that heterogeneous models often succeed on complementary prompts, creating opportunities for mutual learning without a designated stronger teacher. To exploit this complementarity, we propose GRAFT (Gated Replacement of Answer-Failed groups with peer Trajectories), an off-policy-aware framework that replaces all-fail groups with informative peer groups. GRAFT transfers both successful and unsuccessful peer responses with peer-computed advantages, while controlling cross-model mismatch through sequence-level compatibility weighting and token-level importance ratio clipping. Across three heterogeneous model pairs and five mathematical reasoning benchmarks, GRAFT consistently improves both models over GRPO with the same per-model rollout budget, gaining 2.1 points on average and up to 4.5 points in model-level average performance. Stored peer trajectories preserve most of the gains, improving over GRPO by 1.8 points on average without simultaneous co-training.

1 INTRODUCTION

Finite rollout budgets can leave RLVR groups without a reward-based learning signal, while heterogeneous models may discover complementary successes. GRAFT replaces receiver all-fail groups with peer trajectories and controls their off-policy influence; it improves both models over GRPO across the reported pairs and benchmarks.

1 INTRODUCTION

  • All-fail groups leave every group-relative advantage at zero, while larger groups improve success odds only at higher rollout cost and still rely on the learner’s own exploration.Other approaches described here also remain dependent on the learner’s own exploration.Source: §1 · p. 1
  • SmolLM3-3B-Base solves 47.9% of Qwen3-1.7B-Base’s all-fail prompts, while Qwen3-1.7B-Base solves 18.7% of SmolLM3-3B-Base’s.These observations come from independent GRPO runs.Source: §1 · pp. 1–2
  • Prompt selection and learning from peer responses must be paired: transferring to prompts with informative self-generated groups changes an update that already has an on-policy signal.HACPO shares beyond receiver-failure prompts, while SGT uses a fixed-weight supervised loss without compatibility-based weighting; applying either to the selected prompts still underperforms GRAFT.Source: §1 · p. 2
  • GRAFT replaces receiver groups where all rollouts fail with peer groups containing successful and unsuccessful responses, retaining source-computed advantages and balancing exchange directions.It controls mismatch with bounded sequence-level compatibility gating and token-level importance-ratio clipping, then processes peer-containing minibatches after on-policy minibatches.Source: §1 · p. 2
  • 2.1 points on average and up to 4.5 points in model-level average score over GRPO (n=8) across three model pairs and five mathematical reasoning benchmarks.GRAFT improves both models in every pair; gains also largely persist with stored peer trajectories (+1.8 points on average) without simultaneous co-training or additional peer rollouts.Source: §1 · p. 2

2 RELATED WORK

Prior RLVR exploration methods rely on the learner’s own sampled trajectories or add rollout cost, while external guidance introduces policy mismatch. Cross-model trajectory-sharing methods address parts of the problem, and GRAFT combines receiver-failure prompt selection with mismatch-aware peer learning.

2 RELATED WORK

  • Dynamic sampling discards zero-variance groups and larger groups improve success coverage, but both cost extra rollouts without guaranteeing success; entropy-guided shaping adds a signal without supplying a correct response.Hints, partial solutions, and expert guidance require an external solution source or stronger model.Source: §2 · pp. 2–3
  • External demonstrations and historical trajectories introduce policy mismatch; GRAFT uses token-level importance-ratio clipping and bounded sequence-level compatibility filtering, whose score is only a proxy across tokenizers.The compatibility score is based on average token log-likelihoods, not an exact cross-tokenizer importance ratio.Source: §2 · p. 3
  • HACPO exchanges peer rollouts with off-policy correction, SGT transfers verified peer successes on receiver-failure prompts, and F-TIS studies collaborative GRPO with a shared vocabulary.GRAFT jointly addresses which peer trajectories provide missing supervision and how their influence is controlled under cross-model mismatch.Source: §2 · p. 3

3 PRELIMINARIES

GRPO assigns verifiable rewards to sampled response groups and computes group-relative advantages, but zero-variance groups provide no policy-gradient signal. Cross-model sharing raises questions about which peer trajectories to transfer and how to optimize them under policy mismatch.

3 PRELIMINARIES

  • GRPO samples n responses for a prompt from an old policy, assigns each a verifiable binary reward, and computes a group-relative advantage for each response.The responses are represented as token sequences, with tokenizers made explicit in Section 4.Source: §3 · p. 3
  • When a response group is entirely correct or entirely incorrect, its reward standard deviation is zero, all advantages become zero, and it contributes no policy-gradient signal.The section notes token-level loss aggregation with asymmetric clipping following DAPO.Source: §3 · p. 3
  • GRAFT selects and balances complementary peer groups, then applies compatibility weighting, token-level clipping, and peer-last updates to control off-policy influence.The token-level importance ratio compares the current policy with the old policy for a response token and its preceding context.Source: §3 · p. 3, Figure 2 · p. 4
  • Cross-model sharing poses two design questions: which peer trajectories should supplement receiver rollouts, and how to optimize them under cross-model policy mismatch.HACPO broadly reuses peer rollouts; SGT transfers a verified peer success when the receiver’s entire group fails and uses an auxiliary negative log-likelihood objective.Source: §3 · pp. 3–4

4 GRAFT: GATED REPLACEMENT OF ANSWER-FAILED GROUPS WITH PEER TRAJECTORIES

GRAFT replaces receiver groups that fail entirely with complementary peer groups, then controls how those off-policy responses influence the receiver. Its selection, compatibility weighting, and update ordering address both when to exchange trajectories and how to learn from them.

4 GRAFT: GATED REPLACEMENT OF ANSWER-FAILED GROUPS WITH PEER TRAJECTORIES

  • GRAFT replaces a receiver’s all-fail group with a peer group and uses source-computed advantages while controlling cross-model mismatch.It addresses both which trajectories to transfer and how the receiver learns from them.Source: §4 · p. 4

4.1 COMPLEMENTARY AND BALANCED GROUP REPLACEMENT

  • Transfer requires a receiver group with no successes and a peer group containing both successful and unsuccessful responses; the full peer group and its original advantages are transferred.Keeping both response types preserves reward contrast without pooling rewards across models.Source: §4.1 · pp. 4–5
  • To balance exchange directions, GRAFT selects the smaller candidate count in each direction, ranking prompts by peer success count and retaining boundary ties.Source: §4.1 · p. 5

4.2 OFF-POLICY-AWARE PEER UPDATES

  • For peer trajectories, GRAFT compares average token log-likelihoods as a compatibility score, drops scores at or below δ, and caps each admitted response’s weight at one.The score is interpreted as a length-normalized likelihood ratio when tokenizations match and as a compatibility score when they differ.Source: §4.2 · p. 5
  • The receiver tokenizes peer strings with its own tokenizer and uses token-level importance ratios for both self-generated and grafted responses, with peer advantages and compatibility weights held fixed.Source: §4.2 · p. 6
  • Grafted minibatches follow on-policy minibatches so ratio clipping can moderate peer contributions after the receiver’s own data has priority.The factorization motivates separate controls for within-receiver policy change and initial cross-model mismatch; it is not an exact importance-weighted objective.Source: §4.2 · pp. 5–6

5 EXPERIMENTS

The experiments compare GRAFT with budget-matched and larger-rollout GRPO baselines, as well as cross-model methods, across three heterogeneous model pairs and five mathematical benchmarks. GRAFT improves aggregate performance across all six model blocks, and the reported gains persist with stored peer trajectories.

5.1 SETUP

  • The study evaluates three heterogeneous model pairs with n=8 responses per prompt, fixed training data, learning rate, and steps, and pass@1 on five mathematical benchmarks.Baselines include GRPO with n=8, 16, and 32 rollouts, plus HACPO and SGT; the compatibility threshold is δ = 0.8 unless stated otherwise.Source: §5.1 · pp. 6–7

5.2 MAIN RESULTS

  • GRAFT gains +0.66 to +4.46 in average score across all six model blocks; both Pair 1 models beat GRPO (n=32), and Pair 2 models are comparable.Source: Table 1 · p. 7, §5.2 · p. 7
  • GRAFT exceeds GRPO (n=32) by 1.18 at 0.45× the cost in Pair 1; stored trajectories cut compute by 27–76% while retaining positive gains in all six blocks.Stored trajectories retain 84% of the online gain on average.Source: Figure 3 · p. 8
  • HACPO falls below budget-matched GRPO in five of six blocks by as much as −4.17 in average score, while SGT ranges from −0.53 to +2.28.GRAFT’s margin over budget-matched baselines emerges early and persists through training.Source: §5.2 · p. 8
  • Qualitative case studies report shared solution steps between GRAFT and peer responses, including root shifting and inclusion–exclusion, alongside receiver GRPO solution structure.Source: §5.2 · p. 8

6 ANALYSIS

GRAFT’s gains are examined through compute cost, reuse of stored peer trajectories, and ablations of its design. The analyses compare peer exchange with larger rollout budgets and test how prompt selection and peer updates work together.

6.1 COMPUTE-EFFICIENT GAINS FROM PEER EXCHANGE

  • At 40.9 GPU-hours, GRAFT reaches a pair-mean score of 35.25, exceeding GRPO (n=32) by 1.18 points at 0.45× its compute and GRPO (n=16) by 2.48 points at comparable compute.This comparison counts the joint cost of obtaining both Pair 1 models.Source: §6.1 · p. 8

6.2 USING STORED PEER TRAJECTORIES DURING TRAINING

  • Stored peer trajectories improve over GRPO (n=8) in all six model blocks by 1.78 points on average, versus 2.11 for online exchange.The stored responses and log-probabilities come from the partner’s independent GRPO run; transfer is unidirectional, so balancing is inactive. Pair 1 receiver checkpoints cost 23.2 GPU-hours, excluding the prior runs used to collect peer logs.Source: §6.2 · pp. 8–9

6.3 ABLATIONS AND ALTERNATIVE DESIGNS

  • Removing compatibility weighting causes the largest ablation drop for SmolLM3 (−8.36 points), while removing the floor causes the largest drop for Qwen3 (−3.56).Removing balancing, using sequence-level rather than token-level ratios, count-matched random prompts, or random admission also lowers performance; peer-last ordering beats peer-first and uniform ordering.Source: §6.3 · p. 9
  • Pooling peer and self groups, transferring only successes, or using HACPO, SGT, or LUFFY-style updates underperforms full GRAFT.With GRAFT’s update but SGT’s unbalanced selection, the gap is 2.00/0.48 points; the floor improves aggregate scores only under full-group transfer.Source: §6.3 · p. 9

7 CONCLUSION

GRAFT replaces receiver groups that fail entirely with peer groups and uses compatibility-aware, clipped updates to control cross-model mismatch. Its studied scope is two-model pairs on mathematical reasoning with base models of at most 3B parameters, and its compatibility score is only a proxy for a density ratio.

7 CONCLUSION

  • GRAFT consistently improves both models over same-budget GRPO across three heterogeneous pairs and can match or exceed GRPO with substantially more rollouts.It replaces all-fail receiver groups with informative peer groups and controls mismatch through compatibility-aware, clipped updates; most gains persist with stored peer trajectories.Source: §7 · pp. 9–10
  • The gain depends on model complementarity and is smallest on Pair 3; the study covers only two-model pairs, math, and base models up to 3B parameters.Across tokenizers, the compatibility score is a proxy rather than a density ratio; exchanges among more than two peers and domains without verifiable rewards are left for future work.Source: §7 · p. 10

A FULL ALGORITHM OF GRAFT

  • GRAFT selects prompts where the receiver fails entirely and the peer has both successful and unsuccessful responses, then balances exchange across directions.Selection retains tied prompts at the boundary; it uses peer-computed advantages and admits peer responses above a compatibility floor with weights, while unselected groups retain receiver responses and advantages.Source: A FULL ALGORITHM OF GRAFT · p. 13
  • The receiver update uses source-computed advantages, compatibility weights, token-level ratios, and peer-containing minibatches placed last.Advantages and compatibility weights remain fixed during optimization.Source: A FULL ALGORITHM OF GRAFT · p. 13

B TRAINING AND EVALUATION DETAILS

Training uses binary verification rewards on MATH problems, with shared settings for GRAFT and independent GRPO unless specified otherwise. Models are validated under a fixed checkpoint-selection rule, and the highest-scoring checkpoint is reported for every method and rollout budget.

B TRAINING AND EVALUATION DETAILS

  • Training uses 7,500 problems from the MATH training split across all difficulty levels, with binary reward from verifying the final boxed answer.No additional format or length reward is used, and responses reaching the generation limit are scored as generated.Source: B TRAINING AND EVALUATION DETAILS · p. 14
  • Both models receive the same prompt batch; each rollout batch gets one optimization epoch divided into four minibatches of 32 prompts.The implementation uses token-mean loss aggregation, dynamic microbatching, and gradient checkpointing.Source: B TRAINING AND EVALUATION DETAILS · p. 14
  • GRAFT uses δ = 0.8 for all three pairs, selected on Pair 1 and fixed before Pairs 2 and 3; it adds no rollouts.Its additional computation is receiver-side likelihood evaluation of transferred responses; larger-budget GRPO changes only rollout count.Source: B TRAINING AND EVALUATION DETAILS · p. 14, Table 3 · p. 14
  • Evaluation covers MATH500, AIME2024, AIME2025, AMC23, and Minerva Math; the aggregate is the unweighted mean of the five benchmarks.Pass@1 averages correctness over responses and then over problems; evaluation uses eight responses per problem.Source: B TRAINING AND EVALUATION DETAILS · p. 15
  • Every model is validated every five steps on aggregate score, and the highest-scoring checkpoint is reported under the same rule for every method and rollout budget.Figure 4 plots five-benchmark average against training step; selected checkpoints are evaluated over five independent runs, while main-table results use the first training run.Source: B TRAINING AND EVALUATION DETAILS · p. 15, Figure 4 · p. 15

C TRAINING STABILITY ACROSS INDEPENDENT RUNS

Across three independent training runs for each of three model pairs, GRAFT retains higher mean aggregate performance than GRPO (n=8) in all six model blocks.

C TRAINING STABILITY ACROSS INDEPENDENT RUNS

  • GRAFT has higher mean aggregate performance than GRPO (n=8) in all six model blocks across independent training runs.Table 4 reports means and sample standard deviations across three runs per method and model pair.Source: C TRAINING STABILITY ACROSS… · p. 16, Table 4 · p. 16
  • Both GRPO (n=8) and GRAFT were independently repeated three times for each of the three model pairs.Each resulting checkpoint was evaluated over five runs with eight samples per prompt.Source: C TRAINING STABILITY ACROSS… · p. 16
  • Table 4 reports mean performance and sample standard deviation across training runs, with differences measured against GRPO (n=8) within each model block.Source: Table 4 · p. 16

D COMPUTE ACCOUNTING

The compute accounting distinguishes checkpoint costs from full-run costs and compares performance against cost under a conservative per-model charge for cross-model training.

D COMPUTE ACCOUNTING

  • GRAFT exceeds GRPO (n=32) by 1.35 points for SmolLM3 and 1.01 points for Qwen3 at approximately 0.84× and 0.91× their respective checkpoint costs.Under the same accounting, GRAFT improves over GRPO (n=16) by 3.68 and 1.29 points at approximately 1.95× and 1.78× the cost.Source: Figure 5 · p. 17, D COMPUTE ACCOUNTING · p. 17
  • GPU-hours equal four GPUs times summed per-step training time, excluding validation and checkpoint writing.Costs are reported both through selected checkpoints and for all three epochs (174 steps).Source: D COMPUTE ACCOUNTING · p. 17
  • For online cross-model methods, joint checkpoint cost runs through the later selected checkpoint; Figure 5 instead charges each model the full joint-run cost through its own selected checkpoint.Independent GRPO's paired checkpoint cost sums the two separate runs.Source: D COMPUTE ACCOUNTING · p. 17, Figure 5 · p. 17, Table 5 · p. 18

E FULL RESULTS FOR STORED TRAJECTORY EXPERIMENTS

Stored peer trajectories improve average scores over GRPO across all six model blocks, retaining most of the online GRAFT gains without simultaneous co-training.

E FULL RESULTS FOR STORED TRAJECTORY EXPERIMENTS

F FULL RESULTS FOR ABLATIONS AND ALTERNATIVE DESIGNS

Ablations show that tested alternatives reduce aggregate scores relative to full GRAFT, while peer-minibatch order affects clipping and examples illustrate how GRAFT responses combine solution strategies.

F FULL RESULTS FOR ABLATIONS AND ALTERNATIVE DESIGNS

  • All listed variants lower aggregate scores relative to full GRAFT for both models, although individual benchmark scores can improve.Compatibility weighting has the largest degradation for SmolLM3; removing only the floor has the largest among tested variants for Qwen3.Source: F FULL RESULTS FOR ABLATIONS AND… · p. 20, Table 7 · p. 20
  • The tested threshold δ = 0.8 yields the highest aggregate score for both receivers; count-matched random selection also reduces aggregate performance.The ablations keep the compute budget fixed at n=8 rollouts per model.Source: F FULL RESULTS FOR ABLATIONS AND… · p. 20, Table 7 · p. 20

G IMPLEMENTATION OF ALTERNATIVE DESIGNS

  • Alternative designs selectively substitute prior-method prompt or update rules, including HACPO and SGT variants and LUFFY-style peer weighting.Other tests use count-matched random prompt selection or random response admission while retaining aspects of GRAFT's compatibility weighting.Source: G IMPLEMENTATION OF ALTERNATIVE DESIGNS · p. 22
  • The pooled-groups variant retains receiver and peer responses and computes advantages over their concatenation, changing advantage normalization, optimized responses, and loss normalization.Source: G IMPLEMENTATION OF ALTERNATIVE DESIGNS · p. 22

H CLIPPING DYNAMICS UNDER PEER MINIBATCH POSITION

  • Peer-last ordering clips 0.28–0.44% of grafted-response tokens, versus 0.13–0.25% under uniform ordering and less than 0.01% under peer-first ordering.The authors note this pattern is consistent with earlier receiver updates moving peer-token ratios away from one before peer optimization.Source: H CLIPPING DYNAMICS UNDER PEER… · p. 23, Table 9 · p. 23

I QUALITATIVE ANALYSIS

  • The qualitative analysis presents three examples per target model, pairing each target's GRPO response with its peer's GRPO response and the target's GRAFT response.The examples cover six cases across Qwen3-1.7B and SmolLM3-3B.Source: I QUALITATIVE ANALYSIS · p. 24

I.1 QWEN3-1.7B

I.2 SMOLLM3-3B

Related papers

Loading 2609.37868v1…