PaperlayerView exact revision on arXiv

Source-linked AI summary

Probe the Harness: Setup Checks for Stale-Data RL Comparisons in Language Models

Taiheng Pan

arXiv:2610.02911v1cs.LGcs.CL

TL;DR

Stale-data methods are judged against importance-corrected baselines, but harness details can reverse their observed ranking. The paper introduces PTH checks for quantities that define those comparisons. With the harness checked, TIS matches SAN on verl, while in the trainer TIS learns steadily and SAN keeps a margin.

  • Problem

    Stale-data reinforcement learning methods are compared with importance-corrected baselines, but the harness quantities that define these comparisons can go unchecked.

    Source: §1 · p. 1

  • Method

    PTH provides seven checks that measure the quantity defining a comparison at each harness layer and across the comparison as a whole.

    Source: §1 · p. 2, §8 · p. 6

  • Results

    With defining quantities checked, TIS matches SAN on verl and learns steadily in the trainer, where SAN keeps a 7 to 10 point margin.

    Source: §1 · p. 2

  • Takeaways & Limitations

    PTH checks each harness layer so that a stale-data comparison measures the methods it names.

    Source: §8 · p. 6

    The trainer and verl differ in five respects, and locating the source of SAN’s trainer margin calls for a factorial comparison over them.

    Source: §5 · p. 5

Abstract

from arXiv · show

Methods for training language models on stale samples are judged by comparisons against importance-corrected baselines. We show that details of the experimental harness can reverse the observed ranking of methods, and we introduce PTH (Probe The Harness), a set of checks that makes the harness visible. Our case is a comparison between SAN, a behaviour-free method, and truncated importance sampling (TIS) on verl and in a single-GPU trainer, in which SAN first finished ahead in both stacks. Four details of the harness changed this comparison: the PPO ratio was taken against the learner's own recomputed probabilities, the data seed did not reach the TIS arm, the replay queue reused its first batch for 33 updates, and two loss normalisers differed from their description. In each case the logged quantity looked consistent with a working setup, while the quantity that defines the comparison went unchecked. With the harness checked, TIS matches SAN on verl, and in the trainer TIS learns steadily while SAN keeps a margin. We contribute the signature of each detail and its effect on the comparison, reference results for TIS and uncorrected GRPO under sampler lag, and the PTH checklist.

1 Introduction

With the harness checked, TIS matches SAN+ on verl and learns steadily in the single-GPU trainer, where SAN keeps a margin. The paper introduces PTH checks because harness details can reverse method rankings while standard logs leave defining quantities unchecked.

1 Introduction

  • With refresh interval 96, PPO-clip fell below 30% after peaking at 73 to 74% on GSM8K, while SAN+ held at 75 to 77%; the clip was inactive because the ratio used learner-recomputed probabilities.Replacing PPO clipping with truncated importance sampling removes the drop.Source: §1 · p. 1
  • SAN initially finished ahead of TIS and PPO clipping in both stacks, but four harness details changed the comparison or its interpretation.The details concerned the policy in the PPO ratio, arm configuration, data selection, and implemented loss.Source: §1 · pp. 1–2
  • The paper contributes reference results for correctly configured TIS and uncorrected GRPO under sampler lag, alongside the signature and comparison effects of each harness detail.It also extends guidance on the PPO ratio from one harness layer to four.Source: §1 · p. 2
  • PTH comprises seven checks built on one principle: measure the quantity that defines the comparison (Section 6); the passage also notes logged labels and results, seeds, update ages, batch identities, and untested equation agreement.Source: §1 · p. 2

2 The harness and the method under test

The paper compares SAN with importance-corrected methods in verl and a single-GPU trainer, using different stale-data setups. SAN caps token surprisal relative to its sequence mean and uses a robust-mean normaliser.

verl

  • In verl, the sampler is refreshed every N updates; TIS multiplies the PPO loss by a detached, capped ratio of learner to sampler probabilities.The setup uses GRPO advantages, one optimizer step per batch, and GSM8K on Qwen2.5-Math-1.5B.Source: §2 · pp. 2–3

The single-GPU trainer

  • The single-GPU trainer creates stale data with a replay queue, training each update on a batch generated k updates earlier.Its TIS arm is REINFORCE with a detached, capped learner-to-sampler weight.Source: §2 · p. 3

SAN

  • SAN caps each token’s weighted surprisal at its sequence mean and normalises by a robust mean, without using sampler probabilities.SAN+ and SANref use frozen-reference anchors.Source: §2 · p. 3

3 The initial comparison

In the initial comparison, the SAN family finished ahead across both stacks and all variants, while trainer diagnostics suggested a possible mechanism for TIS’s drop.

Initial comparison

  • SAN finished ahead in both stacks and every variant; at refresh interval 64, all nine SAN-family runs exceeded all six baseline runs.In the trainer, lower effective sample size and near-zero token coefficients accompanied sharply rising gradient norms during the TIS drop.Source: §3 · p. 3

Trainer diagnostics

  • The trainer diagnostics suggested that falling importance-weight effective sample size, near-zero token coefficients, and rising gradient norms accompanied the TIS drop.Section 4 traces these results to harness layers.Source: §3 · p. 3

Interpretation

  • The paper traces the initial comparison’s results to details in different layers of the harness.Source: §3 · p. 3

4 Four layers of the harness

The initial comparison changed when the authors checked the quantities that define each arm’s policy ratio, data order, replay batch, and implemented loss. Each layer had a distinct signature and effect on the comparison.

4.1 Policy in the PPO ratio

  • At refresh interval 96, recomputed learner probabilities made every PPO ratio 1 and the clip inactive; TIS instead ended at .764, .781, and .785, alongside SAN+ at .773, .752, and .772.Across 21 lagged-baseline runs, clip fraction and PPO KL stayed at zero while sampler-to-learner KL reached 7 to 9 nats per token before refresh.Source: §4.1 · p. 4

4.2 Configuration of each arm

  • The TIS launch configuration omitted its data seed; restoring seeds 43 and 44 closed the separation, with paired differences ranging from SAN-family leads of 4.2 points to a TIS lead of 2.4 points.The three original TIS runs repeated one data order, differing only through sampling randomness.Source: §4.2 · p. 4

4.4 Implemented loss

  • SAN+ and SANref differed from the written description in their normalisers, so their variant differences reflect anchor and normaliser jointly; the authors recommend testing each loss and gradient against its equation.This layer left the numbers unchanged but changed their interpretation.Source: §4.4 · pp. 4–5

4.3 Data for each update

  • At replay age 32, the warm-up queue reused batch 0 for the first 33 updates, and every warm-up TIS run fell to zero after update 50; unique prefill kept both methods learning.The queues had the same data-age schedule, so batch identity—not data age—distinguished the warm-up.Source: §4.3 · p. 4, Figure 2 · p. 5

5 Results with the harness checked

With the harness checked, TIS was stable and level with the SAN family on verl at the default learning rate, while trainer results showed higher SAN accuracy. The two stacks differ in several experimental conditions.

The trainer

  • In the trainer’s unique-prefill rerun, SAN reached 75.4 and 77.0% held-out accuracy, compared with TIS at 68.4 and 67.2%.The token weights alone reached 77.3 and 78.1%, against 71.5 and 70.7% for plain sequence normalisation.Source: §5 · p. 5

Two stacks, two regimes

  • The trainer and verl differ in five setup respects, and locating the source of the SAN margin calls for a factorial comparison across them.The differences include KL penalty, learning rate, data-age schedule, responses per update, and sampler numerics.Source: §5 · p. 5

Reference results on verl

  • TIS ended between .763 and .808 on verl and was level with SAN at 2×10−6, while uncorrected GRPO dropped in every seed at refresh interval 96.At 4×10−6, in one seed, TIS ended at .837 and SAN+ at .733.Source: Table 3 · p. 5, §5 · p. 5

6 PTH: probe the harness

PTH checks the quantities that define a stale-data comparison, alongside quantities that are convenient to log. Its seven checks cover individual harness layers and the comparison as a whole.

6 PTH: probe the harness

  • Record the ratio against the policy that generated the data, including the distribution of πθ/q and the clip fraction for every arm.With a lagged sampler, an exactly zero clip fraction signals that the source of πold should be inspected.Source: §6 · p. 6
  • Print each arm’s resolved configuration and verify that varied fields differ while fixed fields, including the data seed, match.Ordered prompt IDs in the first batches directly confirm the data order.Source: §6 · p. 6
  • Log each training batch’s identity as well as its age, and check that distinct batches match the replay design.Source: §6 · p. 6
  • Test each implemented loss and its gradient against the written equation on a fixed input.Source: §6 · p. 6
  • Map each baseline’s stable range under its intended configuration before comparing methods.In every revised setting in this report, TIS remains stable over all 100 updates.Source: §6 · p. 6
  • Report paired differences for each seed so pooled rankings do not mix seed effects with method effects.Source: §6 · p. 6
  • Review launch scripts and logs line by line against each draft claim; this located all four harness layers in the report.Source: §6 · p. 6

7 Related work

PTH applies implementation-and-seed scrutiny to the pipeline that simulates staleness. Prior work addresses the sampler–learner gap with importance correction and behavior-free alternatives.

7 Related work

  • Deep RL results depend on implementation details and seeds [4], while language-model reasoning benchmarks also depend on decoding settings and hardware [5].Source: §7 · p. 6
  • Staleness corrections include truncated importance weights, sampler-based PPO ratios [16] [20], proximal-policy separation [3] [8], and reshaped clipped objectives [17] [20].Behavior-free methods use learner statistics [11] or surprisal [12]; SAN belongs to that family, while variance control adapts steps to importance weights [6].Source: §7 · p. 6
  • Liu et al. and the verl documentation describe recomputed-policy ratios as a common instability source, detected by PTH check 1.Source: §7 · p. 6

8 Conclusion

The paper concludes that stale-data comparisons measure the harness alongside the methods. Checking defining quantities changes the outcome or interpretation, and PTH turns each harness layer into a check.

8 Conclusion

  • With harness quantities checked, TIS matches SAN on verl and learns steadily in the trainer, where SAN retains a margin; PTH makes comparisons measure the methods they name.The trainer margin is identified as a target for a factorial study of the two stacks.Source: §8 · p. 6

A SAN variants and the trainer ablation

The trainer ablation examines SAN variants and token weighting with a unique-prefill queue. Gradient-scale effects vary across seeds.

A SAN variants and the trainer ablation

B The original trainer grid at replay age 32

In the recorded original trainer grid, TIS stayed above zero outside the 1.5B full-parameter configuration, while its difference from SAN varied in sign across single runs. The grid also covered larger models and a 0.5B model at replay age 16.

B The original trainer grid at replay age 32

  • Outside the 1.5B full-parameter configuration, TIS stayed well above zero, and its difference from SAN ranged from 19.6 points behind to 9.6 points ahead across single runs.Runs used a warm-up queue; reported mean reward was from freshly generated batches over the last 10 of 100 updates.Source: B The original trainer grid at replay… · p. 8
  • At replay age 16, Qwen2.5-0.5B SAN reached .46 to .59 across five seeds.The grid also used 8-bit or paged Adam and shorter responses for the 3B and 7B models to fit one GPU.Source: B The original trainer grid at replay… · p. 8
  • A later-code-version entry at 10^-5 lists .273 (606), .108 (707), .004 (808), and .150 (909).Source: B The original trainer grid at replay… · p. 8
  • The Qwen2.5-3B-Instruct, Qwen2.5-Math-7B, Llama-3.2-3B-Instruct, and Qwen2.5-Math-1.5B LoRA 64 entries list paired scores of .821/.860, .569/.616, .800/.771, and .817/.721, respectively.For Llama-3.2-3B-Instruct, the first score also includes .717 (505); for Qwen2.5-Math-1.5B LoRA 64, the second includes .619 (505) and .815 (505).Source: B The original trainer grid at replay… · p. 8

Related papers

Loading 2610.02911v1…