Range-GRPO: Policy Optimization via Pairwise Relations among Reward Intervals

• A single judge score cannot express uncertainty, and width-based rescaling changes advantage magnitude but not its point-reward-determined direction.Source: §1 · pp. 1–2
• Pairwise midpoint differences weighted by both interval widths can change advantage magnitudes and signs, and recover Dr.GRPO when ranges collapse to points (Liu et al., 2025).Source: §1 · p. 2
• The framework adapts RoWCI for shift-aware judge calibration as the policy response distribution drifts from the labeled calibration distribution.Source: §1 · p. 2
• The paper presents Range-GRPO as the first semi-supervised GRPO framework using pairwise comparisons of conformal reward intervals for on-policy learning signals.Source: §1 · p. 2
• Range-GRPO achieves the highest ID and OOD average accuracies among evaluated semi-supervised methods, improving 5.2% and 7.0% over the strongest baseline, respectively.It also outperforms evaluated range-based methods on the synthetic task.Source: §1 · p. 2

2.1 REWARD SUPERVISION

• Range-GRPO uses judge estimates and limited labeled calibration data to construct conformal reward intervals for unlabeled rollouts.Source: §2.1 · p. 2

2.2 CONFORMAL CALIBRATION

• Policy updates can shift responses away from the calibration distribution, while Range-GRPO accounts for this shift when constructing intervals, unlike UARM’s stated calibration procedure.Weighted Conformal Inference reweights calibration samples using density ratios; RoWCI mitigates weight instability due to nuisance variation in judge inputs.Source: §2.2 · pp. 2–3

2.3 RANGE-BASED LEARNING SIGNALS

• Unlike approaches that use ranges for task-target rewards or rescale point-reward advantages, Range-GRPO compares conformal reward intervals pairwise so uncertainty can affect advantage magnitude and sign.Source: §2.3 · p. 3

3.1 GROUP RELATIVE POLICY OPTIMIZATION

• GRPO compares multiple responses to the same prompt to estimate advantages, which determine the direction and magnitude of policy learning signals.Dr.GRPO omits reward-standard-deviation normalization and response-length normalization from the objective.Source: §3.1 · p. 3

3.2 REWARD RANGE CONSTRUCTION

• Range construction adapts RoWCI by representing responses as low-cardinality judgment states and reweighting calibration data for distribution shift, assuming source and target reward distributions are similar conditional on state.A fixed scorer and calibration scores are combined with density-ratio estimates to form reward ranges; the stated conformal target is nominal marginal coverage 1−α.Source: §3.2 · pp. 3–4
• Figure 1’s training overview shows reward ranges formed from a fixed scorer, calibration scores, and updated density ratios, then compared pairwise to construct policy advantages.Source: Figure 1 · p. 4

4.1 RANGE-GRPO OBJECTIVE

• Pairwise midpoint differences, weighted by both responses’ interval widths, form Range-GRPO advantages; zero-width intervals recover Dr.GRPO exactly.A wider interval for either response attenuates that comparison, with the reference width controlling sensitivity.Source: §4.1 · p. 5

4.2 PRESERVING RANGE INFORMATION THROUGH PAIRWISE CREDIT ASSIGNMENT

• With midpoint rewards fixed, changing interval widths can reverse a response’s Range-GRPO advantage sign, unlike scalar reduction or response-level positive rescaling.In the stated comparison, the wider interval weakens a negative pairwise contribution enough to change the aggregate sign.Source: §4.2 · pp. 5–6

4.3 ON-POLICY TRAINING PROCEDURE

• At initialization, the method fits a state scorer and computes calibration nonconformity scores, then keeps these and the calibration data fixed during training.The fixed scorer uses source states and rewards; source and calibration sets share the labeled prompt distribution.Source: §4.3 · pp. 6–7
• Each iteration generates policy and target density-ratio-estimation rollouts, estimates state density ratios without correctness rewards, builds reward ranges, and updates the policy using pairwise advantages.Target density-ratio-estimation rollouts are excluded from policy updates.Source: §4.3 · p. 7

5.1 TOY EXPERIMENT

• On the synthetic candidate-selection task, the prompt omits the ranking rule, and the labeled pool is used only to fit the state scorer and calibrate reward intervals.The comparison controls data allocation and uses shared on-policy range construction to focus on how methods derive learning signals.Source: §5.1 · p. 7

5.2 MAIN EXPERIMENTS

• Range-GRPO achieves the highest ID and OOD averages among semi-supervised methods, leads on eight of nine benchmarks, and uses an approximately 18.2% smaller total response budget than TraPO.Both averages also exceed those of the TraPO reproduction with an entropy coefficient of 0.001.Source: Table 2 · p. 8, §5.2 · p. 9

5.1 TOY EXPERIMENT

• 39.40% mean test accuracy is the highest among the compared synthetic-task methods, exceeding the best scalar reduction by 7.63 percentage points; coverage exceeds 90% and mean interval width is 0.3595.Empirical coverage is measured on unlabeled responses over the full training phase.Source: §5.1 · p. 8

5.2 MAIN EXPERIMENTS

• The benchmark study uses labeled mathematics prompts and unlabeled MMLU-Pro prompts, evaluates ID mathematics and OOD benchmarks, and constructs calibrated reward ranges using a judge, scorer, and weighted calibration scores.The scorer and calibration scores remain fixed during training, while density ratios are updated each iteration.Source: §5.2 · pp. 8–9
• Range-GRPO also exceeds UARM on ID and OOD averages, exceeds the fully supervised 2K setting on ID, and remains close to it on OOD.The authors interpret the UARM comparison as consistent with preserving interval-dependent pairwise structure.Source: §5.2 · p. 9
• For an unlabeled OOD example, midpoint scalarization gives correct response R7 a negative advantage, while Range-GRPO's interval-width weighting yields a positive one.The six negative comparisons involve wider counterpart intervals than the positive comparison with R1.Source: §5.2 · p. 9

6 CONCLUSION

• Table 3 illustrates a correct OOD response receiving a negative midpoint-based advantage but a positive Range-GRPO advantage after interval-aware weighting.The example labels correctness only for post-hoc interpretation.Source: Table 3 · p. 9
• Range-GRPO compares reward intervals pairwise to assign relative credit, and reduces to Dr.GRPO when intervals have zero width.The conclusion reports that interval information can affect both the direction and magnitude of the learning signal.Source: §6 · pp. 9–10
• The conclusion reports higher ID and OOD averages than other semi-supervised methods and the best result on eight of nine individual benchmarks.It identifies extension to domains with uncertain reward supervision and limited expert annotations as future work.Source: §6 · p. 10

A NOTATION

• The notation appendix groups symbols by role and approximately orders them by first appearance; ζ denotes the UARM signal-variance stabilizer.Source: Table 4 · p. 14, A NOTATION · p. 15

B.1 DATA AND TRAINING

• The synthetic task ranks ten candidate words by length, alphabetizes only the three shortest, and assigns reward according to the selected candidate's rank.A selection is correct when rank(o) = 1, and the train, validation, and test sets have disjoint prompt compositions.Source: §B.1 · p. 15
• The synthetic study allocates 1,200 prompts—200 labeled and 1,000 unlabeled—across scorer fitting, calibration, density ratio estimation, and policy training, with shared partitions across methods.Table 6 also summarizes separate validation and test sets.Source: §B.1 · p. 15, Table 6 · p. 15
• Toy-study model selection uses ten training epochs, validation accuracy, and a three-seed mean; Range-GRPO selects λ = 0.2 from its specified grid.The learning-rate and KL-coefficient choices are selected using mean best validation accuracy across three seeds.Source: §B.1 · p. 15, Table 7 · p. 16

B.2 LEARNING SIGNALS

• The toy comparison applies different advantage rules to shared intervals: scalar baselines use lower bounds, upper bounds, or midpoints, while Range-GRPO uses its pairwise objective with λ = 0.2.UARM uses interval midpoints as reward estimates and retains its published advantage formula.Source: §B.2 · p. 15

(a) Training configuration

• GPT-2-small is trained with batch size 16, eight responses per prompt, ten epochs, AdamW, clipping 0.2, and entropy coefficient 0.01.The listed checkpoint is the one with best validation accuracy; the table gives seeds 6200, 6201, and 6202.Source: (a) Training configuration · p. 16
• Range-GRPO shows a stronger upward validation-accuracy trend, particularly relative to scalar baselines using GRPO advantages.Source: (a) Training configuration · p. 16
• Figure 2 plots validation accuracy at selected hyperparameters, showing mean and sample standard deviation across three seeds.Scalar baselines are shown separately for GRPO and Dr.GRPO advantages; UARM and Range-GRPO curves appear in both comparisons.Source: Figure 2 · p. 16

B.3 RANGE CONSTRUCTION

• Five word features form each response state: corpus frequency, genre count, synset count, noun status, and GPT-2 first-token surprisal.The first three features use log(1 + x), and all five use fixed standardization.Source: §B.3 · p. 16
• The adapted RoWCI scorer is fitted on initial-policy responses, and its fitted parameters and calibration scores stay fixed during training.All methods share the scorer, calibration scores, and DRE source data.Source: §B.3 · p. 16
• At each unlabeled update, density-ratio weights reweight fixed calibration scores to address response-distribution changes, with weights capped at their 95th percentile.The weighted cutoff also accounts for the query response's weight.Source: §B.3 · p. 16
• Unlabeled rewards are excluded from policy optimization and used instead for coverage diagnostics at a nominal level of 90%, rather than to optimize the policy.When the cutoff is infinite, the full range is used.Source: §B.3 · p. 17
• The toy experiment's response budget covers policy training, scorer fitting, calibration, and target DRE, while source DRE reuses scorer responses.Directly computing response features from words avoids requiring an LLM judge.Source: §B.3 · p. 17

C EXPERIMENTAL DETAILS

• The benchmark comparison initializes policies from Qwen2.5-Math-7B and reports results from one training run per configuration.Checkpoint selection uses mean validation accuracy across AIME, AMC, MATH, Minerva, and OlympiadBench.Source: C EXPERIMENTAL DETAILS · p. 17

C.1 TRAPO

• An entropy coefficient of 0.001 improves average ID and OOD performance by 2.6 and 1.2 percentage points, respectively, over 0.01, except on GPQA.The main table reports the 0.001 configuration; all other settings are held fixed in the comparison.Source: §C.1 · p. 17, Table 9 · p. 18

C.2 UARM

• UARM trains a quantile head on a frozen Qwen2.5-Math-7B reward-model backbone, using binary correctness targets and a separate labeled calibration subset.The RM and calibration parameters remain fixed during policy training.Source: §C.2 · p. 17
• UARM trains for ten labeled warmup epochs followed by five epochs combining labeled updates with UARM advantages on unlabeled prompts.The experiment uses 2,048 prompts, split evenly between labeled and unlabeled data.Source: §C.2 · p. 18
• UARM's response budget counts both generated responses and responses processed by the reward model, without assuming equal computational costs.The budget includes RM processing; the comparison table also accounts for TraPO warmup rollouts and Range-GRPO judge generations.Source: §C.2 · p. 18, Table 10 · p. 18

C.3 RANGE-GRPO

• The frozen judge assigns each of eight responses a five-component state scored in {0, 0.5, 1}, using response properties and within-group agreement.The judge receives no reference answers or correctness labels.Source: §C.3 · p. 19, Table 13 · p. 19
• After labeled warmup, the policy generates source and calibration responses; the fitted scorer and calibration scores then remain fixed.DRE is refreshed at each unlabeled update, unlike the fixed scorer and calibration scores.Source: §C.3 · p. 20, Table 14 · p. 20
• Normalized nonconformity scores become more sensitive to scorer predictions near 0 or 1 for the same positive prediction error, down to a ψ(v) floor of 0.05.This makes the scores depend on both the scorer prediction and the absolute prediction error.Source: §C.3 · p. 20
• Query-specific rescaling combines the normalized conformal cutoff with ψ(sZ(Z)), while DRE is refitted using fixed source states and fresh current-policy target states.The rescaling is intended to reflect differences in scorer confidence more directly in the constructed ranges.Source: §C.3 · p. 20
• Training uses ten labeled warmup epochs followed by five active epochs, with binary correctness rewards for labeled updates and Range-GRPO advantages for unlabeled updates.The prompt subsets remain fixed and disjoint; scorer fitting and source DRE share responses, with correctness labels used only for scorer fitting.Source: §C.3 · p. 20
• 201,024 response-budget units are 18.2% below the 245,760 units used for TraPO and UARM, while Range-GRPO achieves strong overall performance.The budget includes both policy rollouts and judge generations.Source: §C.3 · p. 20
• Range-GRPO has wall-clock training time comparable to UARM despite its judge phase, while using four rather than five active GPUs.The reported GPU count refers to active training allocation.Source: §C.3 · pp. 20–21, Table 16 · p. 21