PaperlayerView exact revision on arXiv

Source-linked AI summary

CARM: Cancellation-Aware Response Masking for LLM Reinforcement Learning

Yafei Zhang, Songshuo Lu, Sicong Liao, Zhi Chen, Yaohua Tang

arXiv:2610.02039v1cs.LGcs.AIcs.CL

TL;DR

Practical RL can make sampled responses off-policy, while signed geometric-mean masking can hide opposing token shifts. CARM averages absolute token log-ratios, and the paper reports improvements over evaluated baselines in mathematical reasoning and code generation.

  • Problem

    Practical RL can make sampled responses off-policy, while signed geometric-mean masking may make large opposing token shifts appear aligned.

    Source: §1 · p. 1

  • Method

    CARM masks at the response level by averaging absolute token log-ratios and bounds accepted responses’ out-of-band token fraction and mean violation severity.

    Source: §1 · p. 2

  • Results

    CARM improves mathematical-reasoning and code-generation results over evaluated baselines; on Qwen3.5-9B, it outperforms GeoMean by 3.47 points with masked fractions differing by 0.43 points.

    Source: §5.1 · p. 8, §6 · p. 9

  • Takeaways & Limitations

    CARM is a principled response-selection criterion that complements reward-directed PPO/GRPO optimization.

    Source: §6 · p. 9

    The analysis assumes nonempty responses with finite token log-probabilities, and Proposition 1 does not claim that exact trajectory likelihood ratios are incorrect.

    Source: §3.1 · p. 3, §3.2 · p. 4

Abstract

from arXiv · show

Recent years have witnessed the rapid adoption of reinforcement learning (RL) in large language model (LLM) post-training, with substantial gains in mathematical reasoning and code generation. In practical systems, however, policy updates and differences between rollout and training engines can make sampled responses off-policy. Sequence-level masking addresses this mismatch by deciding whether an entire response should contribute to optimization. A common masking rule uses the length-normalized geometric mean of sampled token probability ratios. Its signed log-ratios can cancel across positions, concealing substantial bidirectional policy drift. We propose \emph{Cancellation-Aware Response Masking} (CARM), a sequence-level mask that takes the absolute value of each token log-ratio before averaging, preventing opposing probability changes from canceling. We prove that accepted responses satisfy a joint bound on the fraction of sampled-token ratios outside a prescribed band and their mean log-distance beyond its boundaries. Experiments on mathematical reasoning and code generation show that CARM improves mean@16 averaged over AIME 2024/2025/2026 and BeyondAIME by up to $3.13$ percentage points over geometric-mean masking, and increases average pass@1 across four code benchmarks by $2.88$ points over the strongest evaluated baseline. These findings support CARM as a theoretically grounded and effective method for response-level off-policy control in LLM reinforcement learning.

1 INTRODUCTION

Practical policy changes and rollout–training differences can make sampled responses off-policy, while signed-log aggregation can conceal opposing token-probability changes. CARM masks at the response level using absolute token log-ratios and gives accepted responses a token-level violation bound.

1 INTRODUCTION

  • Policy updates, actor–learner latency, and rollout–training engine differences can make sampled responses off-policy.The updated policy may assign substantially different probabilities to sampled tokens than the rollout policy.Source: §1 · p. 1
  • Signed log-ratios can cancel, making a response appear aligned even when its token-probability changes are large.For example, token ratios of 10 and 0.1 have geometric mean one despite each sampled-token probability changing tenfold.Source: §1 · pp. 1–2
  • CARM averages absolute token log-ratios to prevent cancellation while retaining the token-level PPO/GRPO surrogate.Its accepted responses satisfy a joint bound on the fraction of sampled-token ratios outside a prescribed band and their mean log-distance beyond its boundaries.Source: §1 · p. 2
  • Experiments on mathematical reasoning and code generation report improvements over sequence- and token-level baselines.Paired-response diagnostics and threshold analyses are also reported.Source: §1 · p. 2
  • Accuracy is higher at comparable average filtering rates, and it is not monotone in the masked fraction.The introduction characterizes response selection behavior without specifying benchmark values.Source: §1 · p. 3

2 RELATED WORK

Related work addresses policy optimization, off-policy control, and sequence-level mismatch through several complementary mechanisms. CARM instead selects whether a complete sampled response contributes to the update.

2 RELATED WORK

  • PPO clipping, GRPO advantages, reward signals, and sampling methods shape how sampled responses contribute to optimization.CARM addresses the complementary question of whether a sampled response contributes to the current update.Source: §2 · p. 3
  • Importance weighting, token-level filtering, adaptive clipping, and divergence controls regulate contributions or update size within a response.CARM instead decides whether to retain the complete response, separating response selection from the surrogate applied to retained tokens.Source: §2 · p. 3
  • Geometric aggregation summarizes sequence mismatch but averages signed log-ratios, allowing opposing shifts to cancel; TRM instead uses a worst-position view.CARM averages absolute log-ratios to measure bidirectional drift while retaining the token-level policy surrogate.Source: §2 · p. 3

3 METHOD

The method defines response-level mismatch from sampled-token probability ratios, contrasts signed geometric aggregation with cancellation-aware aggregation, and interprets the CARM threshold through bounds on token-level violations.

3.1 PROBLEM SETUP

  • The setup defines each sampled-token ratio from current-policy and rollout-policy probabilities and applies a binary response mask to retained-token PPO/GRPO losses.The policy-loss contribution is defined as zero if no response is retained.Source: §3.1 · pp. 3–4

3.2 GEOMEAN MASKING AND RATIO CANCELLATION

  • GeoMean accepts responses when the reciprocal geometric-mean ratio lies within a band, but signed log-ratios can cancel even when every token ratio lies outside it.Proposition 1 gives an even-length response with every ratio outside [1/c, c] but GeoMean score one.Source: §3.2 · p. 4

3.3 CANCELLATION-AWARE RESPONSE MASKING

  • CARM averages token log-ratio magnitudes, so its score is at least GeoMean’s for every response and it rejects at least as many responses at shared thresholds.Mean-absolute aggregation prevents cancellation without selecting only the most extreme token or squaring each deviation.Source: §3.3 · pp. 4–5

3.4 THRESHOLD INTERPRETATION

  • Accepted CARM responses share a budget for the fraction and mean severity of out-of-band ratios, including for asymmetric bands.This is a deterministic bound on sampled-token ratios, not on full conditional-policy KL divergence. For [0.8, 1.28], the bound is at most 8.88% at τ=1.02 and 17.58% at τ=1.04.Source: §3.4 · p. 6

4 EXPERIMENTS

Experiments evaluate CARM on mathematical reasoning and code generation. CARM leads the four-dataset mathematical-reasoning average for both models and improves average code pass@1 over the strongest evaluated baseline.

4.1 EXPERIMENTAL SETUP

  • Experiments train Qwen3.5-4B and Qwen3.5-9B on DAPO Math and Qwen3.5-9B on TACO, using GRPO and three random seeds for both tasks.Mathematical reasoning is evaluated on AIME 2024/2025/2026 and BeyondAIME; code generation uses executable rewards and four code benchmarks.Source: §4.1 · p. 7, §4.3 · p. 8

4.2 MATHEMATICAL REASONING

  • CARM has the highest four-dataset Average for both Qwen3.5-4B and Qwen3.5-9B, with gains over GeoMean on AIME and BeyondAIME.The Average rises from 61.59% to 63.45% for Qwen3.5-4B and from 67.29% to 70.42% for Qwen3.5-9B; three-year AIME averages rise from 69.17% to 70.42% and from 74.03% to 77.50%, respectively.Source: Table 1 · p. 7, §4.2 · p. 7

4.3 CODE GENERATION

  • CARM improves average pass@1 by 2.88 points over IcePop, the strongest baseline on average, across four code benchmarks.The evaluation uses Qwen3.5-9B trained on the TACO training split and tests held-out TACO problems plus LiveCodeBench-v6, HumanEval+, and MBPP+.Source: Table 2 · p. 8, §4.3 · p. 8

5 ANALYSIS

The analyses find that filtering rate alone does not predict accuracy, and that CARM's gains persist under standard masking and across sampling criteria. The analyses also examine update redundancy and the effect of conditioning masks on advantage sign.

5 ANALYSIS

  • The analysis tests whether gains reflect filtering more responses, whether advantage sign should override the gate, and whether gains persist across sampling criteria.Update redundancy describes improved validation accuracy despite omitting response-level updates, without implying rejected responses lack useful information.Source: §5 · p. 8, §5.1 · p. 8

5.1 UPDATE REDUNDANCY AND EFFECTIVE FILTERING

  • On Qwen3.5-9B, CARM outperforms GeoMean by 3.47 points while their mean masked fractions differ by 0.43 points; accuracy is not monotone in filtering.TRM rejects more responses without outperforming CARM, and paired-score differences show that rejection rates alone do not characterize retained trajectories.Source: §5.1 · pp. 8–9, Figure 4 · p. 8

5.2 ADVANTAGE-CONDITIONAL MASKING ABLATION

  • Standard masking reaches 75.63% AIME mean@16 versus 60.28% for negative-only masking, and 50.58% versus 42.22% on LiveCodeBench-v6.The comparison fixes CARM's score and threshold within each task; the figure distinguishes standard masking (solid) from negative-only masking (dashed).Source: §5.2 · p. 9, Figure 5 · p. 9

5.3 SAMPLING CONSISTENCY

  • For Qwen3.5-9B, CARM improves mean@16, best@16, and worst@16 over GeoMean by 3.47, 2.74, and 2.78 points, respectively.CARM also achieves the highest best-of-k and worst-of-k values throughout the displayed range; worst-of-k measures sampling repeatability, not distribution-shift or adversarial robustness.Source: §5.3 · p. 9, Figure 6 · p. 9

6 CONCLUSION

CARM uses absolute token log-ratios to prevent cancellation, and its theoretical comparisons show why this matters: GeoMean can accept severe opposing changes, while CARM yields a stricter response score.

6 CONCLUSION

  • CARM prevents opposing token log-ratios from canceling and gives accepted responses a joint bound on the frequency and severity of ratio-band violations.It complements reward-directed PPO/GRPO optimization and is evaluated for mathematical reasoning and code generation.Source: §6 · p. 9

A.1 PROOF OF PROPOSITION 1

  • A constructed response with equal numbers of ratios B and 1/B has every ratio outside [1/c, c], yet GeoMean reports its minimum score and accepts it at thresholds of at least one.CARM scores this construction as B and rejects it whenever τCARM < B.Source: §A.1 · pp. 13–14

A.2 PROOF OF PROPOSITION 2

  • For every response, the GeoMean score is no greater than the CARM score, so the CARM accepted set is contained in the GeoMean accepted set at the same threshold.The scores are equal when all nonzero token log-ratios have the same sign, and the inequality is strict when both signs occur.Source: §A.2 · p. 14

A.3 PROOF OF PROPOSITION 3

Proposition 3 derives a shared budget for out-of-band frequency and severity; Corollary 1 specializes it to a reciprocal-symmetric band.

A.3 PROOF OF PROPOSITION 3

  • Accepted responses satisfy a joint bound on the fraction of ratios outside the band and their mean log-distance beyond its boundaries.The frequency-only consequence is pout ≤ min(1, log τ / bmin), where bmin is the smaller boundary cost.Source: §A.3 · p. 16

A.4 PROOF OF COROLLARY 1

  • For the reciprocal-symmetric band [1/c, c], the violation fraction pc obeys pc log c + pc δ̄c ≤ log τ, and therefore pc ≤ log τ / log c.Here δ̄c is the mean log-distance beyond the band boundaries among violating tokens.Source: §A.4 · p. 17

B IMPLEMENTATION OF CARM

The implementation applies a binary response-level gate to PPO/GRPO losses while retaining the token-level surrogate, and computes the mask from current log-probabilities during training.

B IMPLEMENTATION OF CARM

  • The response-level gate determines which per-response PPO/GRPO losses contribute to an update while leaving the token-level surrogate unchanged.The hard threshold decision has no gradient through it.Source: B IMPLEMENTATION OF CARM · p. 17
  • During training, each micro-batch forward supplies current log-probabilities for both the mask and token surrogate, and the gate is recomputed on evaluation under an updated policy.Masking is local to loss evaluation, not permanent removal from the rollout batch; the experiments use one optimization epoch per rollout batch.Source: B IMPLEMENTATION OF CARM · p. 17
  • The verl surrogate also applies dual clipping to negative advantages, and the sequence mask multiplies that surrogate without changing its clipping rule or normalization.The reported dual-clipping constant is cdual = 3.Source: B IMPLEMENTATION OF CARM · p. 17
  • The implementation clips log-ratios to [-20, 20]; when this guard is active, the bounds apply to effective ratios and the severity term measures guarded log-distances.For bands strictly inside [e^-20, e^20], including [0.8, 1.28], clipping does not change which tokens violate the band.Source: B IMPLEMENTATION OF CARM · p. 18

C COMPLETE RESPONSE FOR THE DISAGREEMENT CASE

This section presents the complete disagreement case, where signed aggregation can hide token-level drift and yield a different response-mask decision from CARM, alongside a corrected worked function evaluation concluding that m+n=1. It also documents the math, code, and prompt protocols used for training and evaluation.

C COMPLETE RESPONSE FOR THE DISAGREEMENT CASE

  • dGeoMean is 0.002502 and dCARM is 0.024486, with signed aggregation canceling 89.78% of absolute log-ratio mass; GeoMean accepts while CARM rejects.The displayed scores are GeoMean s=1.0025 <= 1.005 (accept) and CARM s=1.0248 > 1.02 (reject).Source: C COMPLETE RESPONSE FOR THE… · p. 19
  • Five pages display all 8,619 response tokens without elision, using tokenizer-aligned boxes colored by signed log-ratio and intensity by absolute magnitude.Red marks log rt > 0 and green marks log rt < 0; the first-page header repeats response scores and masking decisions.Source: C COMPLETE RESPONSE FOR THE… · p. 19
  • When the answer is -1/2, the cited result assigns m = -1 and n = 2; it also reports that their sum, m + n, equals 1.Source: C COMPLETE RESPONSE FOR THE… · p. 21
  • f(-100) = -1/2, since the recurrence gives f(-100) = f(12) and f(12)f(5) = -1 with f(5) = 2.The argument uses f(x + 7)f(x) = -1, which implies f(x + 14) = f(x), and reduces -100 modulo 14 to 12.Source: C COMPLETE RESPONSE FOR THE… · pp. 20–21
  • The shift relation f(x + 7)f(x) = -1 follows by setting u = x + 7 and v = x in the original equation, whose arguments differ by 7.Applying the relation twice yields f(x + 14) = f(x), enabling the reduction of -100 to 12.Source: C COMPLETE RESPONSE FOR THE… · p. 21
  • These passages establish two facts: reducing -93 by -98 gives residue 5 modulo 14, while the evaluation of f at 5 is log₂(9−5)=log₂(4)=2.Source: C COMPLETE RESPONSE FOR THE… · p. 22

D.1 DATA

  • f(-100)=-1/2 gives m=-1, n=2, and m+n=1.The signed fraction convention represents -0.5 as -1/2, with n>0.Source: C COMPLETE RESPONSE FOR THE… · p. 22, §D.1 · p. 23
  • From f(x+7)f(x)=-1, applying the relation at x+7 gives f(x+14)=-1/f(x+7)=f(x); therefore f has period 14.Source: §D.1 · p. 23
  • The worked evaluation gives f(12) = -1/2 and f(-100) = -1/2, so the requested m+n equals 1.The solution identifies m = -1 and n = 2, with a positive denominator.Source: §D.1 · p. 23
  • Math experiments use DAPO Math training problems, Math-Verify answer checking, and shared optimization settings across methods, with only off-policy correction differing.Evaluation samples 16 responses per problem across AIME 2024, AIME 2025, AIME 2026, and BeyondAIME; Table 1 weights the four sets equally.Source: §D.1 · pp. 23–24
  • Code RL uses TACO training problems and a reward capped at the first ten available tests, while held-out benchmark tests are excluded from training reward.All five methods receive the same prompts, decoding budget, executable reward, and optimizer configuration.Source: §D.1 · p. 24
  • Code evaluation reports pass@1 on TACO, LiveCodeBench-v6, HumanEval+, and MBPP+, rejudging generated programs with SandboxFusion.The suites contain 1,000, 1,054, 164, and 378 problems, respectively; HumanEval+ and MBPP+ use EvalPlus tests.Source: §D.1 · p. 24

D.2 PROMPT TEMPLATES

  • Training and evaluation reuse task-specific chat prompts, replacing the problem placeholder with dataset statements; code prompts include starter code only when provided.Figures 7 and 8 show the math and code prompt templates, respectively.Source: §D.2 · p. 24, Figure 7 · p. 24, Figure 8 · p. 24

D.3 TRAINING AND EVALUATION CONFIGURATION

This section specifies the shared training and evaluation configuration, method-specific controls, and threshold grids used for the reported comparisons. It also describes the evaluation protocol and how training iterates and TRM results were selected.

D.3 TRAINING AND EVALUATION CONFIGURATION

  • Table 3 consolidates shared settings and method-specific controls for mathematical reasoning and code generation.The configuration also includes the threshold grids used for reported comparisons.Source: Table 3 · p. 25, §D.3 · p. 25
  • Math evaluation uses three seeds and 16 responses per problem with mean@16; code evaluation uses three seeds and one response per problem with pass@1.Code decoding uses temperature 0.7, top-p = 0.9, top-k = 50, a 16,384-token maximum, and the same batched SandboxFusion evaluator across methods.Source: §D.3 · p. 25
  • Math results use the iterate with the highest three-year AIME mean@16, while code results use the iterate with the highest LiveCodeBench-v6 validation pass@1.Best- and worst-of-k use the same math iterate; Qwen3.5-4B BeyondAIME Null, GeoMean, and CARM results use separate configuration- and iteration-matched runs.Source: §D.3 · p. 25
  • TRM results use sampled-token max-filtering, the proposed approximation when full logits are unavailable, rather than a full-vocabulary KL implementation.The implementation rejects a response when the maximum sampled-token squared log-ratio exceeds 0.05; all reported TRM results use this variant.Source: §D.3 · p. 25

E THRESHOLD RESULTS

  • Table 4 reports GeoMean and CARM threshold sweeps using three-year AIME mean@16, with bold and underline marking the best and second-best values within each model block.The supplied passage identifies the table’s scope and notation but does not provide threshold values.Source: Table 4 · p. 26, E THRESHOLD RESULTS · p. 26

F ADDITIONAL TRAINING DIAGNOSTICS

The additional diagnostics visualize response length, policy entropy, and gradient norm during mathematical-reasoning and code-generation training. A separate figure tracks sequence-masked fractions over time for selected GeoMean, TRM, and CARM configurations.

F ADDITIONAL TRAINING DIAGNOSTICS

  • Figure 9 tracks sequence-masked fractions over training for selected GeoMean, TRM, and CARM configurations in math and code.The left and right panels show Qwen3.5-9B mathematical-reasoning and code-generation training; faint curves are raw measurements and opaque curves are exponential moving averages with weight 0.8.Source: Figure 9 · p. 26, F ADDITIONAL TRAINING DIAGNOSTICS · p. 26
  • Figure 10 compares response length, policy entropy, and gradient norm across five methods during Qwen3.5-9B mathematical-reasoning training.Faint curves show raw measurements, opaque curves show exponential moving averages with weight 0.8, and gradient norm uses a logarithmic scale.Source: F ADDITIONAL TRAINING DIAGNOSTICS · p. 26, Figure 10 · p. 27
  • Figure 11 presents the same response-length, policy-entropy, and gradient-norm diagnostics for Qwen3.5-9B code-generation training on TACO.Its smoothing and axis conventions match the mathematical-reasoning diagnostics, allowing domain-level comparison.Source: Figure 11 · p. 27, F ADDITIONAL TRAINING DIAGNOSTICS · p. 27

G BEST- AND WORST-OF-k SAMPLING METRICS

The section defines best-of-k and worst-of-k sampling statistics to characterize solution coverage and repeatability. It also specifies how the metrics are estimated from 16 responses and how mean@16 is computed.

G BEST- AND WORST-OF-k SAMPLING METRICS

  • mean@16 is the per-problem fraction correct among 16 responses, averaged over problems and equally across AIME 2024, 2025, and 2026.Source: G BEST- AND WORST-OF-k SAMPLING METRICS · p. 27
  • Best-of-k measures whether at least one sampled response is correct, while worst-of-k measures whether all are correct; worst-of-k is not a measure of robustness to distribution shift or adversarial perturbations.Together, the curves characterize solution coverage and repeatability.Source: G BEST- AND WORST-OF-k SAMPLING METRICS · pp. 27–28
  • For each problem, the metrics resample k responses with replacement from 16, apply the maximum or minimum correctness operator, and average 1,000 bootstrap replicates.The same models as Table 1 are used.Source: G BEST- AND WORST-OF-k SAMPLING METRICS · p. 28
  • Equation 73 is the probability that at least one of k independent draws from the empirical response distribution is correct, using replacement rather than the standard without-replacement pass@k estimator.Source: G BEST- AND WORST-OF-k SAMPLING METRICS · p. 28
  • Equation 74 gives the probability that all k draws are correct; it decreases with k, strictly so when 0 < c < 16, and an aggregate increase does not imply improvement on every problem.Worst-of-k emphasizes problems with high per-problem correctness probability.Source: G BEST- AND WORST-OF-k SAMPLING METRICS · p. 28
  • Table 5 reports the exact Qwen3.5-9B values underlying Figure 6 as equal-weighted three-year AIME averages in percent.Source: Table 5 · p. 28
Loading 2610.02039v1…