SCARL: Sequence Confidence-Aligned Reinforcement Learning

Motivation

• The paper links confidence misalignment to entropy collapse and narrowed exploration, where gains can favor per-draw accuracy while large-k coverage stalls.It situates SCARL among approaches that alter clipping, regulate covariance, stabilize updates, or condition updates on confidence or uncertainty.Source: §1 · p. 1

Problem

• Group-relative RLVR’s advantage ignores response confidence, so a barely found success receives the same gradient as a near-certain one, and confident errors are no harder to correct.The paper calls this sequence-confidence misalignment; aligned training would give responses with the same outcome equal confidence.Source: §1 · pp. 1–2

Method

• SCARL reshapes each advantage according to the gap between group-relative sequence confidence and the outcome it certifies, directing updates toward misaligned responses.The alignment gap damps mastered successes, consolidates fragile successes, amplifies corrections of confident errors, and tolerates exploratory errors.Source: §1 · p. 2, Figure 1 · p. 2
• The reshaping is multiplicative and positive, never reverses an update, preserves each outcome class’s advantage budget, and reduces to the base estimator when class confidences agree.It adds no extra passes and one hyperparameter.Source: §1 · p. 2

Results

• SCARL improves average Pass@32 by 2.4 points over DAPO (69.5 vs. 67.1) and leads nine compute-matched methods on coverage and per-draw accuracy.Across four estimators and four backbones, it improves 14 of 16 pairings.Source: §1 · p. 2

2.1 Background: Group-Relative RLVR

• Group-relative RLVR samples responses per prompt, scores them with a verifier, and normalizes rewards within the group; its advantage depends on rewards, not policy confidence.The sequence-level advantage is broadcast to every token, and SCARL is agnostic to the chosen group-relative surrogate.Source: §2.1 · p. 3

2 SCARL: Sequence Confidence-Aligned Reinforcement Learning

• SCARL adds a positive, sequence-level shaping factor to the advantage while leaving rewards, importance ratios, and clipping untouched.The factor is built from behavior-policy log-likelihood, group-relative confidence, the alignment gap, and class-wise normalization.Source: §2.1 · p. 3

2.2 Sequence Confidence

• Sequence confidence is the behavior policy’s response log-likelihood, summing token log-probabilities already computed for importance ratios.The sum is fixed across minibatch updates; within a prompt group, it deliberately treats longer responses as less confident, all else equal.Source: §2.2 · p. 3

2.3 Group-Relative Confidence

• Group-relative confidence standardizes raw response log-likelihoods within a group and applies a temperature-scaled softmax, making confidence relative to sibling responses.The softmax is shift-invariant and positive, and its temperature decouples sharpness from group dispersion.Source: §2.3 · pp. 3–4

2.4 Alignment-Gap Shaping

• The alignment gap measures distance from equal confidence among responses with the same outcome, using the advantage sign to identify the outcome class.Misaligned fragile successes are consolidated and confident errors corrected hardest, while aligned successes and exploratory errors receive small gaps.Source: §2.4 · p. 4

2.5 Class-Wise Normalization

• Class-wise normalization turns raw alignment gaps into shares of each outcome class’s fixed advantage budget, with a weight of one for singleton classes.Weights remain positive, preserve update direction, and equal one when confidences agree within a class, recovering the base estimator.Source: §2.5 · p. 4

2.6 Training Objective

• SCARL transforms advantages while inheriting the base surrogate unchanged, so it composes with ratio-space interventions, dynamic sampling, and alternative estimators.The shaping coefficient depends on the behavior policy; the method uses O(G) scalar operations on already computed quantities, with negligible wall-clock overhead.Source: §2.6 · p. 4

3 Experiments

• The experiments test compute-matched performance, transfer without retuning, whether alignment drives the gain, and whether training exhibits the intended mechanism.The primary setup compares SCARL with existing interventions; the generality, ablation, and mechanism studies address the remaining questions.Source: §3 · p. 4

3.1 Setup

• The primary comparison uses Qwen3-4B-Base; the study also evaluates three other backbones and four base estimators, with five competition-mathematics benchmarks and 32 responses per problem.Methods share a 150-step training recipe; SCARL uses τ = 5, selected in a sweep and reused in the generality study.Source: §3.1 · p. 5

3.2 Main Results: SCARL versus Existing Interventions

• 2.4 points: DAPO + SCARL raises average Pass@32 from 67.1 to 69.5 on Qwen3-4B-Base and leads all nine methods on average Pass@32 and Avg@32.It is best on four of five benchmarks; Dr. GRPO leads on Minerva (59.6 vs. 57.4). Figure 2 shows SCARL leading both metrics through the second half of training.Source: §3.2 · p. 5, Table 1 · p. 6, Figure 2 · p. 6

3.3 Generality across Base Estimators and Model Families

• 14 of 16 estimator–backbone pairings improve with SCARL, with a mean average Pass@32 change of +1.0 and no per-setting retuning.Both exceptions are SAPO pairings, each down −0.2; the three hard-clipping estimators improve in all twelve settings.Source: §3.3 · p. 6, Table 2 · p. 6

3.4 Ablations: Alignment Is the Active Ingredient

• Class-wise normalization is load-bearing: raw-gap shaping drops average Pass@32 to 47.0, while targeting a constant instead of the verified outcome returns performance to baseline.Using mean rather than summed log-likelihood yields 55.3, 1.1 points below plain GSPO; the temperature sweep peaks at 57.1 for τ = 5 and falls to 54.7 at τ = 10.Source: §3.4 · p. 7, Table 3 · p. 7

3.5 Mechanism: Preserved Exploration, Repaired Confidence Alignment

• From step 12 onward, SCARL keeps entropy above the base estimator for all four estimators, while confidence separation rises in all three tested settings.GRPO is an exception to avoiding entropy collapse: SCARL slows its decay but cannot avert collapse under symmetric hard clipping. Cohen’s d rises from 0.65 to 1.06 for Qwen3-4B-Base / DAPO, from 0.50 to 0.97 for DeepSeek-R1-Distill-Qwen-1.5B / GSPO, and from 0.95 to 1.20 for Hunyuan-1.8B-Pretrain / GRPO.Source: §3.5 · p. 7

RL with verifiable rewards

• RLVR replaces learned reward models with programmatic verification and, at scale, elicits long chain-of-thought reasoning; GRPO and its descendants support the group-relative recipe.The paper’s coverage-centred evaluation protocol follows Yue et al. (2025).Source: §4 · pp. 7–8

Entropy collapse and the update pathway

• Prior entropy-collapse interventions change clipping, regulate log-probability–advantage covariance, or refine the ratio pathway; SCARL instead reallocates reinforcement across sequences.The cited approaches include DAPO, covariance interventions, CW-GRPO, GSPO, and SAPO.Source: §4 · p. 8

Confidence-aware policy optimization

• Unlike prior confidence-aware methods, SCARL compares each response with the mean confidence of its own outcome class and uses a positive multiplicative factor with class-wise mean-one normalization.The paper contrasts its design with CDE, EAS, ARM, SEED-GRPO, UCAS, CAPO, and ACE; the normalization recovers the base estimator when class confidences agree.Source: §4 · p. 8

• Group-relative RLVR does not tie a policy’s confidence in a response to its verified outcome.Source: §5 · p. 8
• SCARL reshapes each advantage using its alignment gap, computed from quantities already materialized during training.Source: §5 · p. 8
• SCARL achieves the best coverage and per-draw accuracy among nine compute-matched methods, transfers across four estimators and four backbones without retuning, and preserves exploration while widening confidence–correctness separation.Source: §5 · p. 8

Limitations

• Experiments cover competition mathematics with binary, rule-based verifier rewards; other domains and shaped or continuous rewards are not studied.Source: Limitations · p. 9
• The backbones span three model families but only scales up to 4B parameters.Source: Limitations · p. 9
• Using aligned confidence at inference time for abstention or selection is a natural extension that the authors have not explored.Source: Limitations · p. 9

A Extended Method Discussion

• For softmax policies, entropy change under a policy-gradient step is tied to the negative covariance between token log-probability and advantage; confident successes can drive entropy downward (Cui et al., 2025).Because sequence advantages are broadcast across tokens, the covariance aggregates at sequence level, and advantage-only weighting can over-reinforce confident successes while under-consolidating fragile responses.Source: A Extended Method Discussion · p. 10
• Among positive-advantage samples, SCARL damps higher-confidence responses; among negative-advantage samples, it strengthens corrections to confident modes and spares rare exploratory errors.These effects shrink the covariance associated with entropy decay, slowing that decay at its source rather than through post-hoc clipping or entropy bonuses.Source: A Extended Method Discussion · p. 10
• Within each advantage class, SCARL compares responses against the class mean confidence, making the factor equal one when that class is uniform.For negative advantages, the factor also reflects each response’s share of its class’s confidence; the resulting allocation drives same-class responses toward shared confidence and learning speed.Source: A Extended Method Discussion · p. 10
• At the start of an inner epoch, SCARL acts as a per-sequence, confidence-dependent learning rate on the vanilla policy gradient.Source: A Extended Method Discussion · p. 11
• Group standardization lets one adaptive temperature transfer across prompts, model scales, and training stages; a fixed-temperature variant requires retuning by model and stage.Without group standardization, near-degenerate groups yield uninformative uniform factors, while heterogeneous groups collapse toward a near one-hot vector.Source: A Extended Method Discussion · p. 11
• Equation (7) resembles focal modulation, while Equation (8) measures the factor against the class mean; the discussion also compares its allocation of learning to prioritized experience replay.Source: A Extended Method Discussion · p. 11
• Unlike interventions that regulate how much of an update survives ratios, clipping, or bonuses, SCARL changes sequence-level reinforcement in advantage space and is described as complementary to them.The paper identifies DAPO’s clip-higher, entropy bonuses or covariance corrections, GSPO, and SAPO as interventions acting in other spaces or for other aims.Source: A Extended Method Discussion · p. 11, Table 4 · p. 11
• The confidence-aware comparison organizes methods by confidence signal, modified quantity, modification form, outcome classes, sign protection, and confidence reference point.Source: A Extended Method Discussion · p. 11, Table 5 · p. 12

B Additional Results

• 14 of 16 estimator–backbone pairings improve with SCARL; gains average +1.6 on Qwen3-4B-Base, +1.1 on Qwen3-1.7B-Base, and +0.6 on each of the other two backbones.By estimator, gains average +1.7 for DAPO, +1.3 for GRPO, +0.7 for GSPO, and +0.2 for SAPO.Source: B Additional Results · p. 12, Table 7 · p. 14
• The authors relate smaller gains on the distilled and Hunyuan backbones to less repairable misalignment: one already has high coverage, and the other high confidence–correctness separation.Across all 32 trained runs, coverage exceeds the untrained backbone; adding SCARL widens that margin in 14 of 16 pairings.Source: B Additional Results · p. 12
• 58.0 average Pass@32 versus 56.4 for the best baseline, with a +2.2 margin over DAPO, makes DAPO + SCARL the top method on Qwen3-1.7B-Base.It wins AIME25 and Minerva outright and stays close to the best baseline on the other benchmarks.Source: B Additional Results · p. 12, Table 6 · p. 13
• 0.035/0.050 on GRPO, 0.50/1.02 on DAPO, 0.12/0.49 on GSPO, and 0.07/0.73 on SAPO are the final-ten-step entropy means for base versus +SCARL.SCARL’s confidence-separation statistic exceeds the base run at 83%, 95%, and 92% of steps in the three reported settings, and at 149 of 150 setting-steps over the final fifty steps.Source: B Additional Results · pp. 12–13
• Table 8 provides per-benchmark results for component ablations and a temperature sweep on Qwen3-1.7B-Base with GSPO.The default τ = 5 row duplicates the full-SCARL run for reference.Source: B Additional Results · p. 12, Table 8 · p. 15

C Formal Statements and Proofs

• SCARL’s shaping factors are strictly positive and, for classes with at least two samples, each factor is smaller than the class size.Multiplying an advantage by a positive factor preserves its sign; a zero advantage remains zero.Source: C Formal Statements and Proofs · p. 16
• For each nonempty target class, the normalized shaping factors sum to one, preserving class mass.The within-class empirical mean of the factors is exactly one.Source: C Formal Statements and Proofs · p. 16
• When confidence is constant within each target class, every shaping factor is one and SCARL exactly matches its base estimator.This includes the case where all group confidences are equal; shaping is active only when confidence varies within a class.Source: C Formal Statements and Proofs · p. 16
• Without class-wise normalization, uniform confidence at group size G = 8 gives positive and negative raw gaps in a 7:1 ratio, and omitting normalization reduces average Pass@32 by 9.4 points below the untrained backbone.Class-wise normalization removes the cross-class scaling distortion while leaving within-class allocation unchanged.Source: C Formal Statements and Proofs · p. 16

D Reference Implementation and Practical Notes

• The implementation operates on one prompt group and consumes quantities group-relative trainers already materialize, without touching other trainer components.The released code names the stabilizer eps, corresponding to δ in the equation.Source: D Reference Implementation and… · pp. 16–17
• Shaping factors are computed once per rollout from the old policy and reused unchanged across that epoch’s minibatches, with no gradient flowing through them.If old-policy log-probabilities are recomputed, confidence should use the same source as the importance ratios.Source: D Reference Implementation and… · p. 16
• The reference function computes sequence confidence from masked old-policy log-probabilities, forms group-relative softmax gaps, normalizes them within advantage-sign classes, and scales the advantages.It uses the same log-probability source as the importance ratios.Source: D Reference Implementation and… · p. 17
• The practical notes allow either standard-deviation convention when used consistently and describe the released max(σG, 10^-8) stabilizer as differing from the equation only near-zero dispersion.The historical flag scarl_sign_aware_weighting names the same alignment-gap shaping computation.Source: D Reference Implementation and… · p. 17

E Training and Evaluation Details

• All methods share the stated training and evaluation configuration, while estimator recipes vary only in surrogate or advantage transformation.The configuration uses one node with eight NVIDIA A100 40GB GPUs, vLLM rollout, and FSDP training.Source: E Training and Evaluation Details · pp. 17–18, Table 9 · p. 17
• The confidence-aware baselines use their native configurations; UCAS is reported with repository defaults after comparing them with the paper’s recommended settings.Entropy Advantage Shaping uses GRPO, while SEED-GRPO applies semantic-entropy weighting to a Dr. GRPO advantage.Source: E Training and Evaluation Details · pp. 17–18, Table 10 · p. 17
• Pass@32 uses a bootstrap estimator averaging any-correct outcomes over 1,000 size-32 resamples from each problem’s 32 responses.Evaluation uses temperature 0.6, top-p 0.95, top-k 20, and a maximum of 4,096 response tokens.Source: E Training and Evaluation Details · pp. 17–18

E.1 Prompt Templates

• Answer-line prompts and the DAPO scorer are used for training, AIME24, AIME25, and AMC23, whereas MATH-500 and Minerva Math use boxed prompts and Math-Verify.Both conventions present each problem as a single user turn through the tokenizer’s chat template.Source: §E.1 · p. 18

F Benchmark Details

• The evaluation sets are MATH-500, Minerva Math, AMC23, AIME24, and AIME25.MATH-500 is a 500-problem subset of MATH; Minerva Math is a quantitative-reasoning set, and the remaining sets collect public competition problems from their named years.Source: F Benchmark Details · p. 18, Table 11 · p. 18