PaperlayerView exact revision on arXiv

Source-linked AI summary

S3AC: Stabilized, Selective, and Scalable Actor-Critic for LLM Reasoning

Shuo Shen, Guangran Cheng, Chengqi Lyu, Xiangsen Wang, Yanan Sun, Wenwei Zhang

arXiv:2610.35518v1cs.LGcs.CL

TL;DR

Critic-free methods for long-CoT RLVR lack token-level credit assignment and often rely on grouped rollouts for prompt-specific baselines, while training a critic is difficult. S3AC trains a critic for token-specific advantages, prompt-specific baselines, and pre-rollout prompt selection through DAPS. It outperforms GRPO-X across evaluated group-rollout benchmarks and sustains single-rollout training at 35B.

  • Problem

    Critic-free RLVR methods commonly lack token-level credit assignment and use grouped rollouts for prompt-specific baselines, while long-CoT critic training is challenged by limited trajectory coverage and early value-head gradients.

    Source: §1 · p. 1, §3.3 · p. 4

  • Method

    S3AC pretrains and updates a critic to provide prompt-specific baselines and token-level GAE advantages, then reuses its values with outcome history in DAPS to select prompts before generation.

    Source: §1 · p. 2, §3 · p. 3, §3.3 · p. 4

  • Results

    S3AC outperforms GRPO-X on all seven evaluated benchmarks at 35B and all five at 397B, and sustains single-rollout performance at 35B while tested critic-free baselines decline after initial gains.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    DAPS reuses the trained critic for prompt selection, reducing discarded responses under group rollout and improving performance in both rollout settings.

    Source: §5 · p. 8

    Experiments cover reasoning-style verifiable-reward tasks, not agentic multi-turn RL, and do not systematically assess variability across independent training seeds.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Critic-free methods are widely used in reinforcement learning with verifiable rewards (RLVR) for long chain-of-thought (CoT) reasoning, avoiding difficult and costly critic training. Common alternatives lack token-level credit assignment and typically rely on grouped rollouts to estimate prompt-specific baselines. A critic can address both limitations, but long-CoT critic training is hindered by insufficient coverage of trajectories with diverse lengths and early gradients dominated by the newly initialized value head. We introduce S3AC (Stabilized, Selective, and Scalable Actor-Critic) for LLM reasoning, built around a practical critic-training recipe. The trained critic provides token-specific advantages and prompt-specific baselines. Using the prompt-level baseline to estimate difficulty, we propose Difficulty-Aware Prompt Sampling (DAPS). DAPS combines this estimate with outcome history to select prompts before generation, reducing rollout waste without offline difficulty initialization. S3AC improves group-rollout performance across dense and mixture-of-experts models from 1.5B to 397B parameters, outperforming a modern GRPO variant (GRPO-X). At 35B, it sustains single-rollout training while tested critic-free baselines deteriorate after initial gains.

1 Introduction

Critic-free RLVR methods lack prompt- and token-specific learning signals, while long-CoT critic training faces coverage and optimization challenges. S3AC trains a critic for policy learning and prompt selection, improving results across model scales and rollout settings.

1 Introduction

  • Critic-free methods lack prompt-specific baselines and token-level credit assignment, while long-CoT critic training faces limited supervision on long correct responses and early gradients dominated by the new value head.Grouped-rollout baselines wait for the slowest response, while shared batch statistics do not account for prompt difficulty.Source: §1 · p. 1
  • S3AC broadens critic pretraining across policy strengths, controls initial value predictions, and relaxes critic gradient clipping to support online token-specific advantages.The critic uses generalized advantage estimation (GAE) throughout the response.Source: §1 · pp. 1–2
  • DAPS combines pre-rollout critic estimates with outcome history to select prompts of intermediate estimated difficulty, reusing the policy-learning critic without offline difficulty initialization.The prompt-conditioned estimate is available before rollout.Source: §1 · p. 2
  • S3AC outperforms GRPO-X on all seven evaluated benchmarks at 35B and all five at 397B, and sustains single-rollout performance at 35B while tested critic-free baselines decline after initial gains.Evaluations span two dense and two multimodal MoE models from 1.5B to 397B; DAPS also improves both rollout settings and reduces trajectories discarded by group filtering.Source: §1 · p. 2

2 Related Work

Prior work addresses critic training, critic-free policy optimization, and prompt-difficulty sampling through separate approaches. S3AC combines critic initialization, online value learning, and critic-guided prompt selection in one framework.

2 Related Work

  • Prior critic-based methods target distinct aspects of actor–critic training, while S3AC brings critic initialization, online value learning, and critic-guided prompt selection together.Examples include VC-PPO’s value pretraining and decoupled GAE parameters [Yuan et al., 2025], VAPO’s response-length adaptation [Yue et al., 2025b], and value-estimation approaches including Monte Carlo rollouts, explained-variance gating, and critic ensembles [Pan et al., 2026].Source: §2 · p. 2
  • Critic-free methods derive baselines from grouped responses, batch statistics, or per-prompt tracking, typically using sequence-level reward statistics rather than values over intermediate response prefixes.Examples include GRPO, RLOO, and DAPO [Shao et al., 2024], REINFORCE++ [Hu et al., 2025], FlashREINFORCE, and SPO.Source: §2 · p. 2
  • Existing difficulty samplers use multiple responses and discard uninformative groups, reward history, offline-initialized tracking, or a separate prompt-value model; DAPS combines pass-rate history with the existing S3AC critic.DAPS avoids a separately trained difficulty model and supports both group and single rollout.Source: §2 · pp. 2–3

3 Method

S3AC uses a learned critic to provide prompt-conditioned baselines and token-specific advantages, and reuses its prompt estimate to guide prompt selection before generation. Its method combines actor–critic updates with critic pretraining and online adaptation, plus Difficulty-Aware Prompt Sampling (DAPS).

3 Method

  • S3AC trains a value model for policy learning and prompt selection, using prompt-conditioned baselines and token-specific advantages.The critic’s initial-state value also guides prompt sampling before generation.Source: §3 · p. 3

3.1 Actor–Critic Formulation

  • A token-level MDP represents each generated token as an action conditioned on the prompt and response prefix, with a verifier reward on the final token.The critic estimates expected return at each state; GAE uses its values to compute token-specific advantages, and the actor is updated with CISPO.Source: §3.1 · p. 3

3.2 How the Critic Shapes Learning Signals

  • The critic’s values track prompt-level differences in empirical returns, while its changing values across response positions produce distinct learning signals even for failed responses.Figure 2 compares initial values with held-out returns and illustrates raw advantages for long failures; Figure 3 shows values rising as reasoning identifies candidate values and the requested term.Source: §3.2 · p. 3, Figure 2 · p. 3, Figure 3 · p. 4

3.3 Training a Good Critic

  • Value pretraining pools trajectories from policies of different strengths to broaden coverage across response lengths and outcomes, while small value-head initialization and a higher critic clipping threshold address early head-dominated gradients.The critic is then updated online as the policy evolves, using a higher learning rate and more epochs than the actor, with separate GAE settings.Source: §3.3 · p. 4

3.4 Difficulty-Aware Prompt Sampling

  • DAPS combines the initial critic value with outcome history to select prompts of intermediate estimated difficulty before generation.New prompts can be scored directly from the critic without offline rollouts; the acceptance rule retains a chance of selecting very easy or very hard prompts, and DAPS improves performance in both rollout settings while reducing group-rollout discards.Source: §3.4 · p. 5

4 Experiments

S3AC was evaluated across dense and MoE models, group and single rollout, critic pretraining choices, and difficulty-aware prompt sampling. It improved benchmark performance and training progress across these settings.

4.1 Experimental Setup

  • The experiments cover two dense 1.5B models and two MoE models, with group rollout using 16 responses per dense prompt and eight per MoE prompt, versus one for single rollout.Baselines share each model’s checkpoint, training data, and evaluation protocol.Source: §4.1 · p. 5

4.2 Main Results

  • 80.5 versus 76.4 at 35B and 87.3 versus 85.1 at 397B were the group-rollout averages, with S3AC gains on all seven and five benchmarks, respectively.The results support scalability from 1.5B dense models to 397B MoE models.Source: §4.2 · p. 6
  • At 35B, S3AC scored 80.9 versus 70.4 for REINFORCE++ under single rollout, a 10.5-point gain over the best critic-free baseline.S3AC retained most of its gains through 1,600 gradient steps while both tested critic-free baselines declined after initial improvement.Source: §4.2 · pp. 6–7

4.3 Analysis of Critic Pretraining

  • Coverage pretraining produced higher early explained variance and lower value loss than initial-policy-only pretraining, with faster learning and higher final performance at the same actor-update budget.For responses longer than 24K tokens, correct samples were 5.0% under the initial policy and 61.8% at gradient step 1,600; the comparison also differed in data amount.Source: §4.3 · p. 7, Figure 5 · p. 7
  • Reducing value-head initialization standard deviation to 1/(H+1) lowered early value loss from 0.72 to 0.45; raising the clipping threshold from 1 to 10 cut clipped updates from 99.7% to 12.5%.The higher threshold retained protection against extreme gradients while final value loss remained comparable in that separate initial-policy-data ablation.Source: §4.3 · pp. 7–8, Table 4 · p. 8

4.4 Difficulty-Aware Prompt Sampling

  • DAPS improved best evaluated scores on Intern-S2-Preview-35B and R1-Distill-1.5B under both rollout settings and reduced trajectories discarded by group filtering.It also allocated a larger share of training trajectories to prompts with 0.2 < pref ≤0.8 after the initial sampling ramp.Source: §4.4 · p. 8, Table 5 · p. 8

5 Conclusion

S3AC supports long-CoT RLVR with a critic-training recipe that supplies prompt-specific baselines and token-level advantages, while DAPS uses critic values and outcome history to select prompts before generation.

5 Conclusion

  • S3AC combines broader trajectory coverage, small-scale value-head initialization, and relaxed critic gradient clipping to train a critic for long-CoT RLVR.The critic supplies prompt-specific baselines and token-level advantages.Source: §5 · p. 8
  • Across 1.5B–397B models, the actor–critic approach outperforms GRPO-X under group rollout and sustains stable single-rollout training at 35B.Source: §5 · p. 8
  • DAPS reuses critic values and outcome history to select prompts before generation, reducing discarded responses under group rollout and improving performance in both rollout settings.Source: §5 · p. 8

Limitations

The evaluation is limited to reasoning tasks with verifiable rewards, and does not systematically measure variability across independent RL training seeds; S3AC also incurs additional compute overhead.

Limitations

  • Experiments cover math, science, and multimodal reasoning with verifiable rewards, not agentic multi-turn settings involving tool calls and environment turns.Whether the findings carry over to agentic RL remains open.Source: Limitations · p. 9
  • The reported experiments do not systematically assess variability across independent RL training seeds.Source: Limitations · p. 9
  • S3AC costs 27–40% more amortized wall time per actor gradient step than GRPO-X because it trains and runs a critic alongside the policy.Source: Limitations · p. 9

Ethical Considerations

The paper cautions that benchmark gains do not establish real-world safety or reliability, and its training configuration uses shared base checkpoints within each model alongside method-specific procedures.

Ethical Considerations

  • Higher benchmark scores do not establish safety or reliability in real-world applications, where models may still produce incorrect or harmful content.Deployment requires application-specific evaluation and safeguards.Source: Ethical Considerations · p. 9

A.1 Training Configurations

  • All methods start from the same base checkpoint within each model; group-rollout runs use dynamic sampling that discards all-correct or all-incorrect response groups.On Intern-S2-397B, S3AC and GRPO-X use an adaptive length regularizer, with adjusted rewards applied to both S3AC actor and critic training.Source: §A.1 · p. 12

A.2 Evaluation

The evaluation appendix reports model-specific sampling and scoring settings and describes the baselines’ reward normalization and entropy-control configurations.

A.2 Evaluation

  • Qwen2.5-Math-1.5B uses temperature 0.7, top-p 0.7, avg@16, and a 4,096-token combined prompt-and-response cap.Source: §A.2 · p. 12
  • R1-Distill-1.5B uses temperature 0.8, top-p 0.95, a 16,384-token response limit, and avg@32; small score differences warrant caution given repeated-evaluation variability.Intern-S2-Preview-35B and Intern-S2-397B use temperature 0.8 and top-p 1.0, with response limits of 64K and 62K tokens; their reported averaging varies by benchmark.Source: §A.2 · p. 12

A.3 Baselines

  • GRPO-X uses group z-score advantages, while REINFORCE++ normalizes rewards by batch mean and standard deviation; the MoE GRPO-X baselines also use GEPO entropy control.FlashREINFORCE uses a batch-mean baseline and sequence-level trust region.Source: §A.3 · p. 12

B Additional Results and Training Dynamics

The appendix reports stable training for several single-rollout policy-loss variants and provides additional training curves, statistics, and critic-pretraining settings. It also documents the data sources and configuration used for the 35B critic experiments.

B Additional Results and Training Dynamics

  • Table 7 reports training statistics, evaluation response lengths, and truncation rates for the runs.Training metrics average 10 batch-level records over 80 actor gradient steps; a dash denotes unavailable statistics, and REINFORCE++ uses a post-collapse window.Source: B Additional Results and Training… · p. 13, Table 7 · p. 13
  • CISPO, DIS, and sequence-masked DIS scored 79.8–80.2 while training stably from the same actor and critic, in single-seed robustness-check runs.The runs had different restart histories.Source: B Additional Results and Training… · p. 13
  • Figures 7 and 8 provide full benchmark learning curves for group-rollout Intern-S2-397B and Intern-S2-Preview-35B, respectively.Figure 7 also reports rollout entropy as a 40-gradient-step moving average; Figure 8 includes the seven-benchmark average shown in Figure 4a.Source: Figure 7 · p. 13, B Additional Results and Training… · p. 13, Figure 8 · p. 14
  • Figure 9 tracks average benchmark score, rollout entropy, and response length during single-rollout Qwen2.5-Math-1.5B training with DAPS.Scores are offline avg@16; entropy and length use 40-step moving averages with raw values also shown.Source: Figure 9 · p. 14

C.1 Experimental Setup

  • The 35B coverage critic uses 327,680 trajectories (7.08B response tokens), versus 65,536 trajectories (0.74B tokens) for the initial-policy-only critic.At 1,600 actor gradient steps, their seven-benchmark averages are 80.78 and 79.86, respectively.Source: §C.1 · p. 15
  • Value pretraining uses two epochs of terminal-reward squared-error loss, a value-head initialization standard deviation of 1/(H+1), and gradient clipping at 10.Attention, the MoE router, and any vision tower and projector are frozen; the learning rate decays from 10^-5 to 5 × 10^-6 after a 3% warmup.Source: §C.1 · p. 15

C.2 Data Coverage and Token-Level Values

The appendix reports broader long-response coverage in the 35B coverage critic’s pretraining data and illustrates how its values track reasoning progress. It also describes gradient diagnostics, comparisons across critic scales, and the DAPS evaluation plots.

C.2 Data Coverage and Token-Level Values

  • Among responses longer than 24K tokens, 5.0% were correct in the initial-policy sample versus 61.8% at gradient step 1,600, illustrating the coverage critic’s access to rarer long correct trajectories.Figure 10 compares response-length distributions and accuracy by length, with bins spanning 2K tokens.Source: §C.2 · p. 15, Figure 10 · p. 15
  • In two examples, the coverage critic’s value rises as responses identify candidate values and determine the requested term, or compute 0.1622 and match it to option H (0.1614).These examples illustrate fine-grained value estimates for token-level credit assignment.Source: §C.2 · p. 15, Figure 11 · p. 16

C.3 Value-Pretraining Optimization

  • Figure 12 measures head and backbone gradient norms over the first 10 critic gradient steps of 35B value pretraining, before clipping, against a threshold of 10.The passage identifies the figure as supporting the gradient-imbalance analysis but does not report the measured norm values.Source: Figure 12 · p. 15, §C.3 · p. 15

C.4 Value Pretraining at Different Model Scales

  • The 397B critic achieved lower and more stable value loss and smaller gradient norms than the 35B critic in the reported pretraining runs.The 35B run used 1.13M trajectories (6.24B response tokens); the 397B run used 1.85M trajectories (10.0B response tokens).Source: §C.4 · p. 16, Figure 13 · p. 16

D DAPS Details

  • Figure 14 compares DAPS with uniform sampling on 35B and R1D using group-rollout scores, single-rollout scores, and cumulative discarded trajectories.Scores are seven-benchmark macro averages for 35B and AIME avg@64 for R1D; the caption describes the plotted measures but does not establish a winner.Source: Figure 14 · p. 17

D.1 Experimental Setup

The setup specifies prompt-scoring and acceptance-ramp settings and explains how paired DAPS comparisons are evaluated. The reported additional results show a sustained late-stage 35B advantage, while R1D gains vary and prompt scoring adds computation.

D.1 Experimental Setup

  • DAPS settings use three candidates and a 0.99 count-decay factor per rollout batch, with 40 initial uniform-sampling steps followed by an 80-step acceptance ramp.Prefetched prompts are rescored after each rollout batch using a window twice the previous candidate count; Table 9 lists prompt batch sizes and acceptance floors.Source: §D.1 · p. 17, Table 9 · p. 17
  • The paired ablations retain 4,096 prompts per 35B single-rollout batch, while the main results use 8,192; R1D uses two 32-response evaluations per question to obtain AIME 2024/2025/2026 avg@64.Each pair reports each method’s best evaluated checkpoint within a common gradient-step budget, while Figure 14 shows scores at each checkpoint.Source: §D.1 · p. 17

D.2 Additional Results

  • Figure 14 shows a sustained late-stage DAPS advantage for 35B under both group and single rollout, while R1D gains vary across training.Under group rollout, DAPS reduces cumulative discarded trajectories on both models; prompt scoring adds computation, particularly with larger single-rollout batches.Source: §D.2 · p. 17

Related papers

Loading 2610.35518v1…