Risk-Sensitive RL for Alleviating Exploration Dilemmas in Large Language Models

1 Introduction

• RLVR often sharpens pretrained policies around homogeneous solutions, improving pass@1 but stagnating or degrading pass@k.This concentration limits discovery of novel reasoning strategies.Source: §1 · pp. 1–2
• Sharply peaked initial policies can trap standard RL near suboptimal modes, preventing discovery of diverse reasoning paths.Source: §1 · pp. 1–2
• RS-GRPO replaces mean-reward optimization with a risk-seeking objective interpolating between mean and maximum rewards.Its risk-sensitive advantage emphasizes hard prompts and under-explored solution regions.Source: §1 · p. 2
• RS-GRPO is a drop-in advantage-calculation replacement that significantly improves pass@k while maintaining or enhancing pass@1.The experiments cover six mathematical reasoning benchmarks and six LLMs.Source: §1 · p. 2
• The paper provides theoretical and empirical evidence that risk-sensitive optimization avoids local-optimum failure when the initial policy is sharply peaked.Source: §1 · p. 2

2 Related Work

• Intrinsic-motivation methods adapted from RL can require auxiliary networks, complicating training and scaling for LLMs.Source: §2 · pp. 2–3
• Entropy maximization, self-reflection, and entropy-dynamics methods offer alternative approaches to preventing exploration collapse.Source: §2 · p. 3
• Compared with many pass@k methods, risk-sensitive RL handles both continuous and binary rewards.Several prior methods are restricted to binary reward signals.Source: §2 · p. 3
• Risk-sensitive RL produces a denser advantage signal because it retains a non-zero gradient for high-accuracy prompts, supporting the pass@k–pass@1 trade-off.Source: §2 · p. 3
• Risk-sensitive RL extends risk-sensitive decision-making toward risk-seeking behavior intended to promote exploration from sharply peaked pretrained policies.Source: §2 · p. 3

3 Background

• Language generation is modeled with policy πθ, which samples response y for prompt x and receives reward r(x, y).Source: §3 · p. 3
• The standard objective maximizes expected reward through a policy-gradient update weighted by the advantage function.Source: §3 · p. 3
• In practice, the gradient estimate samples N responses per prompt from πθ(·|x).Source: §3 · p. 3
• Pass@k estimates the probability that at least one of k generated responses is correct, serving as an inference-time exploration objective.Source: §3 · p. 4

4 Risk-Sensitive Reinforcement Learning

• The objective values all high-reward outcomes and smoothly interpolates between mean-reward and maximum-reward optimization.Source: §4 · p. 4
• β controls risk sensitivity: β →0 recovers expected reward, whereas β →+∞ approaches maximum reward and encourages exploration.Source: §4 · p. 4

4.1 Policy Gradient for the Risk-Sensitive Objective

• The risk-sensitive policy gradient is derived theoretically and approximated in practice using sampled responses and an empirical advantage.Source: §4.1 · pp. 4–5
• Only the advantage computation changes, making the risk-sensitive advantage a drop-in replacement for existing GRPO-based algorithms.Source: §4.1 · p. 5

4.2 Analysis of the Risk-Sensitive Advantage Function

• As β increases under continuous rewards, the advantage sharpens toward high-reward samples while suppressing low-reward samples.Source: §4.2 · p. 5
• Under binary rewards, increasing β prioritizes correct responses on hard prompts and shifts total advantage toward lower-accuracy prompts.Source: §4.2 · p. 5
• Unlike several pass@k methods, the formulation supports continuous rewards and retains non-zero gradients on high-accuracy prompts.This provides a denser signal for balancing pass@k and pass@1.Source: §4.2 · p. 5

5 Why Risk-Sensitive RL is Better

• Standard mean-reward optimization can remain trapped in high-probability local optima when the initial policy is sharply peaked away from the global optimum.The section frames this as the exploration problem addressed by risk-sensitive RL.Source: §5 · p. 5

5.1 Empirical Perspective

• Reward 1.0 is reached by risk-sensitive policies with β ≥4, whereas β = 0 and β < 4 remain trapped at the suboptimal reward 0.6.The bandit initializes the policy sharply around a suboptimal arm, modeling pretrained-policy bias.Source: §5.1 · p. 6, Figure 3 · p. 6

5.2 Theoretical Perspective

• Risk-sensitive and standard updates are compared after one policy-gradient step in a bounded-reward K-armed bandit with a unique optimal arm.The theoretical setup uses softmax policies and equal learning rates for the comparison.Source: §5.2 · p. 6
• Standard policy gradients can decrease the optimal action’s probability when an intermediate-reward suboptimal action exceeds the current average reward.Lemma 2 identifies policy parameters for which this decrease occurs.Source: §5.2 · p. 6
• For sufficiently large β, risk-sensitive updates increase the optimal action’s probability, explaining their ability to escape the local optimum.Lemmas 2 and 3 connect the theoretical update behavior to the empirical bandit result.Source: §5.2 · p. 7
• Beyond a threshold, increasing β still improves the optimal action but yields smaller gains, so β should balance exploration against convergence speed.The diminishing improvement is formalized by Lemma 4 and aligned with the convergence behavior in Figure 3.Source: §5.2 · p. 7

6.1 Setup

• The experiments train six base models on mathematical reasoning and evaluate six benchmarks, generating up to 1024 candidate solutions per problem.MATH500 uses N = 32 for computational feasibility, while most other benchmarks use N = 1024.Source: §6.1 · p. 8

6.2 Performance on Pass@k evaluation of Risk-sensitive RL

• RS-GRPO consistently improves pass@k over GRPO across five LLMs and six benchmarks, although some models remain below the base model at high k.For Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, and Qwen3-4B, GRPO can underperform the base model when k > 256; RS-GRPO exceeds the base model in those cases.Source: Figure 4 · p. 7, §6.2 · p. 8

6.3 Different Impact of Risk-sensitive Hyperparameter β

• Increasing β improves cumulative training solve rate while slowing reward growth, and β = 2 yields roughly 5% pass@32 gains with maintained or improved pass@1.On Qwen2.5-Math-7B, β = 2 produces a 1–2% pass@1 improvement over standard RL.Source: §6.3 · pp. 8–9, Figure 5 · p. 8

6.4 Comparison to Other Pass@K Optimization Baselines

• Compared with other pass@k methods, RS-GRPO generally matches pass@32 while consistently outperforming them in pass@1.The authors attribute this to denser advantage signals from the risk-sensitive objective; methods with strictly positive advantages can cause entropy collapse.Source: §6.4 · pp. 9–10
• RS-GRPO exceeds GRPO by about 2% in average pass@1 across three models and by about 4% in average pass@32 across comparative datasets.Table 2 covers five models and three training datasets, while RS-GRPO generally matches baseline pass@32 and improves pass@1 in the detailed comparison.Source: Table 2 · p. 9, §6.4 · p. 10

6.5 Analysis of Pass@k Improvement

• RS-GRPO finds more unique answers than GRPO, indicating increased diversity of reasoning paths.The analysis samples 1024 solutions per problem across five mathematical benchmarks using Qwen2.5-Math-7B trained on deepmath103k.Source: §6.5 · p. 10, Figure 7 · p. 10
• The accuracy transition shows 8% of GRPO-unsolved prompts entering the (0, 0.4] range under RS-GRPO, versus 3% shifting oppositely.Meanwhile, 3% of prompts move from accuracy 1 to (0.6, 1), versus 1% in the opposite direction, explaining larger pass@k than pass@1 gains.Source: §6.5 · p. 10, Figure 7 · p. 10

• RS-GRPO addresses the exploration dilemma by using a risk-seeking objective to explore under-explored solution regions and discover novel reasoning paths.Source: §7 · p. 10
• Experiments on mathematical reasoning benchmarks show improved pass@k while maintaining or improving pass@1, producing a more favorable trade-off than prior methods.Source: §7 · p. 10

• All experiments use a fixed risk-seeking parameter β; tested schedules and difficulty-based adaptation did not improve pass@1 beyond fixed β = 2.The authors identify dynamic balancing of exploration and exploitation as an open problem.Source: A Limitations · p. 15

B Details About Other Pass@k Optimization

• In binary-reward analysis, positive and negative advantages distinguish correct and incorrect responses, while cumulative advantage sums absolute values across responses.Source: B Details About Other Pass@k… · p. 15, Figure 8 · p. 16
• Prior methods include binary-reward objectives [9], leave-one-out best-of-N training [56], and smoothed maximum optimization [57].Source: B Details About Other Pass@k… · p. 15
• RS-GRPO supports continuous rewards, whereas methods such as [9] are confined to binary rewards.Source: B Details About Other Pass@k… · p. 16
• Existing pass@k methods can produce vanishing advantage estimates once sample accuracy exceeds a threshold, eliminating optimization weight.The magnitude of the advantage estimate drops to zero in the Cumulative column of Fig. 8.Source: B Details About Other Pass@k… · p. 17
• With β = 2 and baseline k = 4, RS-GRPO provides denser advantage signals; methods [9] reach comparable pass@32 but substantially lower pass@1.Walder and Karkhanis [57] instead shows persistently positive advantages, rapid entropy collapse, and poor training performance.Source: B Details About Other Pass@k… · p. 17

C.1 Proof for Theorem 1

• The risk-sensitive policy-gradient proof uses the log-derivative trick and subtracts a 1/β baseline to obtain an unbiased, lower-variance estimator.Source: §C.1 · p. 17

C.2 Theoretical Analysis of Risk-Sensitive Policy Gradient

• The analysis models a softmax policy over K actions, defines optimal-action mass, and compares one-step standard and risk-sensitive policy-gradient updates under shared learning rate α.Source: §C.2 · p. 17
• The formal results assume nontrivial reward variation, while allowing multiple optimal actions and separately identifying the unique-optimum case used in simpler analysis.Source: §C.2 · pp. 17–19
• Under an intermediate-reward action and suitable policy conditions, a risk-sensitive update can reduce probability assigned to every optimal action, including during early training.The required conditions can occur when policy mass concentrates on suboptimal actions.Source: §C.2 · p. 18
• For sufficiently large β, non-optimal actions can retain negative but increasing advantages, while optimal-action advantages remain positive but decrease as β grows.Source: §C.2 · p. 20

D.1 Entropy Analysis

• Entropy analysis finds that larger β typically improves training solve rate and exploration, but entropy does not consistently increase with β and may be a biased exploration indicator.The lowest entropy occurs at β = 0.Source: §D.1 · p. 20, Figure 9 · p. 21

E Implementation Details

• The cited dataset references include math12k, dapo17k, deepmath103k, CMIMC25, MATH500, and AIME24, with repository identifiers provided for each.Source: E Implementation Details · p. 21
• The VeRL implementation uses vLLM, omits KL regularization, applies model-dependent response limits, and uses Math_Verify2 with binary correctness rewards.Source: E Implementation Details · p. 21
• Experiments keep hyperparameters identical; RS-GRPO uses β = 8 in Fig. 4 and β = 2 against pass@k baselines using k = 4.Source: E Implementation Details · p. 21, Table 3 · p. 22
• Evaluation samples use temperature 1.0 and top-p 0.7, with N = 1024 for most benchmarks and N = 32 for MATH500.Source: E Implementation Details · p. 21
• Testing pass@1 averages six benchmarks, whereas testing pass@32 averages five benchmarks excluding MATH500, using the unbiased estimator from [8].Source: E Implementation Details · pp. 21–22
• The bandit experiments use 100 deterministic-reward actions, a softmax policy, batch size N = 16, and learning rate 0.1.Source: E Implementation Details · p. 22