PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning Done Right for Explicit Confidence Targets

Zefeng He, Yichen He, Yuan Lin, Yang Yang, Yu Cheng

arXiv:2610.35059v1cs.LGcs.CL

TL;DR

LLMs can give plausible but incorrect answers, while fixed-penalty abstention cannot adapt to varying confidence requirements. RLEC trains a single model to respond to user-specified confidence targets by preserving target signals and stabilizing rewards. Experiments report target control across benchmarks and model scales, with general capabilities largely preserved.

  • Problem

    Fixed error penalties cannot adapt an abstaining model to different confidence requirements at inference, and prior single-model attempts have struggled to distinguish target values.

    Source: §1 · pp. 1–2

  • Method

    RLEC preserves confidence-target learning signals, bounds reward magnitudes without changing the desired answer condition, and schedules rewards to balance learning and stability.

    Source: §1 · p. 2

  • Results

    Across diverse benchmarks and model scales, RLEC makes explicit-target training feasible, largely preserves general capabilities, achieves strong calibration, and supports fine-grained control.

    Source: §1 · pp. 2–3, §6 · p. 12

  • Takeaways & Limitations

    RLEC directly incorporates user-specified confidence targets into the policy, enabling answer-or-abstain control without an external decision system.

    Source: §1 · p. 3, §6 · p. 12

    The stated selective-accuracy bound is idealized, not an empirical guarantee, because learned policies may optimize or generalize imperfectly and uncertainty may not match true correctness probabilities.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

Large language models (LLMs) can produce plausible but incorrect answers when they are uncertain, which can mislead users and undermine model reliability. Training models to abstain with reinforcement learning is a natural approach, but most existing methods use a fixed error penalty and therefore cannot adapt their behavior to different confidence requirements at inference time. We study \emph{explicit confidence targets}, where a user specifies a target \(p\) in the prompt and the model directly chooses whether to answer or abstain, with the goal of achieving an accuracy of at least \(p\) among answered questions. However, learning to respond to different confidence targets with a single model has proven difficult, and prior attempts have failed to distinguish different targets. We show that this failure comes from two problems: the reward signal that distinguishes different values of $p$ is often lost during optimization, while the original error penalty grows without bound as $p$ approaches one. We propose RLEC (\textbf{R}einforcement \textbf{L}earning with \textbf{E}xplicit \textbf{C}onfidence Targets), which preserves the learning signal needed to distinguish confidence targets, introduces a stable reward family that keeps the desired answer condition unchanged, and uses reward scheduling to balance effective learning and training stability. Together, these designs enable the first successful training of a single LLM for explicit confidence target control. Extensive experiments across diverse benchmarks and model scales show that RLEC largely preserves general capabilities and achieves competitive calibration performance. RLEC also supports fine-grained target control without an external decision system or threshold calibration after training. By making explicit confidence targets part of the trained policy, RLEC opens a new and more direct path toward behavioral calibration.

1 Introduction

Fixed error penalties cannot express different user confidence requirements, and prior single-model attempts struggled to adapt to targets. RLEC addresses this by preserving target-dependent learning signals while stabilizing rewards, and experiments show target control across benchmarks and model scales.

1 Introduction

  • A fixed error penalty represents only one risk preference, whereas explicit confidence targets let a user specify p and ask one model to answer or abstain accordingly.Earlier ERT attempts did not adapt hallucination rates across targets, leaving open whether the difficulty arose from model capacity or training method.Source: §1 · p. 2
  • GRPO can erase target-dependent reward-scale information: groups with one reward value yield zero advantages, while standard-deviation normalization removes scale effects in two-value groups.The resulting advantages depend on the fraction of the two reward values rather than their magnitudes.Source: §1 · p. 2
  • The original error reward grows without bound as p approaches one, so large-p samples can dominate gradients and destabilize training once reward magnitudes affect advantages.RLEC uses a stability coefficient and schedules it because fixed values trade off learning-signal strength against training stability.Source: §1 · p. 2
  • RLEC learns different behaviors for explicit confidence targets across eight benchmarks, largely preserves RLVR capabilities at both model scales, and supports smooth target control on AIME.It achieves calibration performance comparable to RLCR at 4B, and a lower overconfidence rate and higher overall score at 30B.Source: §1 · pp. 2–3
  • RLEC preserves confidence-target signals, stabilizes reward magnitudes without changing the desired decision rule, and schedules rewards to balance learning strength and training stability.The paper reports this enables the first successful training of a single LLM for explicit confidence-target control.Source: §1 · p. 2

2 Background

Fixed-penalty abstention makes the answer-versus-abstain choice depend on a single error penalty, while explicit confidence targets put a desired accuracy level in the prompt. The section explains the reward-based decision rules and the challenge of training one policy to follow varied targets.

2.1 Fixed-Penalty Abstention

  • A fixed-penalty reward gives correct answers 1, abstentions 0, and incorrect answers −λ; it makes λ a single operating point, so adapting to different error costs generally requires another model or retraining.TruthRL uses λ = 1 with GRPO, while Reinforced Hesitation trains separate models for different λ values.Source: §2.1 · p. 3
  • Under the idealized assumptions that q(x) is the true correctness probability and the policy follows the threshold rule exactly, selective accuracy satisfies the stated bound, but this is not an empirical guarantee.Learned policies may optimize or generalize imperfectly, and their uncertainty may not match true correctness probabilities.Source: §2.1 · p. 3

2.2 Explicit Confidence Targets

  • Explicit confidence targets let users specify required accuracy p in the prompt, and a reward-maximizing policy answers when q(x) > p and abstains when q(x) < p.At q(x) = p the policy is indifferent; at p = 0, ties are resolved in favor of answering.Source: §2.2 · p. 4
  • The goal is one policy that follows arbitrary p values without retraining, with the model directly choosing answer or abstention rather than relying on an external confidence-threshold system.The policy should answer when expected reward is positive, not default to abstention on every question.Source: §2.2 · p. 4
  • ERT sampled p during training but reported hallucination rates largely unchanged across targets; prior explanations cited noisy optimization or limited model capacity.These findings left unresolved whether the difficulty was inherent to a single model or attributable to the training method.Source: §2.2 · p. 4

3 Method

RLEC preserves confidence-target information lost by standard GRPO advantage normalization, replaces the unbounded error penalty with a stable reward family, and schedules rewards to balance learning signal and training stability.

3.1 Signal Preservation

  • Only rollout groups with all three outcomes retain the reward-scale signal for p; such groups are rare and decline as one-reward groups become dominant.With one or two reward values, standardized advantages lose the p-dependent reward scale or provide no signal.Source: §3.1 · p. 5
  • Removing group standard-deviation normalization while retaining mean-centering preserves the effect of p-dependent reward gaps on advantages.The resulting updates can therefore retain direct reward information that distinguishes confidence targets.Source: §3.1 · p. 5

3.2 Stable Rewards

  • The original incorrect-answer reward −p/(1 −p) tends to −∞ as p approaches one, allowing large-p groups to dominate gradients and destabilize training.Batch-to-batch variation in these groups can cause sharp fluctuations in gradient norms.Source: §3.2 · p. 5
  • RLEC scales rewards by a p-dependent positive factor, bounding the error penalty while preserving the theoretical answer condition: answer when q(x) > p.The reward family uses stability coefficient b to control the reduction at large p; every b preserves that condition.Source: §3.2 · pp. 5–6

3.3 Reward Scheduling

  • A relatively large fixed b can weaken learning to abstain and distinguish targets, while a small fixed b strengthens early learning but risks gradient dominance and instability.The trade-off concerns the strength of centered advantages versus training stability.Source: §3.3 · p. 6
  • RLEC linearly increases b from 0 to 1 over T training steps, retaining a stronger early signal while improving stability later.The schedule is designed to combine the learning benefit of small b with the later stability benefit of larger b.Source: §3.3 · p. 6

4 Experiment

Experiments across two Qwen3 scales and eight benchmarks test target adaptation, calibration, design choices, and additional operating properties. RLEC learns target-dependent abstention, compares favorably with RLCR at 30B, and supports fine-grained control, including beyond its training range.

4.2 Adaptation to Confidence Targets

  • RLEC’s abstention rises from approximately 19% to 49% and answered-question accuracy from approximately 68% to 81% as p increases from 0.1 to 0.9; other direct-decision methods remain nearly unchanged.On held-out HotpotQA and DAPO-Math checkpoints, RLEC separates by target while reproduced ERT curves nearly overlap.Source: §4.2 · p. 7, Figure 3 · p. 7

4.3 Comparison with Calibration Methods

  • At 30B, RLEC achieves both lower AOC and higher Score than RLCR, while at 4B it has lower AOC but lower Score and largely preserves RLVR Forced Accuracy.The comparison reports Micro results pooled over examples and Macro results averaged equally across benchmarks.Source: §4.3 · p. 8, Table 1 · p. 9

Standard Deviation.

  • Standard-deviation normalization produces nearly overlapping target curves, supporting the analysis that it weakens the reward-scale signal induced by p.Removing group standard deviation does not consistently improve or degrade answer ability.Source: Standard Deviation · p. 9

Choice of Stability Coefficient.

  • With fixed b=0, target curves separate early but fluctuate sharply; b=1 curves nearly overlap, while b=0.5 separates only late in training.Across test benchmarks, Forced Accuracy, AOC, and Score remain broadly consistent across b choices; the scheduled configuration has the highest aggregate Forced Accuracy while retaining competitive calibration.Source: Choice of Stability Coefficient · pp. 9–10

4.5.1 Fine-Grained Confidence Control

  • At 0.01 target intervals, abstention generally rises and answered-question accuracy improves, generally meeting or exceeding the target, though adjacent values are not strictly monotonic.Evaluation on AIME24 and AIME25 extends to p=0.95, although training uses targets only up to p=0.90.Source: §4.5.1 · p. 10, Figure 6 · p. 10
  • As p rises, wrong answers decrease faster than correct answers, particularly at lower targets where the correct-response proportion stays nearly unchanged.RLEC initially improves answered-question accuracy primarily by abstaining from likely errors rather than discarding correct answers.Source: §4.5.1 · p. 10

4.5.2 Implicit Confidence Profile

  • On the 30B model, RLEC’s operational profile has higher standard deviation than RLCR’s: 0.2335 versus 0.2126 for Micro and 0.2713 versus 0.2646 for Macro.The construction estimates aggregate profile mass from RLEC abstention rates without assuming a shared sample-level latent confidence variable.Source: §4.5.2 · p. 11, Figure 7 · p. 11

4.5.3 Risk-Coverage Trade-off

  • On 30B risk–coverage curves, the methods perform similarly, but RLEC reaches lower coverage in both Micro and Macro settings, enabling more conservative operating points.RLEC is slightly better in Micro, while RLCR is slightly better in Macro.Source: §4.5.3 · p. 11, Figure 7 · p. 11

5 Related Work

Prior work studies how GRPO normalization affects optimization, while RLEC uses reward scale to encode confidence targets. Related calibration methods either use fixed risk preferences or produce confidence scores that require an external decision rule; RLEC directly adapts answering and abstention to targets.

5.1 Reinforcement Learning with Verifiable Rewards

  • Prior GRPO studies report differing effects of standard-deviation normalization on optimization, bias, convergence, stability, and sample efficiency.Dr. GRPO [23], Turtel et al. [31], Bereket and Leskovec [2], Ge et al. [9], and CAL-GRPO [14] examine different benefits and drawbacks.Source: §5.1 · pp. 11–12
  • RLEC retains the confidence-target signal through mean centering without standard-deviation normalization, which otherwise removes the reward-scale effect for common two-reward groups.This helps explain why prior work [36] struggled to distinguish targets.Source: §5.1 · p. 12

5.2 Hallucination Mitigation and Behavioral Calibration

  • Fixed-risk abstention methods cannot adapt answer-or-abstain decisions to different user requirements at inference time, and ERT [36] did not learn distinct target-conditioned behaviors.The passage describes fixed-risk RL methods and ERT's reported limitation.Source: §5.2 · p. 12
  • RLCR [8] trains confidence reporting, which can require an external decision rule, whereas RLEC directly adapts answer or abstention decisions to explicit targets.Some confidence-estimation approaches also require a separate calibrator or calibration set [20, 32].Source: §5.2 · p. 12

6 Conclusion

The paper analyzes why prior explicit-target training fails and proposes RLEC to address the identified training problems. Experiments support the feasibility of explicit confidence-target training and show calibration, capability preservation, and target control without an external decision system.

6 Conclusion

  • RLEC addresses lost confidence-target signals and unstable reward scale with a stable reward family and reward scheduling.The design aims to balance effective learning and training stability.Source: §6 · p. 12
  • Experiments across diverse benchmarks and model scales show that explicit confidence-target training is feasible and RLEC largely preserves general capabilities.Source: §6 · p. 12
  • RLEC achieves strong calibration and fine-grained control without an external decision system.The conclusion describes explicit, user-specified confidence targets as a direct path toward behavioral calibration.Source: §6 · p. 12

A.1 Implementation Details

The appendix summarizes the RL training setup, evaluation conventions, and prompt choices. It also notes that computational constraints precluded repeated training runs for each setting.

A.1 Implementation Details

  • RL training uses verl, with hyperparameters summarized in Table 2; Clip-Higher, entropy regularization, and a KL penalty are not used.Source: §A.1 · p. 15, Table 2 · p. 15
  • RLCR abstention uses an external confidence threshold, while other methods use the designated abstention string; at p = 0, only correct answers count as correct.AOC is set to zero when no answers are attempted.Source: §A.1 · p. 15
  • Because computational resources limited repeated training runs, results report Avg@16 for AIME and Avg@2 for other benchmarks to reduce evaluation variance.Source: §A.1 · p. 15
  • ERT and TruthRL use the RLEC prompt, while RLVR uses its standard prompt for training and p = 0, then the RLEC prompt for p > 0.SimpleQA-Verified and AA-Omniscience use a Qwen3.6-27B grading prompt for semantic equivalence.Source: §A.1 · p. 15

A.2 Benchmarks

The appendix specifies two benchmark settings and describes the baseline reward and decision conventions. The selected MMLU-Pro subsets limit that benchmark's influence on aggregate results.

A.2 Benchmarks

  • The modified HotpotQA setting removes zero, one, or both relevant paragraphs from each eight-paragraph example to test abstention with incomplete evidence.Source: §A.2 · p. 15
  • Evaluation uses four MMLU-Pro science subsets—biology, chemistry, physics, and health—to prevent its larger benchmark from dominating aggregate results.Source: §A.2 · p. 16

A.3 Baselines

  • RLVR uses binary correctness rewards without explicitly rewarding abstention; its group standard-deviation normalization is removed in the main experiments for closer comparison.Appendix B.2 examines the effect of that choice.Source: §A.3 · p. 16
  • TruthRL [35] uses a fixed ternary reward, while RLCR [8] applies an external threshold to confidence scores and ERT [36] samples confidence targets.RLEC is compatible with TruthRL and ERT under the stated parameter choices.Source: §A.3 · pp. 16–17

B.1 Choice of Stability Coefficient

The linear schedule is the default because it balances target differentiation and training stability while preserving answer ability. Calibration improves over RLVR across stability choices, while removing group standard-deviation normalization shows no clear answer-ability effect in this experiment setting.

B.1 Choice of Stability Coefficient

  • On AIME Forced Accuracy, fixed b = 0, fixed b = 0.5, and LR Schedule each score more than three points below the linear schedule.Differences on most other benchmarks are smaller or less consistent; global learning-rate decay is not equivalent to scheduling relative reward scale across targets.Source: §B.1 · p. 18
  • All stability choices substantially improve calibration over RLVR; fixed b = 0.5 has the lowest aggregate AOC and highest Macro Score while retaining relatively strong Forced Accuracy.Source: §B.1 · p. 18
  • The linear schedule separates p = 0.9 from lower targets midway through training and maintains ordering, while fixed b = 0 fluctuates and fixed b = 0.5 separates only after about 400 steps.Larger b stabilizes updates but weakens differentiation; smaller b strengthens target-dependent learning but reduces stability, and the linear schedule establishes distinctions early before stabilizing them.Source: §B.1 · pp. 18–19
  • The linear schedule is the default, combining stronger aggregate Forced Accuracy, earlier and more persistent target separation, and a simpler form than the sinusoidal alternative.The sinusoidal schedule has comparable Macro results across the three metrics but no consistent advantage.Source: §B.1 · p. 19

B.2 Standard Deviation

  • Removing group standard-deviation normalization leaves RLVR aggregate Forced Accuracy nearly identical to standard RLVR, with neither variant consistently outperforming the other across benchmarks.The comparison uses the same binary reward.Source: §B.2 · p. 19, Table 4 · p. 19

B.3 Increasing the Rollout Group Size

Doubling ERT’s rollout group size does not make its behavior distinguish different confidence targets. Larger groups can expose more reward outcomes, but they do not restore target information lost during advantage estimation.

B.3 Increasing the Rollout Group Size

  • Increasing ERT’s responses per prompt from 8 to 16 approximately doubles rollout generation cost without enabling target differentiation.Source: §B.3 · pp. 19–20
  • On HotpotQA, groups containing all three reward values rapidly disappear and single-reward groups dominate later training.Source: §B.3 · p. 19
  • DAPO-Math retains more groups with all three rewards, yet abstention curves still largely overlap across confidence targets.Source: §B.3 · p. 19
  • Larger groups increase the chance of observing all three outcomes but cannot replace preserving the target signal during advantage estimation.Source: §B.3 · p. 20

B.4 Scaling RLEC to Longer Contexts

A 16K context primarily benefits difficult mathematical reasoning, with little material change to calibration. The appendix also relates the evaluation score across confidence targets to RLCR’s reward through an integral and its finite-target approximation.

B.4 Scaling RLEC to Longer Contexts

  • Training and inference context windows increase from 4K to 16K with other hyperparameters held identical to Table 2.Source: §B.4 · p. 20
  • The 16K context produces longer 30B responses and improves DAPO-Math validation reward, while HotpotQA trajectories remain largely unchanged.At p = 0.9, the 16K model abstains less than its 4K counterpart while preserving target ordering.Source: §B.4 · p. 20
  • AIME Forced Accuracy rises from 27.29% with 4K context to 38.96% with 16K context; aggregate Forced Accuracy improves slightly while AOC and Score remain broadly comparable.Results on the remaining benchmarks are mixed and generally show smaller differences.Source: §B.4 · pp. 20–21, Table 5 · p. 21

C Relation Between Score and the RLCR Reward

D Limitations and Future Work

The idealized reward objective guarantees requested accuracy among answered questions, but the trained model does not consistently meet every target in practice.

D Limitations and Future Work

  • Under idealized assumptions, the reward objective induces q(x) > p and guarantees Acc(p) ≥ p among answered questions.Source: D Limitations and Future Work · p. 22
  • Limited model capacity, imperfect correctness estimates, finite rollouts, and stochastic optimization can prevent the learned policy from realizing the idealized rule; distribution shifts can further weaken compliance.Source: D Limitations and Future Work · p. 22
  • Requested accuracy is not consistently attained on every benchmark, with the largest visible gaps on difficult knowledge tasks such as SimpleQA-Verified and AA-Omniscience, particularly for the 4B model.Source: D Limitations and Future Work · p. 22

Related papers

Loading 2610.35059v1…