PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Language Models to Fail Gracefully

Daman Arora, Xuecheng Liu, Himanshu Gaurav Singh, Fahim Tajwar, Abitha Thankaraj, Ruslan Salakhutdinov, Haiwen Feng, Yuda Song, Andrea Zanette

arXiv:2610.20401v1cs.LG

TL;DR

Models can be rewarded for guessing when they cannot reliably determine an appropriate action, leaving users to verify outcomes. GracefulRL reshapes reinforcement-learning optimization to support selective fallback without fallback labels or additional inference-time machinery. Across studied tasks, it learns favorable success–error tradeoffs while preserving task performance.

  • Problem

    When models cannot reliably determine appropriate actions, reinforcement learning can incentivize guessing, shifting verification to users and allowing unchecked mistakes to propagate.

    Source: §1 · p. 1

  • Method

    GracefulRL reshapes the optimization landscape to encourage fallback over error while maintaining pressure to solve tasks, learning fallback from interaction without fallback labels or inference-time decision machinery.

    Source: §1 · p. 2

  • Results

    GracefulRL learns selective fallback while preserving strong task performance across navigation, mathematical reasoning, and Minesweeper; on AIME, error falls from 79.1% to 8.9% with a 3.1 percentage-point success reduction versus standard RL.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across the studied settings, fallback behavior can be learned directly as part of the policy while achieving favorable tradeoffs between task success and error.

    Source: §7 · p. 11

    GracefulRL is designed for online RL; off-policy training, coding agents, and long-horizon agentic training were not studied, and suitable fallback behavior may be difficult to specify across domains.

    Source: §7 · p. 11

Abstract

from arXiv · show

Reinforcement learning makes language models better at solving tasks, but the same mechanism also rewards guessing. When success is the only way to earn reward, opting for a safe fallback is always worse than a long-shot attempt. When a wrong action can corrupt a database or ship broken code, that incentive can be costly. Therefore, we propose training models to fail gracefully: to preserve successful behavior whenever possible while learning to execute a safe fallback policy when acting would be unreliable. However, naively using policy gradients with intermediate reward for fallback behavior is brittle due to unfavorable optimization dynamics. We address this challenge with GracefulRL, which reshapes the optimization landscape to promote graceful failure. Importantly, this behavior emerges directly from outcome-level reward signals, without annotations specifying when to fall back or additional inference-time machinery such as a verifier. Across a range of tasks and scales, we train models that largely preserve their task-solving capability while selectively falling back when acting would be unreliable. More broadly, our results show that reinforcement learning can not only incentivize models to succeed, but also to reason about the limits of their capabilities and change course when they determine that success is out of reach.

1 Introduction

Because models can be unable to determine reliable actions, reinforcement learning may reward guessing and leave users to verify outcomes. The paper proposes GracefulRL to preserve task-solving while learning selective fallback, and reports results across several settings.

1 Introduction

  • Standard RL rewards successful completion, but accuracy and success rate reveal little about behavior when models cannot reliably determine appropriate actions.Guessing can shift verification work to users, and unchecked mistakes can propagate into consequential actions.Source: §1 · p. 1
  • Graceful failure means changing course when success is unlikely, such as reverting changes, telling the user the model is stuck, or handing off the task.The paper says this can avoid harmful actions and help build user trust.Source: §1 · p. 2
  • Assigning intermediate reward to fallback can make standard policy-gradient training brittle, including collapse to always falling back and ceasing task solving.The paper attributes this to optimization dynamics that fail to reliably produce the behavior specified by the objective.Source: §1 · p. 2
  • GracefulRL reshapes the optimization landscape to maintain pressure to solve tasks while encouraging fallback over errors when success is unlikely.Fallback timing is learned from interaction without labels identifying hard problems, a separate uncertainty model, or an inference-time decision rule.Source: §1 · p. 2
  • GracefulRL learns selective fallback while preserving strong task performance across navigation, mathematical reasoning, and multi-turn Minesweeper.On AIME, error falls from 79.1% to 8.9% with only a 3.1 percentage-point success reduction versus standard RL.Source: §1 · p. 2

2 Problem Definition and Setup

The setup models task outcomes as success, error, or fallback, with fallback between success and error in preference. It defines rates for evaluating these outcomes and compares methods by success and error.

2 Problem Definition and Setup

  • Standard RLVR maximizes average task success probability over the task distribution, using a verifier to classify generated sequences as success or error.The policy generates an action sequence for each task; the verifier/reward function assigns the outcome.Source: §2 · p. 3
  • GracefulRL adds fallback to the outcome space and orders outcomes as Success ≻ Fallback ≻ Error, with fallback semantics determined by the environment.Examples include abstaining on reasoning problems, restoring a database, or handing a task to another model or sub-agent.Source: §2 · p. 3
  • Fallback@1 is one minus Success@1 and Error@1, and method A Pareto-dominates B when it has at least as high success and at most as high error, with one strict improvement.Source: §2 · p. 3

3 The Challenge of Optimizing Fallback Behavior

Directly maximizing expected reward with an intermediate fallback reward can be sensitive to initialization and reward magnitude. Policy gradients can vanish near saturated policies, making recovery from a suboptimal outcome difficult.

3 The Challenge of Optimizing Fallback Behavior

  • Expected-reward policy gradients with an intermediate fallback reward can be highly sensitive to initialization and reward magnitude, yielding either fallback-only behavior or no learned fallback.The approach may require tuning for each model and task combination.Source: §3 · p. 3
  • As a policy approaches saturation on any outcome, the standard policy-gradient norm is O(1 − p_o(x)), so updates toward better outcomes can become vanishingly small.The result holds under the stated regularity conditions; for softmax policies, including autoregressive language models, the paper gives the interpretation under a mild boundedness condition on logit gradients.Source: §3 · pp. 3–4

4 Optimizing for Graceful Failure

GracefulRL replaces direct expected-reward optimization with a log-probability objective that preserves a recovery signal near suboptimal saturated policies. Its algorithm applies this objective to the ternary reward, with smoothing to moderate rare-outcome weights.

4 Optimizing for Graceful Failure

  • The paper motivates a surrogate objective because assigning fallback an intermediate reward specifies the desired behavior but does not ensure that it is easy to optimize.Source: §4 · p. 4
  • GracefulRL combines weighted log-probabilities of success and non-error outcomes, creating barriers against near-certain error and near-zero success while preferring success over fallback over error.The weight c interpolates between maximizing success alone and avoiding errors while treating success and fallback equivalently.Source: §4 · p. 4
  • Near saturation on fallback or error, GracefulRL retains a gradient bounded away from zero, while standard policy gradient can become trapped near those outcomes.When success is initially unlikely, GracefulRL first shifts probability from error to fallback, then toward success.Source: §4 · p. 4, Figure 2 · p. 5

4.1 The GracefulRL algorithm

  • The GracefulRL objective is TailRL specialized to ternary rewards, and the algorithm adds smoothing constant ϵ to shrink large inverse-frequency weights for rare success or non-error trajectories.This may reduce variance at the cost of some bias.Source: §4.1 · p. 5

5 Experiments

The experiments test GracefulRL in navigation, mathematical reasoning, and sequential Minesweeper decisions, examining whether it can preserve task performance while learning to fall back safely.

5.1.1 Results on SafeMaze

  • GRPO and RLOO can collapse to always falling back, while GracefulRL Pareto-dominates reward-tuning and DPG with higher Success@ 1 and lower Error@ 1.Standard policy-gradient behavior is sensitive to fallback reward: RLOO collapses for every tested c ≥ 0.3, while more rollouts particularly help at higher c.Source: Figure 3 · p. 6, §5.1.1 · pp. 6–7

5.1 SafeMaze: A Controlled Testbed

  • SafeMaze makes reaching the Goal harder than taking the always-clear Safe fallback, while wall hits and timeouts count as errors.The Goal must be reached within 50 steps; experiments use a 4M-parameter residual CNN policy.Source: §5.1 · p. 6

5.2.1 Results on Language-Model RL

  • GracefulRL Pareto-dominates RLCR+Thresholding across success-error frontiers; on AIME, errors fall from ∼79% to ∼9% with about 3% points of success sacrificed.The model verbalizes uncertainty and fallback; on AbstentionBench, abstention F1 rises from 0.472 at c = 0.2 to 0.677 at c = 0.8, versus 0.590 for Qwen3-4B-Instruct-2507.Source: Figure 4 · p. 7, §5.2.1 · pp. 8–9, Figure 5 · p. 8

5.2 Language-Model RL for Mathematical Reasoning

  • In mathematical reasoning, the model must solve problems within its capabilities and abstain when continued reasoning is unlikely to succeed.GracefulRL jointly trains one model across five fallback utilities, using conditioning suffixes to request different degrees of caution.Source: §5.2 · p. 7

5.3 Multi-Turn RL in Minesweeper

  • Minesweeper tests repeated decisions with irreversible moves, requiring the agent to decide at each turn whether available evidence justifies further action.The agent is trained on 8-turn puzzles and evaluated on 12-turn puzzles; a fallback avoids mines but may leave the puzzle unfinished.Source: §5.3 · p. 9

5.3.1 Results on Minesweeper

  • At c = 0.5, GracefulRL reaches 77.4% success and 6.0% error in Minesweeper, improving both metrics over MaxRL.MaxRL errors increase late in the game, while GracefulRL keeps error probability lower near the end; RLOO with c = 0.5 collapses to neither successes nor errors.Source: §5.3.1 · pp. 9–10, Figure 7 · p. 10

6 Related Work

Related work spans policy-gradient optimization, likelihood-based RL objectives, uncertainty estimation, abstention training, and safe reinforcement learning. The paper distinguishes GracefulRL as learning how a policy should act under uncertainty, including in sequential settings.

Policy-gradient optimization challenges

  • The paper relates its analysis to known policy-gradient vanishing-gradient results and says GracefulRL retains a non-vanishing optimization signal near saturated outcomes.Razin et al. (2024) studied small gradients when reward variance is small; this work focuses on three-outcome saturation.Source: §6 · p. 10

Likelihood-based objectives for RL

Uncertainty quantification for LLMs

  • Uncertainty-quantification methods estimate confidence or use inference-time stopping, while this work learns the fallback action within the policy for sequential decision-making.The authors also state that their approach does not require specialized datasets, is not LLM-specific, and extends to multi-step interactions.Source: §6 · pp. 10–11

Learning abstention with RL

Safe and constrained reinforcement learning

  • Unlike Recovery RL, which uses a separate recovery policy near constraint-violating regions, this work jointly learns task success and graceful fallback without an intervening recovery policy.The passage situates this contrast within safe and constrained reinforcement learning.Source: §6 · p. 11

7 Conclusion and Limitations

The paper concludes that GracefulRL learns favorable task-success and error tradeoffs by incorporating fallback into the policy. It also identifies unstudied training settings and unresolved questions about defining and generalizing graceful failure.

7 Conclusion and Limitations

  • Across the studied settings, GracefulRL learns favorable tradeoffs between task success and error, with fallback learned directly as part of the policy.Source: §7 · p. 11
  • The method is designed for online RL; its behavior under off-policy training has not been studied, and compute constraints excluded coding agents and long-horizon agentic training.Source: §7 · p. 11
  • Fallback behavior may differ substantially across domains; defining it, deriving scaling laws, and evaluating generalization to unseen tasks remain open directions.Source: §7 · p. 11

A Proof of the Results in Section 3

The appendix formalizes why ordinary policy gradients can stall near saturated policies and why the log-barrier objective retains a signal near suboptimal saturation, under stated regularity conditions.

A Proof of the Results in Section 3

  • The appendix’s proofs establish gradient suppression for the expected-reward objective and a non-vanishing log-barrier gradient under non-degeneracy.The log-barrier result concerns suboptimal saturation; the proofs are at the population-gradient level, not stochastic estimators.Source: A Proof of the Results in Section 3 · p. 17

A.1 Setup

  • The setup relates each outcome probability’s gradient to its conditional score and treats success, fallback, and error as trajectory outcomes.The proof roadmap first fixes this terminology and establishes identities connecting outcome probabilities to conditional scores.Source: §A.1 · pp. 17–18, A Proof of the Results in Section 3 · p. 17

A.2 Bounded Conditional Scores

  • A bounded logit-gradient assumption yields bounded conditional score functions for softmax policies.For autoregressive language models, the actions are vocabulary tokens and states include the prompt and preceding tokens.Source: §A.2 · p. 18, §A.1 · p. 18

A.3 Policy Gradient Vanishes Near Saturation

  • Under bounded conditional scores, the policy-gradient norm is at most G(1 − po) for any outcome o, so it vanishes as that outcome saturates.This establishes gradient suppression near saturation for success, fallback, and error.Source: §A.3 · pp. 18–19

A.4 Gradient of the log-barrier objective does not vanish

  • The log-barrier gradient remains bounded above and away from zero near fallback- or error-saturated policies, assuming non-degeneracy.The non-degeneracy condition supplies the lower bound; it is not needed for the policy-gradient upper bound.Source: §A.4 · pp. 19–21

B Effects of the ϵ parameter

The appendix analyzes smoothing as a variance–bias tradeoff, while the SafeMaze experiments show that fixed fallback rewards can produce sharply different—and often degenerate—outcomes.

B Effects of the ϵ parameter

  • Smoothing introduces only O(ϵ) bias in the expected gradient; under the stated variance condition, sufficiently small positive ϵ lowers variance and MSE.The squared bias is O(ϵ^2), so the linear variance decrease dominates for sufficiently small ϵ.Source: B Effects of the ϵ parameter · p. 23
  • A small positive ϵ can lower estimator variance enough to reduce MSE, and empirically improves performance under smaller compute budgets.At fixed rollout count, small ϵ keeps the expected update close to the unsmoothed target; larger ϵ reduces variance while adding bias.Source: B Effects of the ϵ parameter · pp. 22–23

C.1 SafeMaze details

  • SafeMaze uses a 15 × 15 grid with policies pretrained on goal-seeking and safe-square-seeking trajectories, with the goal reachable within the time limit.Source: §C.1 · p. 24

C.2 RLOO with a Fixed Fallback Reward

  • With fixed fallback reward, c = 0.1 drives fallback probability toward zero, c = 0.2 leaves fallback dominant with success plateauing, and c ≥0.3 converges to fallback only.The authors motivate online reward tuning because useful behavior occurs only in a narrow reward range.Source: §C.2 · p. 24, Figure 8 · p. 24

C.3 Hyperparameters for the Main SafeMaze Comparison

  • For the main SafeMaze comparison, the runs use 32 environments, 64 rollouts, and 20,000 optimization steps, with fallback rewards swept across five values.The online reward-tuning baseline sweeps α values in ct = αbpS,t.Source: §C.3 · p. 24

C.4 Varying Rollout Counts and Safe Rewards

These experiments examine rollout count, smoothing, and entropy tuning, then describe the language-model training and validation setup. Results show that more rollouts improve SafeMaze reliability, while entropy bonuses do not prevent collapse in the tested regime.

C.4 Varying Rollout Counts and Safe Rewards

  • Increasing rollout count improves the SafeMaze reliability frontier, and at c = 0.8 only 64 rollouts retain nonzero success while 16 and 32 collapse to fallback only.Larger rollout counts are more likely to include rare successes, yielding a more informative update.Source: §C.4 · p. 24, Figure 9 · p. 25

C.5 Smoothing-Parameter ϵ effects

  • Larger smoothing improves reward earlier in SafeMaze training, whereas smaller values become preferable with more compute.The figure’s comparison uses 32 rollouts and c = 0.4.Source: §C.5 · p. 25, Figure 9 · p. 25

C.6 Entropy-Bonus Tuning

  • Every tested entropy setting drives pS to zero; GRPO and RLOO coefficients up to 0.3 collapse to fallback, while RLOO coefficient 1 collapses to error.The sweep fixes c = 0.4; the authors attribute the failure to entropy regularization still suffering vanishing gradients near saturation.Source: §C.6 · p. 25, Figure 10 · p. 25

D.1 Training Hyperparameters

  • The language-model runs use matched shared optimization and sampling hyperparameters; GracefulRL uses Algorithm 1 with ϵ = 1 and five conditioning levels.RLCR uses RLOO with the calibration reward; Tables 1 and 2 contain the configurations.Source: §D.1 · p. 26, Table 1 · p. 26, Table 2 · p. 27

D.1.1 RLCR Training Prompt

  • Qwen3 models learn the RLCR confidence instruction after a couple of training steps, without extra SFT warm-start data.Source: §D.1.1 · p. 27

D.1.2 GracefulRL Conditioning Suffixes

  • GracefulRL appends one of five conditioning suffixes to each math problem, mixes the variants during training, and uses the corresponding fallback utility in its reward.Table 3 lists the suffixes.Source: §D.1.2 · p. 27, Table 3 · p. 28

D.2 Validation and Checkpoint Selection

  • Checkpoints are selected on MATH500 validation reward before test evaluation; for 1.7B, GracefulRL selects step 1,500 and RLCR step 600, while both 4B methods select step 800.The conditioned model’s validation reward averages across its five conditioning levels; RLCR uses mean Brier reward.Source: §D.2 · p. 28, Figure 11 · p. 28

D.3 Validation on All Datasets

The language-model analyses break out dataset frontiers, smoothing-budget effects, RLCR’s accuracy–calibration tradeoff, and conditioned training’s success and error dynamics. The reported dynamics show lower errors at higher fallback rewards alongside modest success reductions.

D.3 Validation on All Datasets

  • Figures 12 and 13 expand the aggregate comparison into separate reliability frontiers for AIME, AMC, JEEBench, and MATH500 at the 1.7B and 4B scales.Their final panels report aggregate frontiers across datasets.Source: §D.3 · p. 28, Figure 12 · p. 29, Figure 13 · p. 29

D.4 Smoothing with a Limited Language-Model RL Budget

  • With Qwen3-1.7B-Base at c = 0.4, ϵ = 1 achieves higher training reward than ϵ = 0 throughout the first 400 steps.The comparison highlights smoothing under a limited language-model RL compute budget.Source: §D.4 · p. 30, Figure 14 · p. 30

D.5 RLCR Can Trade Accuracy for Calibration

  • RLCR’s Brier score continues improving after accuracy peaks around step 400, as accuracy then falls; on DAPO-17k, the 1.7B model generates only confidence values 0, 0.1, and 0.2.The authors hypothesize calibration optimization may trade off task-solving ability and report that the confidence-output issue is absent in the 4B model.Source: §D.5 · p. 30, Figure 15 · p. 30

D.6 Success and Error Dynamics on AIME

  • Across five conditioning values on combined AIME, success generally improves during training, while higher fallback rewards substantially lower errors with modestly reduced success versus c = 0.Figure 16 plots success and error percentages across the conditioning levels.Source: §D.6 · p. 31, Figure 16 · p. 31
  • The AIME compute sweep evaluates the 4B model after 200, 400, and 800 training steps, with each frontier marker corresponding to a different fallback reward.Dashed curves fit the resulting frontiers.Source: Figure 17 · p. 31

D.7 Reliability Frontiers with More Training Compute

This section examines reliability frontiers with more training compute, online reward tuning and fallback behavior, and examples of how the trained model expresses uncertainty. It also describes the AbstentionBench evaluation protocol and its limitation in distinguishing answerable prompts beyond the model’s reach.

D.8 Online Reward Tuning Can Lose Fallback Behavior

  • Fallback fell below 0.5% around step 260 for Qwen3-1.7B-Base and around step 850 for Qwen3-4B-Base; the latter ended below 0.1% fallback with about 57% failure.The online reward-tuning baseline adapts fallback reward to estimated on-policy success, yet its fallback behavior is unstable and the authors deem it ill-suited to learning fallback here.Source: Figure 18 · p. 32, §D.8 · p. 32

D.7 Reliability Frontiers with More Training Compute

  • With 1×, 2×, and 4× training compute, the Qwen3-4B-Base frontier moves upward, yielding higher Success@ 1 at a given Error@ 1.The frontiers compare runs after 200, 400, and 800 optimization steps across the same five fallback rewards.Source: §D.7 · pp. 31–32

D.9 Abstention Reasons

  • The appendix presents post-training chains of thought and highlights notable verbalizations as examples of abstention reasons.The supplied passages introduce examples but do not state a general finding about the reasons.Source: §D.9 · p. 32

Algebra / computation exploded

  • On a cell-loop partition problem, the model says the placement constraints make counting nontrivial and ultimately responds “I don’t know.”Its intermediate size-counting argument does not account for whether loops fit without overlap.Source: Algebra / computation exploded · pp. 33–34

Never found a workable approach

  • For the hexagon problem, the model cannot establish a relationship between the hexagon side and the triangle sides, expresses uncertainty, and responds “I don’t know.”It considers similarity but does not derive a solution.Source: Never found a workable approach · p. 35

Intermediate result fails a sanity check

  • On the urn-probability problem, the model’s derivation reaches a contradiction with integer constraints, so it concludes it is not confident and responds “I don’t know.”The passage presents this as an unresolved derivation, not a verified proof that no solutions exist.Source: Intermediate result fails a sanity check · p. 36

Cites time / effort already spent

  • For the coin-change problem, the model calls manual enumeration too time-consuming, cannot establish an exact count, and ends with “I don’t know.”It suggests possible failure patterns but says they are unverified and may not be exhaustive.Source: Cites time / effort already spent · p. 37

D.10 Evaluation on AbstentionBench

  • AbstentionBench covers 20 datasets across varied domains and six types of unanswerability, pairing unanswerable questions with answerable ones to prevent indiscriminate abstention from scoring well.The figure caption describes per-dataset Abstention F1 comparisons, but the supplied passages do not provide the results.Source: §D.10 · p. 38, Figure 19 · p. 38
  • Evaluation uses 8,220 prompts per condition, including 3,678 unanswerable prompts, after stratified sampling of at most 500 examples per dataset.Official 3,500-example subsamples are applied first for BBQ, SQuAD 2.0, and ALCUNA; five smaller datasets are evaluated in full.Source: §D.10 · p. 38
  • GracefulRL is evaluated across four risk-cue values, c ∈{0.2, 0.4, 0.6, 0.8}, with one generation per prompt and 32,880 generations total.Decoding uses temperature 0.6, top-p 0.95, a 4,096-token generation budget, and a 32,768-token context.Source: §D.10 · p. 38
  • GracefulRL abstentions are detected by exact match to its literal “I don’t know” signal, while Qwen3-4B-Instruct-2507 uses the authors’ evaluation pipeline.Exact-match detection avoids the LLM-as-a-judge used in the original evaluation.Source: §D.10 · pp. 38–39
  • AbstentionBench’s answerable/unanswerable labels do not distinguish hard answerable prompts beyond the model’s reach, which GracefulRL also treats as abstentions.The evaluation nevertheless uses F1 as required by the original dataset.Source: §D.10 · p. 39

E Minesweeper Environment and Experimental Protocol

The Minesweeper protocol defines a 6 × 6 game, how episodes end, and how puzzles are generated and presented to the model. It also reports shared training settings for four c-sweep runs.

E.3 On-policy training and checkpoint selection

  • Table 4 lists shared training settings for the four Minesweeper c-sweep runs, and training uses the Miles codebase with its listed hyperparameters.Source: Table 4 · p. 40, §E.3 · p. 40

E.1 Game, observations, and outcomes

  • A game succeeds when all 30 safe cells are revealed; a mine, malformed action, or out-of-bounds action is an error, while reaching the turn limit is fallback.Repeatedly revealing an already visible cell uses a turn without changing the board.Source: §E.1 · p. 40

E.2 Puzzle generation and data splits

  • Puzzle generation uses random seeds and selects boards solvable by a heuristic within budget but not by a uniform safe-cell policy, yielding 9,500 training and 500 testing puzzles.A deterministic constraint solver must finish each board without guessing; the turn bounds are [6, 8] for 8-turn environments and [6, 12] for 12-turn environments.Source: §E.2 · p. 40

E.4 Model prompt and interaction format

  • Episodes use the model’s native chat template, with system and user messages and no example moves.Source: §E.4 · p. 41
  • The model receives the current board and has at most 8 turns to reveal every safe cell, with a 4,096-token limit per turn and a final-line reveal(row, col) action.Replies without a valid final-line action or with an out-of-bounds cell lose the episode; revealing an already visible cell consumes a turn without losing it.Source: §E.4 · p. 41

Related papers

Loading 2610.20401v1…