Source-linked AI summary
Learning from Failures in Multi-Attempt Reinforcement Learning
Stephen Chung, Wenyu Du, Jie Fu
TL;DR
Single-turn RL tasks may provide sparse rewards and do not require models to learn to respond to feedback. The paper trains LLMs on a multi-attempt question-answer task that retries after incorrect responses with feedback. On math benchmarks, multi-attempt-trained models improve more with additional evaluation attempts than single-turn-trained models.
Problem
Single-turn RL question-answering may provide sparse rewards and does not require LLMs to learn how to respond to user feedback.
Source: §1 · p. 1Method
The paper extends question answering to multiple attempts, giving feedback after incorrect responses and continuing until correctness or attempt exhaustion.
Source: §1 · p. 1Results
45.6% to 52.5% for multi-attempt training versus 42.3% to 43.2% for single-turn training as math-benchmark evaluation attempts increase from one to two.
Source: §1 · p. 2Takeaways & Limitations
Multi-attempt training modestly improves base math-benchmark performance and significantly enhances correction of mistakes based on user feedback.
Source: §5 · p. 6
Abstract
from arXiv · showhide
Recent advancements in reinforcement learning (RL) for large language models (LLMs), exemplified by DeepSeek R1, have shown that even a simple question-answering task can substantially improve an LLM's reasoning capabilities. In this work, we extend this approach by modifying the task into a multi-attempt setting. Instead of generating a single response per question, the model is given multiple attempts, with feedback provided after incorrect responses. The multi-attempt task encourages the model to refine its previous attempts and improve search efficiency. Experimental results show that even a small LLM trained on a multi-attempt task achieves significantly higher accuracy when evaluated with more attempts, improving from 45.6% with 1 attempt to 52.5% with 2 attempts on the math benchmark. In contrast, the same LLM trained on a standard single-turn task exhibits only a marginal improvement, increasing from 42.3% to 43.2% when given more attempts during evaluation. The results indicate that, compared to the standard single-turn task, an LLM trained on a multi-attempt task achieves slightly better performance on math benchmarks while also learning to refine its responses more effectively based on user feedback. Full code is available at https://github.com/DualityRL/multi-attempt
1 Introduction
The paper extends single-turn RL question answering to multiple attempts with feedback, targeting sparse rewards and the lack of training for responding to user feedback. A small multi-attempt-trained model improves more when allowed additional attempts than a single-turn-trained baseline.
1 Introduction
- Single-turn RL tasks reward one response and may provide sparse rewards, while not requiring models to learn to respond to user feedback.The paper proposes multi-turn RL as an alternative.Source: §1 · p. 1
- The multi-attempt task retries after an incorrect response with feedback, ending when the answer is correct or attempts are exhausted.The number of remaining attempts is decremented after an incorrect response.Source: §1 · p. 1
- The training pipeline applies standard RL to a math problem dataset and uses PPO, broadly following DeepSeek R1 Zero.The supplied passage describes the pipeline as simple.Source: §1 · pp. 1–2
- Multi-attempt training is intended to teach feedback response and self-refinement, including correcting failed attempts rather than repeating them.The authors suggest explicit training for self-refinement may help the Aha Moment emerge more easily.Source: §1 · p. 2
- 45.6% to 52.5% on math benchmarks for multi-attempt training, versus 42.3% to 43.2% for single-turn training, when evaluation attempts increase from one to two.Figure 1 averages evaluation accuracy across five benchmarks.Source: §1 · p. 2, Figure 1 · p. 2
2 Approach
The approach extends single-turn question answering into a multi-turn RL task in which a model can make further attempts after an incorrect response. It samples the number of attempts, conditions responses on the dialogue history, and optimizes rewards across turns.
2 Approach
- The multi-turn objective lets the dialogue continue after an initial response, with each response conditioned on previous prompts and responses.A transition function determines the next prompt or whether to terminate the episode.Source: §2 · p. 4
- The dialogue ends when the answer is correct or attempts are exhausted; otherwise, the model is prompted to try again.The proposed task uses a specific transition function to manage the attempts.Source: §2 · p. 4
- For each question, the experiment uniformly samples the remaining-attempt count from 1 to a maximum M, such as 5.This lets the model learn to respond carefully when only one attempt remains and explore more freely when more attempts are available.Source: §2 · p. 4
- Rewards are +1 for a correct answer in any attempt, −0.5 for an incorrectly answered response in the correct format, and −1 otherwise.Rewards are not discounted according to attempts used, so early exploration incurs no attempt-based penalty; the objective can be optimized with standard RL such as PPO.Source: §2 · p. 4
3 Related Work
The related work situates this approach among model-based planning, multi-turn RL, external-feedback self-correction, and methods that use historical logs. The paper distinguishes its use of self-generated failed attempts and a general RL environment from these approaches.
Learning to Plan in RL
- Unlike Thinker [5], which uses a learned world model to explore alternatives, this approach augments the environment to allow multiple attempts.The authors describe Thinker as a model-based RL algorithm.Source: §3 · pp. 4–5
Multi-turn Tasks for LLMs
- Unlike SCoRe [6], this method uses ground-truth reward to verify prior attempts; other multi-turn studies [7] [8] [9] [10] focus mainly on RL algorithms, while this work uses standard PPO.The authors say SCoRe requires careful calibration and two-stage training to prevent policy collapse.Source: §3 · p. 5
Self-Correction and Historical Logs
- Prior self-correction work uses reflection or external tools [11] [12] [13] [14] [15], whereas this work extends DeepSeek R1's single-turn QA task into a multi-attempt RL setting [2].Historical-log methods use past relevant logs; this approach instead uses the model's own failed attempts as examples for subsequent attempts [16–18].Source: §3 · p. 5
4 Experiments
Multi-attempt training improves refinement across evaluation attempts, while its standard single-attempt performance is only modestly better than the single-turn baseline. The experiments use a 1.5B model trained on math questions and compare both training reward and benchmark accuracy.
4 Experiments
- The experiment fine-tunes Qwen 2.5 Math 1.5B on 8K math questions, sampling 1–5 attempts per question in the multi-attempt setting.Training runs for 160 episodes, totaling 1.28M samples; the baseline uses standard single-turn rewards.Source: §4 · p. 5
- The multi-attempt model collects more training reward than the baseline, as the multi-attempt setting is inherently easier to solve.Figure 4a plots training reward against training steps.Source: §4 · p. 5, Figure 4 · p. 5
- The multi-attempt model slightly outperforms the baseline in average accuracy across five benchmarks under standard single-attempt evaluation.Figure 4b tracks evaluation accuracy across training steps; the benchmarks are AIME 2024, MATH 500, AMC 2023, Minerva Math, and OlympiadBench.Source: Figure 4 · p. 5, §4 · p. 6
- 45.58% to 53.82%: multi-attempt training improves math-benchmark accuracy as allowed attempts increase from 1 to 4, while the baseline gains only marginally.The comparison assesses refinement based on previous failed attempts.Source: §4 · p. 6
5 Conclusion
The conclusion reports modest gains in base math-benchmark performance and stronger mistake correction from feedback. It points to more nuanced feedback and auxiliary tasks as directions for future exploration.
5 Conclusion
- Multi-attempt training modestly improves base performance on math benchmarks while significantly enhancing correction based on user feedback.The paper proposes extending the question-answering task used in DeepSeek R1 with a multi-attempt mechanism.Source: §5 · p. 6
- The authors identify more nuanced feedback and auxiliary tasks as possible directions for fostering different LLM capabilities.These are presented as future exploration, not established outcomes.Source: §5 · p. 6
- The work extends DeepSeek R1's question-answering task with a multi-attempt mechanism.Source: §5 · p. 6