Source-linked AI summary
Maximizing Confidence Alone Improves Reasoning
Mihir Prabhudesai, Lili Chen, Alex Ippoliti, Katerina Fragkiadaki, Hao Liu, Deepak Pathak
TL;DR
Existing reasoning-oriented RL relies heavily on externally engineered rewards or ground-truth answers, motivating a method that can improve models without such supervision. RENT uses the model’s negative token-distribution entropy as an intrinsic reward, and reports improved reasoning across multiple benchmarks and model families. The approach is intended for settings where external supervision is unavailable, but the authors caution that confident errors can cause catastrophic failures.
Problem
Current language-model reasoning improvements rely heavily on external supervision and correctness-based rewards, despite reward engineering being difficult and ground-truth answers being unavailable in some settings.
Method
RENT performs unsupervised reinforcement learning by using the negative entropy of the model’s predicted token distributions as a dense intrinsic reward.
Results
RENT improves reasoning performance across GSM8K, MATH500, AMC, AIME, and GPQA and across Qwen, Mistral, and Llama model families and sizes.
Takeaways & Limitations
Entropy minimization can improve language-model reasoning in regimes where external supervision is unavailable, and the reward is presented as applicable across a wide range of domains.
Takeaways & Limitations
The method can be confidently wrong, and calibration errors may cause catastrophic failures; deployment without safeguards could be dangerous.
Abstract
from arXiv · showhide
Reinforcement learning (RL) has enabled machine learning models to achieve significant advances in many fields. Most recently, RL has empowered frontier language models to solve challenging math, science, and coding problems. However, central to any RL algorithm is the reward function, and reward engineering is a notoriously difficult problem in any domain. In this paper, we propose RENT: Reinforcement Learning via Entropy Minimization -- a fully unsupervised RL method that requires no external reward or ground-truth answers, and instead uses the model's entropy of its underlying distribution as an intrinsic reward. We find that by reinforcing the chains of thought that yield high model confidence on its generated answers, the model improves its reasoning ability. In our experiments, we showcase these improvements on an extensive suite of commonly-used reasoning benchmarks, including GSM8K, MATH500, AMC, AIME, and GPQA, and models of varying sizes from the Qwen, Mistral, and Llama families. The generality of our unsupervised learning method lends itself to applicability in a wide range of domains where external supervision is unavailable.
1 Introduction
RENT replaces external correctness rewards with the model’s own confidence, measured through negative entropy, to support unsupervised reasoning improvement. The method emphasizes confidence near final answers and improves performance across diverse benchmarks and model families.
- RENT motivation and contribution: Lower entropy represents more confident token predictions, whereas higher entropy reflects greater uncertainty or more diverse predictions.
- RENT motivation and contribution: RENT uses negative entropy, or model confidence, as a reinforcement-learning reward without external rewards or ground-truth answers.The signal is computed from the model’s predicted token distributions.
- RENT motivation and contribution: Entropy minimization near the end of the reasoning chain, especially around final-answer tokens, correlates most strongly with improved accuracy.Early response tokens show little correlation with accuracy.
- RENT evaluation: RENT improves reasoning performance across GSM8K, MATH500, AMC, AIME, and GPQA, spanning Qwen, Mistral, and Llama models of varying sizes.The reported improvements are described as consistent across model families, sizes, and benchmarks.
2 Related Work
Prior work studies supervised reasoning rewards, confidence and calibration, test-time adaptation, and intrinsic-reward unsupervised RL. RENT differs by minimizing rather than maximizing entropy and by using a dense, general signal for reasoning.
- Reasoning RL and confidence: Reasoning-oriented language-model RL commonly prompts chains of thought and rewards outputs according to their agreement with ground-truth answers.
- Reasoning RL and confidence: Confidence measures estimate whether generated outputs are correct, while calibration evaluates alignment between confidence and actual correctness.
- Test-time adaptation: Test-time adaptation updates models on test-distribution data without labels, often addressing distribution shifts between training and testing environments.
- Test-time adaptation: Tent minimizes prediction entropy during test time, whereas TTRL uses majority voting as a sparser and less general reward than entropy.The passage notes that majority voting cannot be applied to long-form free-response questions.
- Unsupervised reinforcement learning: Prior unsupervised RL methods often maximize entropy for exploration or skill discovery, while RENT minimizes entropy by reinforcing high-confidence outputs for language-model reasoning.
3 Method
RENT formulates language-model reasoning as one-step unsupervised RL: generated responses receive negative-entropy rewards instead of ground-truth-based rewards. GRPO optimizes relative reward against reference policies, while token-level entropies are averaged into a response-level signal.
- RL formulation: The language model acts as a policy that generates a response from a prompt, and RENT rewards the response without using the ground-truth answer.This creates an unsupervised reinforcement-learning formulation for language-model generation.
- Group Relative Policy Optimization: GRPO compares the current policy’s reward with the average reward of a group of reference policies to improve learning stability.The comparison uses relative rather than absolute performance.
- Group Relative Policy Optimization: The GRPO objective combines expected reward under the current policy with the reference group’s average reward, making improvement relative to baselines.
- Entropy reward: For each generated token, the model produces a vocabulary distribution whose entropy measures uncertainty in the next-token prediction.
- Entropy reward: RENT averages token entropies across the response and uses their negative as the reward, encouraging confident, peaked distributions without labeled targets.Lower entropy corresponds to lower uncertainty in the response.
4 Experiments
RENT is evaluated across reasoning benchmarks, model families, reward baselines, and token-selection strategies. Results indicate that entropy minimization improves reasoning without external supervision, with confidence-oriented rewards strongest near response endings.
- Experimental Setup: RENT is evaluated independently on GSM8K, MATH500, AMC, AIME, and GPQA, using models from the Qwen, Mistral, and Llama families.Experiments cover varying model sizes and include both standard and Math-specialized Qwen models.
- Main Results: Entropy minimization improves reasoning across model families, model sizes, and benchmarks without external supervision.The authors report strong improvements even for models already proficient at instruction following.
- Is It Just Formatting?: RENT generally outperforms reinforcement learning using only a format reward, indicating gains beyond answer-format compliance.The authors note that format reward can be competitive when benchmarks are easy and formatting dominates performance.
- Main Results: Accuracy improves alongside confidence during RENT training, supporting a strong correlation between answer confidence and answer accuracy.This trend is shown for Qwen2.5-Math-7B on AMC and Qwen2.5-7B-Instruct on MATH500.
- Comparison to Concurrent Work: RENT is best on average among RENT, TTRL, Intuitor, and spurious rewards, and performs best on MATH500, AIME, and GPQA.Performance is similar to TTRL and Intuitor on most benchmarks, but RENT has a large advantage on AIME; spurious rewards are not competitive.
- Which Tokens to Minimize Entropy Over: The highest confidence–accuracy correlation occurs for later response tokens, while early-token strategies correlate less strongly with accuracy.The experiments compare last-chunk, first-chunk, last-token, random, and post-thought-process selection strategies.
5 Limitations
RENT’s unsupervised confidence-based learning is limited relative to methods using external supervision and can fail when the model is confidently wrong.
- Unsupervised learning alone cannot match methods that use ground-truth answers.
- Overconfidence and calibration errors can cause RENT to fail catastrophically without safeguards.
- Despite this risk, confidence generally correlates with accuracy, and performance improves when confidence is used alone.
6 Conclusion
The paper presents RENT, an unsupervised reinforcement learning method that uses entropy as a reward, and reports improved reasoning performance across several benchmarks.
- RENT uses entropy as a reward in unsupervised reinforcement learning.
- RENT improves language-model reasoning performance on GSM8K, MATH500, AMC, AIME, and GPQA.
- The reward function is described as general and applicable across domains where external supervision is unavailable.
A Hyperparameters
This section provides the paper’s hyperparameter reference.
- Table 5 contains the full list of hyperparameters.