Source-linked AI summary
Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Models
Pengyi Li, Matvey Skripkin, Alexander Zubrey, Andrey Kuznetsov, Ivan Oseledets
TL;DR
Existing RL methods for language-model post-training often depend on human annotations, external models, or engineered rewards. RLSC instead uses the model’s own confidence as a self-supervised objective, and achieves substantial gains across five reasoning benchmarks with only 16 completions per question and 10 or 20 training steps. The approach offers a lightweight post-training method, although its smoothing and sample-count effects remain incompletely ablated.
Problem
Existing reinforcement-learning methods often depend on costly human annotations, external models, or complex reward engineering.
Method
RLSC converts agreement in the model’s output distribution into a differentiable self-confidence objective, using a frozen model for sampling and weighting.
Results
+13.4% on AIME2024, +21.2% on MATH500, +21.7% on Minerva Math, +20.8% on Olympiadbench, and +9.7% on AMC23 after lightweight training.
Takeaways & Limitations
RLSC provides a lightweight fine-tuning method that eliminates labels, preference models, and handcrafted rewards while improving Qwen2.5-Math-7B across reasoning benchmarks.
Takeaways & Limitations
The paper defers a full ablation of smoothing terms and sample counts to future work.
Abstract
from arXiv · showhide
Large language models (LLMs) excel at reasoning, yet post-training remains critical for aligning their behavior with task goals. Existing reinforcement learning (RL) methods often depend on costly human annotations or external reward models. We propose Reinforcement Learning via Self-Confidence (RLSC), which uses the model's own confidence as reward signals-eliminating the need for labels, preference models, or reward engineering. Applied to Qwen2.5-Math-7B with only 16 samples per question and 10 or 20 training steps, RLSC improves accuracy by +13.4% on AIME2024, +21.2% on MATH500, +21.7% on Minerva Math, +20.8% on Olympiadbench, and +9.7% on AMC23. RLSC provides a simple, scalable post-training method for inference models, requiring only a small number of samples and unlabelled supervision.
1 Introduction
RLSC addresses the cost and supervision demands of existing reinforcement-learning methods by using model self-confidence as the reward signal. With lightweight training, it improves performance across multiple reasoning benchmarks.
- Motivation: Existing RL methods often require costly human-labeled data or engineered rewards, while TTRL uses 64 responses per question for majority-vote pseudo-labels.These requirements create annotation and computational overhead.
- Approach: RLSC uses the model’s own confidence in generated outputs as a reward signal, removing human labels, external models, and manual reward shaping.The method obtains feedback directly from the model’s responses.
- Results: +13.4% on AIME2024, +21.2% on MATH500, +21.7% on Minerva Math, +20.8% on Olympiadbench, and +9.7% on AMC23.The evaluation uses Qwen2.5-Math-7B, 16 samples per question, and 10 or 20 training steps.
- Implications: RLSC achieves strong performance with minimal training data and low compute cost, supporting resource-constrained use.The framework is presented as a practical training method for inference benchmarks.
2 Method
RLSC replaces external pseudo-labeling with a self-confidence objective that sharpens the model’s output distribution. It trains by weighting sampled responses according to a frozen model’s confidence, using a small completion batch and standard optimization.
- From TTRL to Mode Sharpening: TTRL selects the most frequent completion from many sampled outputs, but this pseudo-labeling process is computationally expensive.TTRL typically generates 64 outputs per input and requires separating answers from reasoning traces.
- From TTRL to Mode Sharpening: Majority voting selects the distribution mode, so optimizing agreement between samples increases probability mass on the most likely answer.RLSC replaces external pseudo-labeling with a direct internal mode-sharpening objective.
- From TTRL to Mode Sharpening: The self-confidence objective maximizes the probability that independent samples are identical, favoring concentrated and confident output distributions.The expression reaches its maximum when the distribution collapses around one response.
- Self-Confidence Loss and Gradient: The gradient is computed with a log-trick, using a frozen p_old model for sampling and weighting.The frozen copy prevents gradients from propagating through p_old.
- Self-Confidence Loss and Gradient: The loss increases log-probabilities for responses that the old model rated more confidently without requiring labels or an external reward model.The broader loss class can use additive smoothing to stabilize optimization when p_old is peaked or sparse.
- Training Procedure: The minimal setup omits auxiliary datasets, instruction tuning, preference models, and handcrafted rewards for zero-label reinforcement learning.This supports an efficient training pipeline with limited additional supervision.
3 Experiments
RLSC is evaluated across mathematical reasoning benchmarks using accuracy, with qualitative analyses showing shorter, more direct reasoning after fine-tuning. The reported results indicate broad improvements, while response-length reduction and hyperparameter effects remain incompletely quantified.
- 3.1 Results Analysis: RLSC is evaluated on mathematical reasoning benchmarks including AIME24, MATH500, AMC23, GSM8K, Minerva Math, Olympiadbench, and MMLU Stem.The benchmark table compares baseline Qwen2.5 models with RLSC-tuned variants using accuracy percentages.
- 3.1 Results Analysis: Accuracy, rather than Pass@1, is used because the authors prioritize correct answers without trial and error.Accuracy is defined as the ratio of correctly answered samples to total evaluation samples.
- 3.1 Results Analysis: RLSC produces significant improvements across the core benchmarks, including a 21.7% improvement on Minerva Math at the 7B parameter scale.The reported gains also cover AIME24, MATH500, Olympiadbench, and AMC23.
- 3.2 Emergent Behavior: Concise Reasoning without Prompting: RLSC fine-tuning yields shorter, more confident answers that identify solutions earlier and avoid redundant reasoning.The authors contrast this behavior with prompting based on the textual cue “Let’s think step by step”.
- 3.2 Emergent Behavior: Concise Reasoning without Prompting: The reduction in response length is not formally quantified, and precise characterization such as entropy or inference steps is deferred to future work.The authors state that the observed trend is consistent across benchmarks.
- 3.3 Qualitative Analysis: Qualitative comparisons report improved task comprehension and reasoning, with accurate conclusions reached through simple reasoning paths rather than lengthy derivations.The comparison includes MATH500 examples where the initial model produced incorrect or failed reasoning.
- 3.3 Qualitative Analysis: A full ablation of smoothing terms and sample counts is deferred, although initial experiments suggest stability across a wide range of hyperparameters.
4 Related Work
Prior reasoning-oriented RL methods obtain reinforcement signals from human feedback, verifiable answers, adversarial model interactions, or consensus over sampled responses. These strategies reduce some supervision burdens but retain dependencies on annotations, external models, reward engineering, or substantial sampling costs.
- Reinforcement Learning in Reasoning Tasks: Reinforcement learning has become important for improving LLM reasoning, including decomposition into intermediate steps and deep deliberation.
- Reinforcement Learning in Reasoning Tasks: RLHF uses human annotations or learned preference models to generate rewards, making it labor-intensive and costly.
- Reinforcement Learning in Reasoning Tasks: RLVR compares model outputs with reference answers to compute verifiable rewards, but still requires human-labeled question-answer pairs.
- Test-Time Training: SPC and AZR use adversarial dual-model frameworks in which a generator creates challenging reasoning steps and a critic learns to detect them.
- Test-Time Training: TTRL samples multiple responses and uses majority voting to create pseudo-labels, but typically requires 64 samples per question and incurs substantial computational overhead.
- Summary and Motivation: Across these approaches, reinforcement signals depend on combinations of human annotations, external models, or complex reward engineering.
5 Conclusion
RLSC is presented as a lightweight fine-tuning method that replaces labels, preference models, and handcrafted rewards with self-confidence. The conclusion connects this approach to mode sharpening and reports improved outputs with limited sampling and training.
- RLSC eliminates the need for labels, preference models, and handcrafted rewards in lightweight fine-tuning.
- Case Study: The function case study analyzes piecewise ranges and derives a largest value of a equal to 2 under the stated intersection condition.
- Majority voting is recast as agreement optimization, yielding a differentiable self-supervised objective for mode sharpening.
- Case Study: The model-output case study contrasts a correct RLSC answer of 13√ with an incorrect base-model answer of 10 for the distance problem.
- RLSC delivers substantial accuracy gains on Qwen2.5-Math-7B using 16 completions per question and 10 training steps without external supervision.
A Case Study
The case study presents mathematical reasoning examples and compares model outputs on AIME-style problems. It includes dynamic programming for a token game and logarithmic transformations for another problem.
- The AIME case study uses complex mathematical reasoning problems as the evaluation setting.
- Game Theory: The token-game solution defines winning and losing positions and counts losing positions through iterative dynamic programming up to n = 2024.
- Game Theory: The implementation computes the recurrence across all positions from 1 through 2024 to identify Bob’s guaranteed wins.
- Logarithms: A logarithmic-expression solution adds three transformed equations to derive a + b + c = −13/12 before solving each variable.
- Logarithms: The worked calculation combines the solved fractions to obtain −25/8 for the resulting logarithmic expression.