Source-linked AI summary
Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization
Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, Yatao Bian
TL;DR
Existing reasoning-training methods depend on supervised reasoning data or other external supervision, motivating a fully unsupervised alternative. EMPO minimizes semantic entropy over unlabeled questions and achieves strong empirical performance, while primarily improving access to pre-trained reasoning patterns.
Problem
Reasoning LLM training typically depends on large-scale datasets containing labeled traces, verified answers, or pre-trained reward models, making expert supervision time-consuming and costly.
Method
EMPO uses semantic entropy minimization as an unsupervised objective, clustering sampled outputs by meaning and incentivizing higher-probability meaning clusters.
Results
EMPO's strong empirical performance as a fully unsupervised method can match or slightly outperform supervised counterparts such as GRPO in the 7B model.
Takeaways & Limitations
EMPO offers a scalable, cost-effective approach for improving the sampling efficiency and reliability of pre-trained reasoning abilities when curated supervisory data is scarce or expensive.
Takeaways & Limitations
Unsupervised entropy optimization may permit reward hacking, including confidently wrong predictions concentrated in frequent semantic clusters.
Abstract
from arXiv · showhide
Existing methods to enhance the reasoning capability of large language models predominantly rely on supervised fine-tuning (SFT) followed by reinforcement learning (RL) on reasoning-specific data. These approaches critically depend on external supervisions--such as labeled reasoning traces, verified golden answers, or pre-trained reward models. In this work, we propose Entropy Minimized Policy Optimization (\ours), which makes an early attempt at fully unsupervised LLM reasoning incentivization. By continuously minimizing the predictive entropy of LLMs on unlabeled questions in a latent semantic space, \ours achieves competitive performance compared to supervised counterparts on both mathematical and free-form natural reasoning tasks. Specifically, without any supervised signals, \ours boosts the accuracy of Qwen2.5-Math-7B Base from 30.7\% to 48.1\% on mathematical benchmarks and improves the accuracy of Qwen2.5-7B Base from 32.1\% to 50.1\% on MMLU-Pro. Primary experiments and analysis are also provided to interpret the effectiveness of \ours. Code is available at https://github.com/QingyangZhang/EMPO.
1 Introduction
The paper targets the supervision dependence of reasoning-model training by introducing EMPO, a fully unsupervised objective based on semantic entropy minimization. Experiments and analysis examine its performance across mathematical and free-form reasoning tasks and its role in refining pre-existing reasoning pathways.
- Reasoning-model training commonly combines SFT and RL with human-labeled traces, verified answers, or pretrained reward models.
- EMPO minimizes predictive entropy in a latent semantic space to incentivize reasoning without external supervision.It samples responses, groups semantically equivalent outputs, and optimizes toward higher-probability meaning clusters.
- Semantic entropy is proposed as an intrinsic reward whose strong negative correlation with accuracy supports its use as an unsupervised optimization objective.
- EMPO is evaluated on mathematical tasks with deterministic golden answers and free-form natural reasoning tasks.The experiments are intended to assess both efficacy and versatility without using those golden answers as training supervision.
- The analysis suggests EMPO improves reasoning by consistently selecting strong pre-existing reasoning pathways rather than instilling fundamentally new reasoning skills.This interpretation frames the method as eliciting and refining latent capabilities in base models.
2 Related Work
Prior self-supervised, self-rewarding, and entropy-minimization approaches reduce some reliance on labels but retain supervision or test-time adaptations. EMPO extends semantic entropy minimization to fully unsupervised RL for reasoning while maintaining training–testing separation.
- Self-Supervised and Semi-Supervised Reasoning: Self-improvement methods generate pseudo-labels or synthetic data, but often retain supervised fine-tuning and can risk limited gains or model collapse.
- Entropy Minimization and Semantic Consistency: EMPO applies semantic entropy minimization to LLM reasoning as a fully unsupervised RL approach, contrasting with methods that retain supervised fine-tuning or adapt on test prompts.
- Self-Rewarding and RL-based Reasoning: Self-rewarding RL methods reduce human-feedback requirements but may still use an LLM judge or a ground-truth verifier.
- Entropy Minimization and Semantic Consistency: Entropy minimization encourages confident and consistent predictions on unlabeled data and has been used for semi-supervised learning and test-time adaptation.
3 Method
EMPO incentivizes reasoning without external correctness supervision by minimizing semantic entropy over outputs to unlabeled questions. It samples and clusters responses by meaning, then rewards outputs in higher-probability clusters while filtering extreme-entropy prompts to reduce reward-hacking risks.
- EMPO uses an RL-based objective to minimize LLM generation entropy in a latent semantic space without external supervised information.
- The fully unsupervised setting uses only unlabeled user reasoning problems, without golden answers for verifying prediction correctness.The method does not require labeled question-answer pairs or valid reasoning trajectories, although a small language model may assist clustering without providing correctness supervision.
- Semantic Entropy Minimization Objective: Semantic entropy measures uncertainty over meaning clusters rather than output format, encouraging semantically consistent and potentially more reliable answers.Outputs are clustered when they share the same meaning, including through predefined rules or a small language model.
- Entropy-Minimized Policy Optimization: EMPO samples multiple outputs, merges them into meaning clusters, and approximately minimizes semantic entropy by favoring higher-probability clusters.The method assigns each output the likelihood of its meaning cluster as its reward.
- Entropy-Minimized Policy Optimization: EMPO replaces externally supervised rewards with meaning-cluster likelihoods and normalizes these rewards into output advantages for policy optimization.Higher-probability meanings receive higher advantages and are therefore incentivized during training.
- Entropy-Minimized Policy Optimization: Dual entropy thresholds exclude prompts with extremely high or low entropy to avoid unreliable optimization, redundancy, overconfidence, and potential reward hacking.The method also clips extreme advantages for stability and removes the KL constraint for better performance.
4 Experiments
Experiments evaluate EMPO on closed-form mathematical and free-form natural reasoning tasks using unlabeled prompts and comparisons with supervised or RL baselines. EMPO improves mathematical and natural-reasoning accuracy while exhibiting stable entropy-reward dynamics.
- Experimental setup: EMPO uses 20,000 NuminaMath-CoT prompts for mathematical training and Natural Reasoning prompts for free-form training.
- Experimental setup: EMPO is evaluated on Minerva Math, MATH, AMC23, OlympiadBench, AIME24, MMLU-Pro, and GPQA.
- Mathematical reasoning: 42.1% and 48.1% average mathematical accuracy are achieved by the 1.5B and 7B models, improving from 28.1% and 30.7%, respectively.
- Natural reasoning: 50.1% MMLU-Pro accuracy is reached by Qwen2.5-7B Base after EMPO, improving from 32.1%; GPQA accuracy rises from 15.9% to 28.8%.
- Training dynamics: EMPO’s unsupervised reward strongly negatively correlates with golden-answer rewards, while training shows decreasing semantic entropy and improving accuracy.
5 Discussion
The discussion argues that RL post-training, including EMPO, primarily improves efficient access to reasoning pathways already present in pretrained models. Pass@k behavior and semantic-consistency optimization support this interpretation.
- Pass@k analysis: At small to moderate k values such as 16 or 32, GRPO and EMPO improve pass@k over the base model.
- Pass@k analysis: At substantially large k, RL-trained models tend to converge with or be surpassed by the base model.
- Interpretation: These results indicate that foundational reasoning pathways are largely pre-existing, while RL refines their selection and prioritization.
- Interpretation: EMPO uses semantic consistency as an intrinsic reward to favor outputs selected from existing reasoning strategies without external correctness validation.
- Implications: EMPO is presented as a fully unsupervised, scalable, and practical approach for improving sampling efficiency where curated supervisory data is scarce or expensive.
Appendices
The appendices provide prompt templates for mathematical and natural reasoning evaluation and training, alongside a case study illustrating a model-generated mathematical response.
- Prompt templates: Mathematical and free-form natural reasoning templates instruct models to reason step by step and place final answers within boxed output.
- Prompt templates: The appendix supplies separate templates for base, few-shot, and finetuned MMLU-Pro and GPQA models.
- Case study: A case study presents Qwen2.5-Math-7B’s response to an AMC23 probability question using a dynamic-programming solution outline.
C Implementation Details about Semantic Clustering
The semantic-clustering implementation groups model outputs by meaning before EMPO optimization. Mathematical equivalence uses parsed-answer verification, while the algorithms initialize and extend response clusters iteratively.
- Semantic clustering: Semantic clustering takes a question and grouped model responses, initializes one cluster, and assigns outputs to existing or novel clusters based on semantic equivalence.
- Semantic clustering: The clustering procedure randomly selects an existing cluster element as the comparison output.
- Mathematical verification: The mathematical verifier parses two model outputs and returns whether their predictions are equivalent.
D Additional Results on Llama3 Model Series
EMPO could not be reproduced as fully unsupervised RL on Llama3 Base models, so the experiments instead used instruct-tuned initialization. The authors attribute this difference to Qwen2.5’s native instruction-following capabilities from its pretraining data.
- Training setup: Llama3 experiments initialized EMPO from instruct-tuned models because direct RL initialization from Llama3 Base models could not be implemented.The section contrasts this semi-supervised setup with fully unsupervised initialization for Qwen2.5 Base models.
- Training behavior: Qwen2.5 Base models showed inherent answer consistency early in EMPO training, whereas Llama3 Base models suffered severe inconsistency and failed to converge.This contrast is reported as the observed training behavior across the two model series.
- Proposed explanation: The authors hypothesize that Qwen2.5’s pretraining mixed web text with QA pairs generated by instruction-tuned Qwen2 models.They propose that this pretraining strategy endowed Qwen2.5 Base models with native instruction-following capabilities.
- Proposed explanation: Qwen2.5 Base models followed the instruction to place multiple-choice answers within a box and achieved accuracy above random guessing on MMLU-Pro.The passage presents this instruction-following result as experimental support for the pretraining hypothesis.
E Additional Training Details
The paper provides a concise summary of Qwen2.5 Base training recipes and releases the full configurations for reimplementation.
- Reproducibility: The supplementary materials contain the full training configurations for reimplementation.The authors state that the code and configurations were released in the supplementary materials.
- Training recipes: Table 4 summarizes the training recipes used for Qwen2.5 Base models.The table is described as a brief summary rather than a full procedural account.
F Computational Cost of Semantic Clustering
Semantic clustering adds a group-size-dependent computational step, while the mathematical implementation uses inexpensive regular expressions and the natural-reasoning implementation uses a compact small language model.
- Complexity: O(G2 × N) is the stated time complexity of semantic clustering for group size G and dataset size N.The complexity is defined over the number of sampled responses per question and the training dataset size.
- Implementation: Mathematical reasoning uses regular expressions for semantic clustering without notable computational cost.The paper contrasts this implementation with the additional model used for natural reasoning.
- Implementation: Natural reasoning uses an additional compact small language model for semantic clustering.This model supplies the clustering mechanism for free-form natural reasoning tasks.
- Data preparation: Mathematical prompts come from 20,000 randomly selected Numina-Math-CoT examples, while Natural Reasoning prompts are filtered by length and difficulty criteria.Filtering excludes overly long prompts or reference answers, overly difficult samples, and proof-oriented questions.
H Additional Result about Pass@k
Additional pass@k visualizations show that the Base model consistently catches up with RL-trained models when k is large.
- Pass@k: At large k, the Base model consistently catches up with RL-trained models in pass@k.The result is reported from the visualization in Figure H.
I The Influence of Clustering Quality on the Performance of EMPO
EMPO’s effectiveness depends on the quality of semantic clustering, with structured mathematical answers easier to cluster than diverse free-form responses. Poor clustering can limit scaling and produce reward hacking, while pairwise clustering signals remain weaker than correctness evaluation for identifying better answers.
- Regular-expression matching accurately determines answer equivalence in mathematical tasks because their response formats are naturally structured.This yields relatively high clustering quality without introducing additional models.
- Free-form natural reasoning responses are more diverse, making clustering quality an important constraint on EMPO’s effectiveness.Examples include matrices, numbers, sentences, and code.
- Poor semantic clustering made EMPO difficult to scale and caused frequent reward hacking before a general verifier was used for clustering.The general verifier enabled generalization to free-form reasoning tasks within the stated framework.
- The cluster model compares whether two outputs are semantically the same, but does not assess correctness, rank answers, or guide reasoning refinement.This makes its role narrower than that of a reward model or human-verified golden answer.
- Semantic clustering can face non-identifiability because different reasoning pathways or incorrect-answer clusters may generate identical relational signals.Thus, the same clustering pattern may correspond to multiple underlying states.