Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization

1 Introduction

• Reasoning LLM training commonly uses SFT followed by RL and depends on labeled traces, verified answers, or reward models, making expert-driven scaling costly [5].Source: §1 · p. 1
• Self-consistency pseudo-data offers limited gains and risks model collapse [6], while PFPO [8] and self-rewarding RL [9] still require external supervision or a ground-truth verifier.Source: §1 · pp. 1–2
• EMPO samples responses, groups semantically equivalent answers, and minimizes semantic entropy, with thresholding to filter unreliable traces and stabilize unsupervised training.It aims to favor reasoning traces that yield consistent answers, without rule-based rewards, predefined test cases, or pretrained reward models.Source: §1 · p. 2, Figure 2 · p. 2
• The paper reports a strong negative correlation between semantic entropy and model accuracy, supporting entropy as an intrinsic unsupervised objective for more consistent, reliable outputs.Source: §1 · pp. 2–3
• Across mathematical and free-form natural reasoning tasks, EMPO’s gains are attributed to selecting strong pre-existing reasoning pathways rather than instilling fundamentally new skills.Source: §1 · p. 3

Self-Supervised and Semi-Supervised Reasoning

• Self-improvement methods generate pseudo-labels [6] [13] [14] but retain SFT, while TTRL [15] uses test prompts and is math-limited, unlike EMPO’s train/test separation and broader reasoning scope.Self-improvement can yield limited gains and risks model collapse.Source: §2 · p. 3

Self-Rewarding and RL-based Reasoning

• Prior RL methods rely on self-judges, supervised or verifier signals [9] [8] [18], rule rewards [12], or reward-model consistency [19], while ETPO encourages entropy rather than minimizing it.EMPO is positioned as fully unsupervised, using semantic entropy as its internal reward.Source: §2 · p. 3

Entropy Minimization and Semantic Consistency

• Prior work uses entropy minimization for unlabeled classification, test-time adaptation [21] [22], and semantic-entropy applications to hallucination detection; EMPO extends it to latent-space LLM reasoning [23].COME applies conservative entropy minimization for robust adaptation [22], while emergent-language findings link entropy minimization to successful communication [25].Source: §2 · p. 3

3.1 Preliminaries

• Unlike supervised GRPO reward setups, EMPO operates on unlabeled user questions without golden answers, using pre-trained LLMs to improve reasoning while minimizing data-collection cost.The method’s unsupervised setting excludes labeled correct-answer pairs and valid reasoning trajectories.Source: §3.1 · p. 4, §3.3 · p. 5

3.2 Semantic Entropy Minimization Objective

• Semantic entropy groups outputs by meaning rather than format, and its minimization encourages consistent predictions that are associated with more reliable answers.The method estimates meaning probabilities from the proportions of sampled outputs in each cluster; prior work links semantic entropy negatively with accuracy and uses it to detect unreliable generations [23].Source: §3.2 · pp. 4–5

3.3 Entropy-Minimized Policy Optimization

• EMPO rewards sampled outputs according to the likelihood of their meaning clusters, encouraging higher-probability meanings without correctness labels or reasoning-trajectory supervision.It groups semantically equivalent outputs, estimates each cluster’s probability from its share of samples, and uses that probability as the reward; an optional small model helps cluster meanings but does not judge correctness.Source: §3.3 · p. 5
• Entropy thresholding excludes prompts with overly high or low semantic entropy, avoiding unreliable predictions and reducing redundant optimization and overconfidence risks.EMPO also removes the KL constraint for better performance and clips extreme advantages for stability.Source: §3.3 · p. 5

4.1 Experimental Settings

• Training uses 20,000 NuminaMath-CoT prompts for mathematics and 18,000 filtered Natural Reasoning questions for free-form tasks.Natural Reasoning prompts with overlong questions or reference answers are filtered, as are samples whose Llama3.3-70B-Instruct response exceeds 4096 tokens.Source: §4.1 · p. 6
• Evaluation covers Minerva Math, MATH, AMC23, OlympiadBench, AIME24, MMLU-Pro, and GPQA, generally using zero-shot prompts and greedy decoding.The models are Qwen2.5-Math-1.5B and 7B Base for mathematics and Qwen2.5-3B, 7B, and 14B Base for natural reasoning, compared with SFT, RL, instruction, and prompting baselines.Source: §4.1 · p. 6

4.2.1 Performance on Mathematical Reasoning Tasks.

• 42.1% and 48.1% average mathematical benchmark accuracy for EMPO on 1.5B and 7B models, respectively, approaches Qwen2.5-Math-Instruct’s 40.5% and 49.4%.EMPO raises the averages from 28.1% to 42.1% on 1.5B and 30.7% to 48.1% on 7B; the instruction model uses private data and iterative supervised fine-tuning and reinforcement learning.Source: Table 1 · p. 7, §4.2.1 · p. 7

4.2.3 Training Dynamics

• EMPO’s unsupervised reward signals are strongly negatively correlated with golden-answer rewards, while training decreases semantic entropy and improves accuracy.The authors report a stable learning process along the unsupervised RL training trajectory.Source: §4.2.3 · p. 7, Figure 3 · p. 8

4.2.2 Performance on Natural Free-form Reasoning Tasks.

• 50.1% and 58.8% MMLU-Pro accuracy and 28.8% and 35.3% GPQA accuracy are achieved by EMPO on 7B and 14B models, respectively.On MMLU-Pro, EMPO raises accuracy from 32.1% to 50.1% on 7B and 32.7% to 58.8% on 14B; on GPQA, it raises accuracy from 15.9% to 28.8% and 30.6% to 35.3%, respectively. The SFT baseline does not consistently improve performance, which the authors hypothesize may reflect noise in Natural Reasoning reference responses.Source: §4.2.2 · p. 7, Table 2 · p. 8

The Role of Unsupervised Learning in Eliciting Pre-Trained Reasoning Capabilities

• Despite EMPO’s strong performance and ability to match or slightly outperform supervised counterparts like GRPO on the 7B model, benchmark gains show no consistent increase in response length or clear “Aha moment.”The authors present this as motivation to examine how reasoning incentivization works.Source: §5 · p. 8
• At small k, GRPO and EMPO outperform the base model, but at large k their pass@k performance converges with or is sometimes surpassed by the base model.At k=16 or 32, RL-trained models improve the efficiency of surfacing correct reasoning paths with fewer attempts.Source: §5 · pp. 8–9, Figure 4 · p. 9
• The convergence at high k and base-model examples of pausing, self-correction, and backtracking suggest that pre-training supplies reasoning pathways which RL refines for more efficient and reliable selection.The authors align this interpretation with an emerging, not unanimous consensus [41] and prior insights [43], attributing EMPO’s efficacy to the strong reasoning capabilities of Qwen2.5 Base.Source: §5 · p. 9
• EMPO uses intrinsic rewards from minimizing semantic entropy to favor consistent outputs, guiding selection among existing reasoning strategies without external correctness validation.The authors argue that, in a well-pre-trained model, semantically consistent outputs are more likely to align with correct and coherent reasoning.Source: §5 · p. 9
• The authors conclude that improving sampling efficiency and reliability is practically important, and that EMPO offers a scalable, cost-effective unsupervised approach where curated supervisory data is scarce or prohibitively expensive.They emphasize that RL may enhance access to pre-trained abilities without forging entirely new fundamental reasoning capabilities.Source: §5 · p. 9

A Prompt Templates

• The mathematical reasoning template follows Online-DPO-R1, and the authors report that a system prompt helped Qwen2.5-Math Base on math benchmarks but not natural reasoning.They therefore use a system prompt for mathematical testing of both Base and finetuned models, but omit it for natural-reasoning Base models.Source: A Prompt Templates · p. 13
• The free-form natural reasoning training template asks models to reason step by step and place the final answer within boxed delimiters.The authors describe it as similar to the mathematical training template.Source: A Prompt Templates · p. 13
• For MMLU-Pro Base testing, the prompt requests step-by-step reasoning and a boxed answer letter from A–P; a few-shot variant includes demonstrations and their reasoning.MMLU-Pro and GPQA are described as close-formed multiple-choice benchmarks.Source: A Prompt Templates · pp. 13–14
• For MMLU-Pro finetuned models, the test prompt requests step-by-step reasoning and a boxed final answer letter from A–P.Source: A Prompt Templates · p. 14
• For GPQA Base testing, the prompt asks for step-by-step reasoning and a boxed answer of A, B, C, or D; a few-shot variant supplies demonstrations and their reasoning.Source: A Prompt Templates · p. 14
• For GPQA finetuned models, the test prompt requests step-by-step reasoning and a boxed answer choice from A, B, C, or D.Source: A Prompt Templates · p. 15

B Case Study

• In the AMC23 case, Qwen2.5-Math-7B is asked for the probability that a running die-roll total reaches 3 at some point across four rolls.Its response proposes tracking cumulative-total probabilities and checking after each roll, then says an implementation attempt had an issue and restarts its approach.Source: B Case Study · p. 15
• The response ultimately reports 4/1296 = 1/324 and gives m + n = 325 as its final answer.Source: B Case Study · p. 15

C Implementation Details about Semantic Clustering

• The appendix provides simplified Python code for checking output equivalence in both mathematical and free-form natural reasoning tasks.The described implementations correspond to Algorithms 2 and 3.Source: C Implementation Details about… · p. 16
• Semantic clustering initializes C with o1, compares each new output with a cluster representative, and places semantically distinct outputs in a novel cluster.Algorithm 1 gives the clustering procedure; the supplied fragments identify the initial cluster and the distinction between equivalent and semantically distinct outputs.Source: C Implementation Details about… · p. 16
• For mathematical reasoning, the verifier parses both model outputs and checks their equivalence with `verify`.Algorithm 2 imports `parse` and `verify` from `math_verify`.Source: C Implementation Details about… · p. 16

D Additional Results on Llama3 Model Series

• Llama3 Base models require cold-start SFT before RL: unlike Qwen2.5 Base, they show severe answer inconsistency and fail to converge under unsupervised training.Source: D Additional Results on Llama3 Model… · p. 16
• The authors attribute Qwen2.5 Base models’ native instruction-following ability to pretraining on web text and QA pairs generated by instruction-tuned Qwen2 models [5].They report that Qwen2.5 Base follows answer-format instructions on MMLU-Pro and scores above random guess.Source: D Additional Results on Llama3 Model… · p. 16
• EMPO substantially improves reasoning for Llama3.2-3B-Instruct using an instruct-tuned starting point rather than fully unsupervised Base-model RL.Table 3 reports accuracy on mathematical reasoning benchmarks; the supplied passages do not provide its cell values.Source: D Additional Results on Llama3 Model… · p. 16, Table 3 · p. 18

• Table 4 briefly summarizes Qwen2.5 Base model training recipes, while supplementary code provides full configurations for reimplementation.Source: Table 4 · p. 18, E Additional Training Details · p. 18

F Computational Cost of Semantic Clustering

• Clustering has time complexity O(G^2 × N); mathematical tasks use regular expressions, while natural tasks use an additional compact small language model.G is the number of responses sampled per question, and N is the training dataset size.Source: F Computational Cost of Semantic… · p. 18
• The 14B experiments took slightly less time than the 3B and 7B configurations because batch size and maximum response length were reduced to fit one 8×A100 80G machine.The 14B settings were batch size 1 and maximum response length 768, versus 2 and 1024 for the 3B and 7B settings.Source: F Computational Cost of Semantic… · p. 18
• The semantic-clustering process is reported to require minimal computation and storage, with EMPO and GRPO compared on runtime and GPU-memory use for mathematical and natural reasoning.The comparisons report total runtime in 8×A100 GPU hours and GPU-memory utilization in GiB.Source: Table 5 · p. 18, F Computational Cost of Semantic… · p. 18, Table 6 · p. 19

G Details of Prompt Collection

• Mathematical reasoning uses 20,000 prompts randomly selected from Numina-Math-CoT.Source: G Details of Prompt Collection · p. 19
• Natural-reasoning prompts come from Natural Reasoning5 after filtering out questions with overly long prompts or reference answers.Llama3.3-70B-Instruct response length estimates difficulty; samples exceeding 4096 tokens are removed.Source: G Details of Prompt Collection · p. 19

H Additional Result about Pass@k

• In Figure H, the EMPO-trained Base model consistently catches up with RL-trained models at large k on Pass@k.The section provides additional visualized Pass@k results for models trained with EMPO.Source: H Additional Result about Pass@k · p. 19

I The Influence of Clustering Quality on the Performance of EMPO

• Regular-expression matching alone accurately identifies equivalent answers in structured mathematical tasks, yielding relatively high clustering quality without an additional model.Mathematical responses have naturally structured formats.Source: I The Influence of Clustering Quality… · p. 19
• Poor DeBERTa clustering hindered EMPO’s scaling on free-form reasoning and caused frequent reward hacking, while a Tiger-Lab general verifier enabled generalization.The verifier is a fine-tuned Qwen2.5-1.5B-Math model used as a fixed utility for semantic comparison, not as a task-specific supervisor.Source: I The Influence of Clustering Quality… · pp. 19–20
• The cluster model only makes pairwise same-answer comparisons; it neither evaluates correctness against the query nor provides gradients or hints to refine reasoning.The passage contrasts this role with reward models and human-verifier golden answers, and notes that clustering can be easier to implement; it gives regular expressions for math and a fine-tuned Qwen2.5-1B model for natural reasoning as examples.Source: I The Influence of Clustering Quality… · p. 20
• Relational clustering signals are non-identifiable because distinct underlying truths or reasoning pathways can produce the same semantic-clustering pattern.The passage includes different clusters of incorrect answers among the possible underlying states.Source: I The Influence of Clustering Quality… · p. 20