PaperlayerView exact revision on arXiv

Source-linked AI summary

The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning

Shivam Agarwal, Zimin Zhang, Lifan Yuan, Jiawei Han, Hao Peng

arXiv:2505.15134v1cs.LGcs.AI

TL;DR

The paper asks whether entropy minimization can improve pretrained LLM reasoning without labeled data. It evaluates unsupervised finetuning, entropy-only reinforcement learning, and inference-time logit optimization, finding strong gains across math, physics, and coding tasks, while effectiveness depends on base-model capability and confidence reliability.

  • Problem

    The paper asks whether entropy minimization alone can improve pretrained LLM performance on reasoning tasks without labeled data.

  • Method

    The paper develops EM-FT, EM-RL, and EM-INF, using unlabeled outputs, negative entropy rewards, or inference-time logit optimization as standalone entropy-minimization approaches.

  • Results

    Entropy minimization improves LLMs on complex math, physics, and coding tasks; Qwen models achieve comparable or better performance than RL baselines, and Qwen-2.5-32B outperforms GPT4o on complex scientific coding.

  • Takeaways & Limitations

    The findings suggest that many pretrained LLMs already possess strong reasoning ability that can be enhanced using entropy minimization alone.

  • Takeaways & Limitations

    EM depends on pretrained-model capability and on confidence being a reliable proxy for correctness; it can be ineffective or harmful when those conditions fail.

Abstract

from arXiv · show

Entropy minimization (EM) trains the model to concentrate even more probability mass on its most confident outputs. We show that this simple objective alone, without any labeled data, can substantially improve large language models' (LLMs) performance on challenging math, physics, and coding tasks. We explore three approaches: (1) EM-FT minimizes token-level entropy similarly to instruction finetuning, but on unlabeled outputs drawn from the model; (2) EM-RL: reinforcement learning with negative entropy as the only reward to maximize; (3) EM-INF: inference-time logit adjustment to reduce entropy without any training data or parameter updates. On Qwen-7B, EM-RL, without any labeled data, achieves comparable or better performance than strong RL baselines such as GRPO and RLOO that are trained on 60K labeled examples. Furthermore, EM-INF enables Qwen-32B to match or exceed the performance of proprietary models like GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro on the challenging SciCode benchmark, while being 3x more efficient than self-consistency and sequential refinement. Our findings reveal that many pretrained LLMs possess previously underappreciated reasoning capabilities that can be effectively elicited through entropy minimization alone, without any labeled data or even any parameter updates.

1 Introduction

The paper asks whether entropy minimization alone can improve pretrained LLMs without labeled data, and presents three methods spanning finetuning, reinforcement learning, and inference-time scaling. These methods improve performance on challenging reasoning tasks, but their effectiveness depends on model capability and confidence aligning with correctness.

  • Motivation: Entropy minimization concentrates probability mass on confident outputs and is studied as a way to improve capable pretrained LLMs without labeled data.The approach assumes that confidence is more likely to correspond to correctness when the model is reasonably capable.
  • Three Methods: EM-FT minimizes token-level entropy on unlabeled model outputs, mirroring supervised finetuning and improving math and coding performance.It can outperform labeled GRPO and RLOO on LeetCode and Minerva.
  • Three Methods: EM-RL uses negative token-level or trajectory-level entropy as its only reward and achieves competitive performance with labeled RLOO and GRPO.It outperforms those baselines on LeetCode, Minerva, and AMC.
  • Three Methods: EM-INF reduces distribution entropy by optimizing logits at each decoding step without updating model parameters.It works best on complex, high-uncertainty tasks such as AIME, UGPhysics, and SciCode.
  • Results: Qwen 32B can outperform GPT-4o on SciCode while EM-INF is 3x more efficient than self-consistency and sequential refinement.The result is reported for inference-time entropy minimization on scientific coding.
  • Limitations: EM is less effective when confidence is not a reliable quality proxy or when the pretrained model lacks sufficient capability for the task.The paper reports no improvement on individualistic value reasoning and smaller math gains for Llama-3.1-8B than for Qwen-2.5.

2 Background and Related Work

The paper situates entropy minimization within prior work on entropy-based regularization, unsupervised adaptation, self-improvement, and reinforcement learning. It applies entropy minimization as a standalone objective across post-training and inference-time scaling for LLMs.

  • Related Work: Entropy-regularized RL increases entropy to promote exploration, whereas this work investigates reducing uncertainty in pretrained language-model policies.The contrast is between exploration-oriented policy learning and confidence concentration in already pretrained models.
  • Related Work: Entropy minimization has established uses in semi-supervised learning, domain adaptation, few-shot learning, reinforcement learning, and test-time adaptation.The paper notes that EM has rarely been explored as a standalone training objective for LLMs.
  • Related Work: Self-improvement methods train LLMs through self-training, self-critique, refinement, or self-verification without external feedback.Related approaches use generated outputs or consistency-based signals to support learning without conventional labels.
  • Background: RL optimizes the expected reward of output sequences, while entropy minimization instead minimizes the policy’s Shannon entropy.The policy is autoregressive over output tokens conditioned on an input prompt.
  • Background: Trajectory-level entropy measures uncertainty over complete output sequences, whereas token-level entropy sums uncertainty across generation steps.These estimators can induce different behaviors when used for training.
  • This Work: The paper studies entropy minimization for LLM post-training and inference-time scaling through three methods: EM-FT, EM-RL, and EM-INF.EM-FT uses unlabeled outputs, EM-RL uses negative entropy rewards, and EM-INF adjusts logits without training data or parameter updates.

3 Entropy Minimization for Post-training

The paper develops entropy minimization as an unlabeled post-training objective, implementing it through direct finetuning and reinforcement learning. These methods use model confidence as supervision and achieve strong performance across math and coding tasks while avoiding answer extraction in settings where self-consistency is inapplicable.

  • EM-FT: EM-FT directly minimizes token-level entropy on sampled unlabeled trajectories, paralleling supervised finetuning without labels.
  • EM-RL: EM-RL uses negative trajectory-level or token-level entropy as the sole reward, with policy gradients optimizing the resulting objective.
  • Entropy estimators: Trajectory-level entropy favors a limited set of high-probability reasoning paths, whereas token-level entropy encourages deterministic confidence at each generation step.
  • Results: Entropy rewards apply across tasks without answer-extraction assumptions, unlike self-consistency, which is inapplicable to code generation.
  • Results: 8% average improvement over the base model is observed for EM-FT across math and coding without labels.
  • Results: 5% average gains on Minerva and LeetCode let EM-FT outperform GRPO and RLOO while using one sampled trajectory instead of N = 4.
  • Results: 11% average improvement over the base model is achieved by unsupervised RL, with 4.5% average gains over GRPO and RLOO on AMC, Minerva, and LeetCode.

4 EM-INF: Inference-time Logit Optimization

EM-INF performs inference-time scaling by minimizing output entropy through logit optimization, without updating model parameters. It improves performance across challenging reasoning tasks and can reduce computation relative to common scaling baselines.

  • Method: EM-INF updates output logits at each decoding step to reduce distribution entropy while keeping the model parameters frozen.Gradient descent treats the logits as free parameters; no backpropagation through or parameter update of the model is required.
  • Method: Adaptive temperature scaling reduces entropy by lowering τ until the distribution reaches a target entropy, while preserving logit ordering.The method searches for the largest τ satisfying the target entropy constraint.
  • Method: In high-uncertainty settings, logit optimization can reorder non-top logits, unlike temperature scaling, which only sharpens the distribution proportionally.Both methods preserve the top-logit token and increase its probability.
  • Results: 3% average improvement is observed across almost all evaluated math, coding, and physics task and model classes with test-time entropy minimization.EM-INF also outperforms self-consistency by 4% on AIME and 3.6% on UGPhysics.
  • Results: 3x greater efficiency is reported for EM-INF than self-consistency and sequential refinement, while Qwen-32B can outperform GPT-4o on SciCode.On SciCode, logit optimization is 3% better than adaptive temperature scaling.

5 Limitations: Dependence on Base Model Capability and Confidence

Entropy minimization depends on the pretrained model already possessing useful reasoning behaviors and reliable confidence signals. Its effectiveness can therefore vary substantially across base models and tasks.

  • Scope of Evaluation: The method’s applicability is probed by testing tasks with different biases from pretraining and models lacking target reasoning behaviors.These experiments examine whether entropy minimization transfers beyond the conditions supporting its strongest results.
  • Dependence on Task and Confidence: EM does not improve individualistic value reasoning for Qwen-2.5, where pretrained confidence is biased toward coarse training priors.This task requires predicting an individual’s preferences in new situations, rather than reproducing broad pretraining values.
  • Dependence on Base Model Capability: EM is less effective when the base model lacks target reasoning behaviors, as observed for Llama-3.1-8B on some math tasks.The authors connect unreliable initial reasoning behaviors to inaccurate entropy rewards.

6 Conclusion

The paper presents three unlabeled entropy-minimization methods that improve LLM performance on complex math, physics, and coding tasks. The results also show that success depends on the pretrained model and task, motivating EM baselines for future evaluations.

  • Conclusion: EM-FT, EM-RL, and EM-INF improve complex math, physics, and coding tasks without labeled data.The methods use unsupervised finetuning, negative-entropy reinforcement learning, or inference-time logit optimization.
  • Conclusion: Qwen-2.5-7B achieves comparable or better performance than GRPO and RLOO through EM-FT and EM-RL.These comparisons concern labeled reinforcement-learning methods used as baselines.
  • Conclusion: Qwen-2.5-32B outperforms GPT4o on complex scientific coding through test-time logit optimization without training or parameter updates.This result is reported for EM-INF on scientific coding.
  • Conclusion: EM is less beneficial for tasks and models with inductive biases differing from pretraining.The authors therefore recommend including EM methods as baselines when evaluating future post-pretraining and inference-time scaling algorithms.

A Proof for proposition 1

The proposition analyzes entropy-gradient updates to show that reducing entropy can alter non-top logits while preserving the top-logit index. The accompanying experiments compare EM-INF with inference-time scaling methods and report efficiency and diversity trade-offs.

  • Proposition and Interpretation: Entropy minimization and temperature scaling both reduce output entropy, but logit optimization can reorder non-top logits in high-uncertainty settings.Temperature scaling preserves logit order and changes distribution sharpness proportionally.
  • Proof: The proof constructs logits where a lower-logit index receives a larger entropy-gradient component than a higher-logit index.It uses g(p)=p(log p+H), the mean-value theorem, and the condition pa−pb < e^−(H+1).
  • Proof: The entropy gradient cannot reduce a lower-probability token relative to the highest-probability token.The proof establishes that the index receiving the maximum adjustment cannot be the maximum-logit index.
  • Inference-Time Scaling Comparison: EM-INF can match self-consistency with N=4 on AMC while using 3 times fewer FLOPs.Its benefits diminish over iterations because entropy minimization reduces generation diversity.
  • Inference-Time Scaling Comparison: Adaptive temperature scaling searches for a temperature whose output entropy reaches a target based on the initial entropy and threshold.The procedure uses iterative midpoint updates between minimum and maximum temperatures.

C.2 Qualitative analysis of EM-INF on SciCode

The SciCode case study contrasts a partially correct base-model implementation with a correct EM-INF generation. The comparison links lower-entropy decoding to more deterministic, concise code for an uncertain scientific-coding task.

  • Case Study: The base Qwen2.5-7B-Instruct implementation correctly initializes the diagonal and symmetrizes the matrix but omits noise on diagonal elements.The prompt requires noise on every element before symmetrization.
  • Case Study: EM-INF generates code satisfying all prompt constraints, including applying the noise factor throughout the matrix.The resulting implementation is presented as correct in the SciCode case study.
  • Combined Scaling: Figure 2 reports EM-INF combined with other inference-time scaling methods and indicates negligible additional computation.The figure caption states that EM-INF can improve performance in these combinations.
  • Generation Characteristics: EM-INF reduces token diversity and produces more deterministic, concise generations, sometimes omitting additional comments.The authors associate these characteristics with its lower-entropy output distribution.

D.1.1 Math Reasoning

This section describes the benchmarks and evaluation setup used to assess entropy minimization on math, coding, physics, scientific coding, and value reasoning tasks.

  • Evaluation setup: Math evaluations use Pass@1 accuracy, with the final answer extracted from the model’s last boxed expression.The pipeline adopts PRIME-RL’s data and evaluation procedure for math reasoning.
  • Math Reasoning: MATH 500 contains 500 competition-style problems spanning seven mathematical subjects and five difficulty levels.Problems include boxed answers and step-by-step solutions, enabling exact-match evaluation.
  • Math Reasoning: The evaluation includes AMC, AIME, Minerva Math, and OlympiadBench to cover competition, advanced, undergraduate STEM, and Olympiad-level reasoning.These datasets range from 83 AMC problems and 90 AIME problems to 272 Minerva problems and 674 OlympiadBench problems.
  • Coding: Coding evaluation uses LeetCode and LiveCodeBench, emphasizing competition-level program generation and correctness on test cases.LiveCodeBench V2 contains 511 recent problems, while LeetCode problems are paired with 100 test cases each.
  • Scientific Reasoning: Scientific reasoning evaluation uses UGPhysics and SciCode, which require multi-step physics reasoning or integrated scientific Python solutions.UGPhysics evaluation uses a 372-problem Semiconductor Physics subset; SciCode’s test set contains 65 main problems and 288 subproblems.
  • Value Reasoning: The study also includes IndVal, a World Values Survey-derived dataset of statements expressing individual human values and preferences.The dataset draws on responses from more than 93,000 participants worldwide.

D.2 Training Data

The experiments use sampled math, coding, and value-reasoning prompts with several Qwen and Llama model variants, while comparing training and inference costs under early stopping.

  • Training Data: Math training uses 35K randomly sampled prompts from NuminaMath, spanning high-school through advanced competition problems.NuminaMath contains approximately 860,000 problems, including Olympiad, AMC, AIME, and Chinese K-12 mathematics.
  • Training Data: Coding training uses 25K randomly sampled prompts from the Eurus-2-RL coding split, which contains about 27,000 coding problems with test-case verifiers.The broader Eurus-2-RL dataset contains about 457,000 math problems and 27,000 coding problems.
  • Training Data: Individualistic value reasoning uses 70,200 prompts generated from profiles built from sampled IndVal participants and value statements.Profiles contain 52 known statements, with six query questions sampled per category.
  • Models: Post-training experiments use Qwen2.5-Math-7B for math, Eurus-2-7B-SFT for coding, and Llama-3.1-8B-Instruct to test cross-family generalizability.Eurus-2-7B-SFT is further fine-tuned from Qwen2.5-Math-7B on an action-centric chain-of-thought dataset.
  • Models: Inference-time experiments use Qwen2.5-7B and Qwen2.5-Math-7B-Instruct, plus Qwen2.5-32B-Instruct for SciCode.The 32B model is selected for its stronger capability on complex scientific coding problems.
  • Compute: Training FLOPs are estimated as 6PD and inference FLOPs as 2PD, with validation-set early stopping applied to all methods.For a 7B model, EM-FT costs 1.01 × 10^17 FLOPs under the stated configuration, whereas EM-RL, GRPO, and RLOO cost 13 × 10^17 FLOPs.
Loading 2505.15134v1…