PaperlayerView exact revision on arXiv

Source-linked AI summary

Entropy Regularization: A Free Correction to Cross-Entropy for Verified Demonstrations

Mihir Dhanakshirur, Adam Ousherovitch, Ambuj Tewari

arXiv:2609.30572v1cs.LGstat.ML

TL;DR

In verifiable tasks with multiple correct solutions, models often receive one expert demonstration even though the goal is to produce any verifier-accepted answer, and cross-entropy can be misaligned with that goal. The paper proposes entropy-regularized cross-entropy, using token-level Shannon entropy as a tractable proxy for support size. It reports improved verifier accuracy on math and code benchmarks, with negligible additional training overhead.

  • Problem

    In tasks with multiple correct solutions, post-training may use one demonstration per problem even though the objective is verifier acceptance rather than imitation, and cross-entropy can be misaligned with verifier risk.

    Source: §1 · pp. 1–2

  • Method

    The paper proposes entropy-regularized cross-entropy, using token-level Shannon entropy as a tractable, differentiable proxy for policy support size.

    Source: §1 · p. 2, §6 · pp. 10–11

  • Results

    Entropy-regularized training improves verifier accuracy over standard cross-entropy across mathematical reasoning and code-generation benchmarks, with negligible additional training overhead.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    Controlling policy support can mitigate the failure mode, and entropy regularization offers a practical objective aligned with producing correct outputs in the evaluated tasks.

    Source: §6 · pp. 10–11

    The work evaluates Pass@1, and efficiently selecting regularization strength remains a practical challenge because the minimum nonzero probability is typically unknown.

    Source: §6 · p. 11

Abstract

from arXiv · show

Large language models are often post-trained on expert demonstrations using cross-entropy (CE), even when the downstream objective is not to imitate the demonstrated solution but to produce any output accepted by a verifier. This mismatch is seen in verifiable domains with multiple correct solutions, such as mathematical reasoning and code generation, where training data may contain only one expert solution per problem. We show that minimizing cross-entropy can be misaligned with minimizing verifier risk; two policies can assign identical likelihood to the observed demonstrations while placing different probability mass on incorrect outputs. This is formalized through a learning-theoretic counterexample in which CE minimization selects a suboptimal policy. We identify that controlling the support of the learned policy can solve this problem by preventing probability mass from spreading to unsupported outputs. Since support size is non-differentiable and computationally intractable, we propose entropy-regularized cross-entropy (ER-CE), using token-level Shannon entropy as a tractable proxy. Finally, across mathematical reasoning and code-generation benchmarks, we find that entropy-regularized training consistently improves verifier accuracy over standard cross-entropy. Our results identify a simple failure mode of imitation-based post-training in verifiable tasks and provide a practical objective that is better aligned with producing correct outputs.

1 Introduction

In verifiable tasks with multiple correct solutions, post-training often has only one expert demonstration per problem, although success depends on producing any answer the verifier accepts. The paper argues that cross-entropy can misalign with verifier success and introduces entropy-regularized cross-entropy, reporting improved verifier accuracy across math and code benchmarks.

1 Introduction

  • With one expert solution per problem, the goal is to produce any solution accepted by the verifier, not necessarily to copy the demonstration.The setting includes mathematical reasoning and code generation, where problems can have multiple correct solutions.Source: §1 · p. 1
  • Cross-entropy can penalize deviations from the demonstration without ensuring the model learns to satisfy the verifier.The cited work by Joshi et al. (2026) shows CE fails to PAC-learn verifier reward when the policy class cannot perfectly model the demonstrator, but offers no tractable method to use that insight.Source: §1 · pp. 1–2
  • Policies with identical likelihood on demonstrations can place different probability mass on verifier-rejected answers, making cross-entropy minimization suboptimal for verifier success.The paper gives a learning-theoretic example and motivates favoring policies with lower entropy.Source: §1 · p. 2
  • The paper introduces entropy-regularized cross-entropy and establishes conditions under which it improves on standard cross-entropy minimization.The objective is presented as a tractable correction to the mismatch.Source: §1 · p. 2
  • Entropy-regularized cross-entropy outperforms standard cross-entropy across math problem-solving and code-generation tasks.The paper reports these empirical results in Section 5.Source: §1 · p. 2

2 Related Work

Related work frames the paper's problem as a mismatch between predictive training losses and downstream objectives, and situates its entropy penalty among prior entropy methods. It also contrasts the paper's supervised, policy-modeling approach with related empirical and theoretically motivated methods.

2.1 Aligning Post-Training with the Downstream Objective

  • Decision-focused learning similarly distinguishes predictive loss from downstream decision quality; related inference-aligned losses target decoding strategies, while this paper's mismatch occurs at Pass@1 without search.The paper describes its gap as upstream of and complementary to inference-side work.Source: §2.1 · pp. 2–3

2.2 Entropy Regularization of Output Distributions

  • Unlike entropy-regularization methods that encourage low-entropy outputs or exploration, this work penalizes entropy; Agarwal et al. (2025) instead use unlabeled self-generated outputs and report empirical results.The paper distinguishes its supervised demonstration setting and theoretical justification from EM-FT.Source: §2.2 · p. 3

2.3 Learning from an Expert Policy

  • Joshi et al. (2026) propose a verifier-learning version-space algorithm, whereas this paper retains policy modeling for pretrained models and uses an objective incorporable into standard supervised fine-tuning with negligible overhead.The passage characterizes the version-space method as generally computationally intractable.Source: §2.3 · p. 3

3 Limitation of Cross-Entropy

The section sets up learning from correct demonstrations with verifier risk as the target, then shows that empirical cross-entropy minimization can fail to PAC-learn. The counterexample links this failure to how probability mass outside the demonstrated answers’ support is treated.

3.1 Problem Setup

  • The setup treats questions as X, answers as a finite Y, and hypotheses as distributions over answers conditioned on questions; the learning target is minimum verifier risk.The demonstrator is an unknown true hypothesis, and the verifier indicates correctness for each question-answer pair.Source: §3.1 · p. 4

3.2 Counter-example

  • The empirical cross-entropy minimizer fails to PAC-learn the constructed hypothesis class.Despite the true hypothesis, it selects a policy with non-zero verifier risk, assigning 1/4 probability outside the true hypothesis’s support.Source: §3.2 · p. 5
  • When the sampled questions are distinct, a hypothesis can match every demonstrated answer and achieve lower sample cross-entropy than the true hypothesis.For every sample size, some question distribution yields distinct samples with probability bounded away from zero.Source: §3.2 · p. 5
  • The counterexample exploits an asymmetry: cross-entropy depends on the sampled answer’s probability, while verifier risk depends on probability over the other answers.The verifier-risk comparison therefore depends on probability mass the observed cross-entropy does not constrain.Source: §3.2 · p. 5
  • In the counterexample, the support class has size 2 despite exponentially many probabilistic hypotheses, and a trivial algorithm returning u has zero verifier risk.This motivates losses that explicitly account for candidate support.Source: §3.2 · p. 5

4 Entropy Regularized Cross-Entropy

The section gives a PAC-learning guarantee for modifying cross-entropy with a support-related penalty, then replaces support quantities with differentiable entropy-like surrogates. The guarantee applies under finite support-class and probability-bound assumptions, with specified parameter schedules.

4 Entropy Regularized Cross-Entropy

  • Finite induced support class is the key structural assumption; the proof uses a finite bounded loss class to obtain generalization as sample size increases.The paper notes softmax prevents zero probability for a token sequence in the LLM-verifier setting.Source: §4 · pp. 6, 8
  • The paper states that modifying cross-entropy can PAC-learn these hypothesis classes under a minimal assumption.The theorem additionally requires the stated finite support-class and probability-bound assumptions.Source: §4 · p. 6
  • The method replaces support size with differentiable entropy-like surrogates: S_1(p) = e^H(p), while S_α(p) = e^R_α(p) for α ≠ 1.Here H is Shannon entropy and R_α is Rényi entropy of order α.Source: §4 · p. 6
  • Under finite induced support-class and probability-bound assumptions, the modified cross-entropy objective PAC-learns the hypothesis class when λ_m increases and α_m decreases to zero.The paper also gives a corollary using λ_m = m and α = 1/m.Source: §4 · pp. 6, 8
  • The proof progressively replaces the unsupported-output indicator with negative log probability, then support size with its surrogate, yielding cross-entropy plus an entropy penalty.The three steps move from an indicator-plus-support penalty to cross-entropy-plus-support penalty and then cross-entropy-plus-entropy penalty.Source: §4 · p. 7

5 Experiments

Experiments fine-tune an LLM on one demonstrated solution per problem in math and code, then evaluate whether outputs pass a verifier. Entropy regularization improves reported accuracy, with outcomes varying by task and penalty strength.

5 Experiments

  • Training uses one demonstrated solution per problem, while evaluation checks verifier success; gains largely disappear with answer-only targets and are larger under temperature decoding.The experiments cover math problem-solving and code generation.Source: §5 · pp. 8–9

5.1 Math Problems

  • GSM8K Pass@1 accuracy peaks about 9% above standard cross-entropy as the tested entropy penalty increases.The sweep uses five seeds.Source: Table 1 · p. 9, §5.1 · p. 9

5.2 Code Generation

  • At λ = 1.0, MBPP Greedy Pass@1 reaches 0.5710 ± 0.0058 (+4.40% absolute); at λ = 6.0, it falls to 0.5380 ± 0.0103.The baseline at λ = 0.0 is 0.5270 ± 0.0103.Source: Table 2 · p. 10, §5.2 · p. 10

5.3 Scaling: Harder Problems and a Larger Model

  • Every regularized configuration outperforms the cross-entropy baseline (0.4200 ± 0.0071), with the weakest penalty reaching 0.4583 ± 0.0062; the largest gain is +5.17% absolute at λ = 4.Source: §5.3 · p. 10

6 Conclusion

ER-CE uses token-level Shannon entropy as a tractable proxy for support size to address the mismatch between imitation-based training and verifier accuracy. The conclusion reports improved verifier accuracy across mathematical reasoning and code generation, with a MATH gain also reported for a 7B model.

6 Conclusion

  • ≈5%: ER-CE peaks over normal CE on MATH greedy Pass@1 accuracy, aggregated across 3 seeds.The reported gain remains for Qwen2.5-7B-Instruct, with greedy Pass@1 accuracy plotted against entropy penalty λ and error bars showing ±1 standard deviation over 3 seeds.Source: Table 3 · p. 11, Figure 3 · p. 11
  • ER-CE uses token-level Shannon entropy as a tractable proxy for support size and improves verifier accuracy on mathematical reasoning and code-generation benchmarks with negligible additional training overhead.The paper identifies settings where this objective improves upon standard cross-entropy minimization.Source: §6 · pp. 10–11
  • The study considers only Pass@1; whether entropy minimization helps with tree search, majority voting, or Best-of-N sampling remains open.Selecting the regularization strength is also a practical challenge because the minimum nonzero probability is typically unknown.Source: §6 · p. 11

A Proof of Proposition 3.2

The proof constructs a setting where empirical cross-entropy minimization fails to learn a verifier-optimal policy, then establishes PAC-learning results for losses with increasing regularization. These results provide intermediate steps toward the support-penalty and entropy arguments developed later.

A Proof of Proposition 3.2

  • The counterexample has an optimal policy u with verifier-risk 0, while the empirical cross-entropy minimizer differs from u and has verifier-risk 1/4.The construction shows that empirical cross-entropy minimization does not PAC-learn the hypothesis class under verifier-risk.Source: A Proof of Proposition 3.2 · p. 16

B.1 Step 1

  • Step 1 establishes a generalization-bound argument using a finite loss class and Hoeffding’s inequality, with the comparator h* having verifier-risk 0.The supplied proof identifies the empirical-risk difference as nonpositive before applying the concentration bound.Source: §B.1 · pp. 16–17

B.2 Step 2

  • Under Assumptions 4.2 and 4.3, the theorem establishes PAC learning for the specified regularized loss when λ_m increases to infinity.The corollary sets λ_m = m and gives a sample-complexity result.Source: §B.2 · pp. 17, 19

B.3 Step 3

The proof extends the PAC-learning result to a loss using S_α as α approaches zero, then discusses properties of a tractable support surrogate. The accompanying GSM8K setup specifies its data, model, and training configuration.

B.3 Step 3

  • For λ_m increasing to infinity and α_m decreasing to zero, the theorem establishes PAC learning; a corollary uses λ_m = m and α = 1/m.The theorem is stated under Assumptions 4.2 and 4.3.Source: §B.3 · pp. 20–21

C Discussion on (Sα(·))

  • The support penalty is non-differentiable and computationally tedious, motivating a continuous, symmetric surrogate that preserves its support-size limit as α approaches zero.The proposed surrogate also leaves its value unchanged when a zero-probability outcome is added.Source: C Discussion on (Sα(·)) · p. 22

D Experimental Setup GSM8k

D.1 Data Preparation

  • The GSM8K preparation uses 1,000 randomly sampled training problems and 500 randomly sampled evaluation problems.Prompts request step-by-step solutions ending in an #### answer format, and prompt tokens are masked from both losses.Source: §D.1 · p. 23

D.2 Model Architecture & LoRA Configuration

  • The GSM8K experiments use Qwen2.5-1.5B-Instruct with LoRA, using rank 16 and seven listed attention and feed-forward target modules.Training uses bf16 precision with Flash Attention 2.Source: §D.2 · p. 23

D.3 Training Hyperparameters

  • GSM8K training uses a custom trainer with dynamically calculated in-graph token entropy, cosine decay, batch size 16, and AdamW.The maximum sequence length is 768 tokens, and generated chain-of-thought is capped at 512 tokens.Source: §D.3 · p. 23

D.4 Evaluation details

The evaluation details specify deterministic Pass@1 scoring for GSM8K, while the MBPP setup documents its data preparation, model, and training configuration. The supplied passages do not report per-seed outcome values.

D.4 Evaluation details

  • GSM8K evaluation uses deterministic greedy Pass@1, strict answer matching with a float tolerance below 10^-4, and five random seeds per λ configuration.The reported statistics are the mean and standard deviation across seeds.Source: §D.4 · p. 24

E Experimental Setup MBPP

E.1 Data Preparation

  • MBPP preparation uses 300 randomly sampled training problems and 200 randomly sampled evaluation problems.Prompts request a complete Python function, provide the first unit test as a naming hint, and mask prompt tokens from both losses.Source: §E.1 · p. 25

E.2 Model Architecture & LoRA Configuration

  • The MBPP experiments use Qwen2.5-1.5B-Instruct with LoRA rank 16 and the listed attention and feed-forward target modules.Training uses bf16 precision with scaled dot-product attention.Source: §E.2 · p. 25

E.3 Training Hyperparameters

  • MBPP training uses a custom Hugging Face Trainer with PEFT, cosine decay, batch size 16, and AdamW.The maximum sequence length is 768 tokens, with generated code capped at 256 tokens.Source: §E.3 · p. 25

E.4 Execution-Based Evaluation

The appendices specify execution-based evaluation for code and the MATH experiment pipeline, including memory-efficient entropy computation and answer-equivalence checks. They also describe the sampled evaluation used in the temperature ablation.

E.4 Execution-Based Evaluation

  • Code is evaluated with deterministic greedy Pass@1 in a sandbox that runs dataset assertions and marks exceptions or runs exceeding two seconds incorrect.Each configuration uses five seeds to capture variation in LoRA initialization and training-data order.Source: §E.4 · p. 26

F.1 Data Preparation

  • MATH training uses 1,000 randomly sampled training problems and evaluates on 200 held-out MATH-500 problems, filtering out examples without boxed answers or with Asymptote figures.Prompts request step-by-step solutions ending in a boxed answer; prompt tokens are masked from both losses.Source: §F.1 · p. 27

F.2 Model Architecture and LoRA Configuration

  • MATH experiments use Qwen2.5-7B-Instruct with LoRA and three random seeds per penalty setting.The setup uses sequences up to 1,024 tokens and computes token entropy dynamically during training.Source: §F.2 · p. 27, §F.3 · pp. 27–28

F.4 Chunked Computation of the Entropy Term

  • Chunking entropy computation into 512 response-position blocks reduces loss-head peak memory from O(NV) to O(CV) while remaining numerically identical to the unchunked calculation.Gradient checkpointing discards fp32 intermediates after each forward pass and recomputes them during backward.Source: §F.4 · p. 28

F.5 Answer Equivalence

  • MATH answers are extracted from the final boxed expression, normalized under two conventions, compared numerically within 10^-6 when possible, and otherwise checked symbolically.Symbolic checks accept a zero difference and use a five-second timeout.Source: §F.5 · p. 28

G Temperature Ablation

  • The temperature appendix reports sampled Pass@1 alongside greedy results, using sampling as an empirical analogue of verifier risk while explaining why the main paper emphasizes greedy results.The appendix describes three experiments and their decoding protocols.Source: G Temperature Ablation · p. 29

G.1 Why Sampled Evaluation is Closer to the Theory

Sampled Pass@1 more directly reflects probability mass outside a policy’s argmax output than greedy Pass@1 does. The appendix compares the two evaluation modes, explains their trade-offs, and introduces a Renyi-order sweep.

G.1 Why Sampled Evaluation is Closer to the Theory

  • Sampling k completions estimates the accepted fraction under the decoded policy, whereas greedy Pass@1 measures one argmax trajectory and can miss large differences in verifier risk.At T = 0.8, sampling estimates risk for the tempered policy rather than the original policy; T = 1 recovers the latter exactly.Source: §G.1 · p. 29

G.3 Results

  • The sampled-evaluation gains are larger than the greedy gains on all three benchmarks, consistent with sampling measuring changes beyond the argmax trajectory.The figure captions identify GSM8K, MBPP, and MATH Pass@1 curves across entropy-penalty strengths.Source: §G.1 · p. 29, Figure 4 · p. 30, Figure 5 · p. 31

G.2 Protocol

  • The sampled protocol draws from the full tempered softmax at T = 0.8, using k = 8 for GSM8K and k = 4 for MBPP and MATH.Top-k and top-p truncation are disabled because truncation discards the distribution tail being studied.Source: §G.2 · p. 29

G.4 Why the Main Paper Reports Greedy Pass@1

  • Greedy Pass@1 exceeds sampled Pass@1 at every penalty strength on all three benchmarks, and avoids the effective-temperature confound that affects fixed-temperature sampling.Greedy decoding is invariant to monotone temperature rescaling.Source: §G.4 · p. 30

H Renyi Order Ablation

  • The theory points toward small Renyi orders, while the main trained objective uses α = 1; the appendix tests the empirical consequences of that choice.The theory’s PAC guarantee uses orders approaching zero alongside increasing penalty strength and sample size.Source: H Renyi Order Ablation · p. 34

H.1 Objective and Protocol

  • The ablation sweeps α ∈ {0.1, 0.25, 0.5, 1, 2} and λ ∈ {0, 1, 3, 8}, with Shannon entropy at α = 1 recovering the main objective.The sweep uses 200 evaluation problems and three seeds, so its absolute results should not be compared directly with the main-paper tables.Source: §H.1 · p. 34

H.2 Results

The Renyi-order sweep finds similar performance for α from 0.5 to 2, while lower orders can fail. The appendices qualify the theoretical guarantee and test the entropy penalty in an answer-only setting.

H.2 Results

  • Orders α ∈ [0.5, 2] are statistically indistinguishable in the three-seed GSM8K sweep, with the ordering changing across penalty strengths.The largest mean difference within the range is at most 1.2 pooled standard errors.Source: Figure 7 · p. 34, §H.2 · p. 35, Table 11 · p. 35
  • At λ = 8, α = 0.25 reaches 0.1883, below the 0.4717 cross-entropy baseline; at λ = 1, α = 0.1 and 0.25 remain near baseline.The reported values are 0.4700 for α = 0.1 and 0.4733 for α = 0.25 at λ = 1.Source: §H.2 · p. 35, Table 11 · p. 35
  • The small-α failures are not attributed to numerical artifacts: at α = 0.25, tokens outside the top 100 contribute only 0.053, 0.032, and 0.016 at λ = 1, 3, and 8.The text interprets this as evidence that the failure reflects the objective rather than floating-point arithmetic.Source: §H.2 · p. 35

H.3 Reconciliation With Theorem B.4

  • The small-α PAC guarantee requires α to decrease and λ to increase with sample size, but the softmax’s smallest represented probability makes the required sample sizes unreachable in this setting.At fixed finite α and sample size, the theorem does not predict which order performs better.Source: §H.3 · p. 35

H.4 Limitations

  • The order plateau is based on one model, one dataset, and three seeds, so it indicates no detected ordering rather than exact equivalence.For α = 0.1, pre-clipping gradient norms of 7–10 exceeded the clipping threshold of 1.0; the α = 0.25, λ = 8 collapse remains undiagnosed.Source: §H.4 · pp. 35–36

I Answer-Only Negative Control

  • In the answer-only GSM8K condition, accuracy rises slightly through λ = 2 but then falls by 5.60 points at λ = 8, unlike the gains reported for reasoning-format training.The authors do not claim the small increase as an effect after correcting for the sweep’s eight comparisons.Source: I Answer-Only Negative Control · p. 37, Figure 8 · p. 37, Table 12 · p. 38

Related papers

Loading 2609.30572v1…