PaperlayerView exact revision on arXiv

Source-linked AI summary

When Uncertainty Isn't Enough: An Empirical Study of Self-Correction in Code Generation

Pranav Rakasi, Maanas Lalwani, Arnav Srivastava, Arya Palanivel, Tinuade Adeleke, Ruizhe Li, Sean Wu

arXiv:2608.14659v1cs.AIcs.LGcs.SE

TL;DR

Code-generation failures can be difficult to predict, and it is unclear whether natural-language uncertainty methods transfer to code or support self-correction. The study evaluates uncertainty estimators and correction policies, finding that uncertainty-driven correction often degrades Pass@1 while verification-based correction reliably improves accuracy.

  • Problem

    The study asks whether natural-language uncertainty estimation transfers to code generation and whether uncertainty signals can improve code generation through selective self-correction.

    Source: §1 · p. 2

  • Method

    The study compares five uncertainty estimators for code generation and tests uncertainty scores as signals for selective self-correction.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Across both benchmarks, uncertainty-based self-correction degraded Pass@1 in 5 of 6 configurations, while verification-based self-correction was the only reliably beneficial policy.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Uncertainty signals and execution verification are complementary, with uncertainty estimates serving as gates for verification rather than substitutes for it.

    Source: §4.2 · p. 6

    Experiments use only small 1.3B–3B models, and larger models may have different uncertainty profiles and benefit differently from self-correction.

    Source: Limitations and Future Work · p. 6

Abstract

from arXiv · show

Large language models for code generation often produce incorrect solutions without reliable indicators of failure. We study whether uncertainty estimation methods developed for natural language transfer to code generation, and whether such signals can improve code generation via selective self-correction. We evaluate five uncertainty methods: mean token entropy, verbalized confidence, $P(\text{True})$, entropy ensembles, and semantic entropy probes, across three small code LLMs on HumanEval and BigCodeBench. We find that multi-sample $P(\text{True})$ achieves the strongest correlation with correctness, while all the other methods, including semantic entropy probes, yield only weak correlation. We then use these uncertainty signals to drive three self-correction policies: adaptive decoding, uncertainty-based regeneration, and verification-based regeneration. Our results reveal a stronger negative finding than anticipated: uncertainty-based self-correction fails to reliably improve Pass@1, degrading accuracy in 5 of 6 configurations across both benchmarks ($-3$pp to $-10$pp), and adaptive decoding degrades accuracy in 4 of 6 configurations. Only verification-based self-correction reliably improves Pass@1, with gains of $+6$ to $+26$ percentage points on HumanEval and $+8$ to $+20$ percentage points on BigCodeBench, scaling inversely with baseline strength. These findings replicate consistently across both benchmarks and suggest that cheap uncertainty estimators are insufficient on their own to improve code correctness, and that their practical value lies in serving as gating signals for costlier execution-based correction loops rather than as standalone substitutes for verification.

1. Introduction

The introduction asks whether natural-language uncertainty methods transfer to code and whether they can improve coding assistants cheaply. It reports weak standalone signals and unreliable uncertainty-driven correction, while verification-based correction is reliably beneficial.

Introduction

  • Code failures can be unpredictable, and code-specific semantics—where one wrong token can cause failure but distinct programs can be equivalent—motivate testing whether natural-language uncertainty methods transfer.The proposed assistant would assess confidence and revise or regenerate high-uncertainty answers before users receive them.Source: §1 · pp. 1–2
  • Uncertainty estimates are proposed as cheaper correctness proxies than compiling and running tests when latency or resources matter, including real-time suggestions and prioritizing batch changes for review.Source: §1 · pp. 1–2
  • Multi-sample P(True) alone achieved strong correctness correlation at substantially higher cost; single-pass estimators were weak, and verbalized confidence was unreliable for small models.The weak single-pass methods included semantic entropy probes.Source: §1 · p. 2
  • The framework combines three uncertainty-driven correction policies with full-function regeneration (SLT) and proactive regeneration (TBG), which gates decoding using pregeneration hidden states.TBG gates the decoding strategy before any tokens are committed.Source: §1 · p. 2
  • 5 of 6 configurations saw degraded Pass@1 from uncertainty-based self-correction, while adaptive decoding degraded accuracy in 4 of 6 configurations.These results challenge using weak uncertainty signals as standalone correction triggers.Source: §1 · p. 2
  • Verification-based self-correction was the only reliably beneficial policy, suggesting uncertainty should trigger execution feedback rather than replace it.Source: §1 · p. 2

2. Background and Related Works

Prior work on LLM uncertainty spans decoding-probability, verbalized-confidence, and ensemble approaches; code-generation studies have used uncertainty for abstention or fine-grained decoding, while this work applies it to generation-level self-correction.

Uncertainty Estimation in LLMs

  • MTE aggregates token-distribution entropy, whereas semantic entropy and embedding-dispersion methods target meaning-level variation but require multiple generations (Fomicheva et al., 2020; Li et al., 2025).Kossen et al. (2024) train hidden-state linear probes to approximate semantic entropy from one generation, with nearly zero overhead and comparable performance.Source: §2 · pp. 2–3
  • Verbalized confidence needs no hidden states, but calibration findings conflict; prompt design dominates, and small LLM scores can be nearly uninformative (Yang et al., 2024).Some studies report reasonable calibration (Tian et al., 2023; Lin et al., 2022), while others find overconfidence (Xiong et al., 2023; Kadavath et al., 2022); correctness-probability self-evaluation is called P(True).Source: §2 · p. 3
  • Bayesian Prompt Ensembles combine semantically equivalent prompts through Bayesian variational inference, improving calibration in zero- and few-shot classification but requiring multiple calls.This work adapts the method with cost-controlling design choices and task-appropriate uncertainty targets for code generation.Source: §2 · p. 3

Uncertainty Estimation for Code Generation

  • Prior code work uses uncertainty for abstention or token- and line-level decoding decisions, whereas this study uses whole-generation semantic uncertainty to control higher-level self-correction with minimal added latency.Sharma & David (2025) report weak negative uncertainty-correctness correlation but near-zero incorrect outputs with abstention; Zhu et al. (2025) report +6.1% on MHPP, and He et al. (2025) report 15.5% gains on HumanEval/MBPP.Source: §2 · p. 3

3. Methodology

The methodology evaluates uncertainty scores for predicting code failure and uses them to trigger selective self-correction. It compares token-, confidence-, sampling-, prompt-, and representation-based signals alongside two regeneration policies.

3.4. Evaluation

  • The evaluation covers HumanEval and BigCodeBench with three approximately 3B instruct models, measuring uncertainty by Pearson correlation and calibration plots and correction by pass@k and wall-clock latency.Each method produces a scalar uncertainty score per problem, and code correctness is assessed with the verifiers library.Source: §3.2 · p. 3, §3.4 · p. 5

3.2.1. MEAN TOKEN ENTROPY

  • Mean Token Entropy averages the per-token entropy of vocabulary probabilities during generation, with higher values indicating greater uncertainty.The measure is reported in nats.Source: §3.2.1 · p. 3

3.2.2. VERBALIZED CONFIDENCE SCORES

  • Verbalized confidence prompts the model to provide a 0–1 self-assessed probability that its generated Python solution will pass all tests.The confidence value is extracted by pattern-based parsing and clamped to the valid range.Source: §3.2.2 · p. 3

3.2.3. P(TRUE)

  • Verifier-based P(True) estimates correctness as the empirical success probability across K independently sampled candidate solutions evaluated by an external verifier.The verifier returns a binary pass/fail outcome, distinguishing this observable correctness estimate from self-reported confidence.Source: §3.2.3 · p. 4

3.2.4. ENTROPY ENSEMBLES VIA PROMPT DIVERSITY

  • Entropy ensembles generate one completion per semantically equivalent prompt at temperature 0.9, then use average and variance of MTE across prompts to predict failure.A lightweight logistic regression trained on validation data maps the prompt-level statistics to an uncertainty score.Source: §3.2.4 · p. 4

3.2.5. SEMANTIC ENTROPY PROBES (SEPS)

  • Semantic Entropy Probes cluster multiple solutions by functional behavior and train hidden-state classifiers to predict high or low semantic entropy.The probes use either the second-to-last token (SLT) or token-before-generation (TBG) representation; at test time, one hidden state produces the uncertainty score.Source: §3.2.5 · p. 4

3.3.1. FULL FUNCTION REGENERATION

  • Full-function regeneration uses an SLT uncertainty threshold or test verification to decide whether to regenerate, with each policy capped at five attempts.Uncertainty-based correction returns the attempt with lowest uncertainty if the threshold is never met; verification-based correction returns a passing result when found. Thresholds are selected per model on HumanEval validation data by optimizing F1.Source: §3.3.1 · pp. 4–5

3.3.2. PROACTIVE REGENERATION

  • Proactive regeneration uses a TBG probe after one generated token to choose adaptive decoding or standard greedy decoding for the remainder.The method avoids full regeneration when estimated uncertainty is low.Source: §3.3.2 · p. 5

4. Results and Discussion

Across HumanEval and BigCodeBench, verification-based self-correction is the only policy that reliably improves Pass@1; uncertainty-driven policies are unreliable or harmful. P(True) is the strongest uncertainty signal, but cheap signals alone do not reliably guide correction.

4.1. Uncertainty Evaluations

  • P(True) had the strongest correctness correlations across models (0.842, 0.782, 0.303 for Llama, Qwen, and DeepSeek), while other methods had weak negative correlations (−0.111 to −0.631).Semantic entropy probes were in the same weak regime as entropy-based baselines (−0.196 to −0.475).Source: §4.1 · p. 5, Table 1 · p. 6
  • Multi-sample P(True) requires K stochastic rollouts, while single-pass alternatives are an order of magnitude cheaper but correlate substantially less with correctness.Semantic entropy probes were chosen for correction experiments to test whether a cheap, imperfect signal could drive useful behavior; results instead support using uncertainty as a gate for execution-based correction, not as a substitute for it.Source: §4.1 · p. 5, §4.2 · p. 6

4.2. Self-Correction Using Uncertainty

  • Proactive regeneration (TBG) gates decoding from a single pregeneration hidden state, but its adaptive decoding incurred 2.8× to 10.1× baseline latency.In two of three models, this exceeded the cost of full-function verification self-correction (SLT).Source: §4.2 · pp. 5–6
  • Adaptive decoding degraded Pass@1 in five of six HumanEval configurations (−3.23pp to −6.45pp) and mostly degraded it on BigCodeBench (−4.52pp to +0.65pp).The HumanEval exception was Llama-3.2-3B under TBG gating (+3.23pp); transfer of thresholds calibrated on BigCode to HumanEval suggests sensitivity to dataset differences.Source: §4.2 · pp. 5–6
  • Uncertainty-based self-correction degraded Pass@1 in every HumanEval configuration (−3.23pp to −9.68pp) and was unreliable on BigCodeBench (−3.87pp to +5.81pp).The sole BigCodeBench improvement was for the weakest baseline; on HumanEval, uncertainty-triggered resampling could replace correct greedy outputs with incorrect stochastic ones.Source: §4.2 · pp. 5–6
  • Verification-based self-correction improved Pass@1 in every configuration: +6.45pp (0.38 to 0.45) to +25.81pp (0.61 to 0.87) on HumanEval, and +7.74 to +20.00pp on BigCodeBench.The largest gains were for DeepSeek-Coder-1.3B on both benchmarks, consistent with weaker baselines having more headroom.Source: §4.2 · pp. 5–6

Limitations and Future Work

The study is limited to small models and two benchmarks, while real-world code generation poses harder uncertainty and verification challenges. Future work will scale evaluations and explore hybrid correction strategies and calibration under distribution shift.

Limitations and future work

  • All experiments use 1.3B–3B models, and larger models may have different uncertainty profiles and respond differently to self-correction.The study plans to scale experiments to larger models and more diverse benchmarks.Source: Limitations and Future Work · p. 6
  • Real-world code generation involves longer programs, partial specifications, and incomplete tests, making uncertainty estimation and verification more challenging and valuable.The study evaluates on HumanEval and BigCodeBench; future work will use more diverse benchmarks.Source: Limitations and Future Work · p. 6
  • Proactive regeneration currently supports only adaptive decoding; future work will combine proactive uncertainty gating with selective verification and study calibration under distribution shift across languages and task complexities.Integrating proactive regeneration with verification-based correction remains unexplored.Source: Limitations and Future Work · p. 6

A. Appendix A · A.1. Uncertainty Analysis Details

Appendix A.1 examines how uncertainty relates to correctness across models: some profiles separate correct from incorrect generations, while others show substantial overlap or require threshold calibration.

A.1. Uncertainty Analysis Details

  • Figure 2 reports 106 correct generations peaking at 0.10–0.12 nats and 58 incorrect generations with a broader, overlapping distribution.This Mean Token Entropy profile requires careful threshold calibration.Source: Figure 2 · p. 10, §A.1 · p. 10
  • Figure 3 shows Llama uncertainty overconfidence on incorrect predictions, with substantial overlap at low uncertainty.The caption links this pattern to mixed adaptive-decoding results.Source: Figure 3 · p. 10
  • Figure 4 shows Qwen correct generations concentrated at low uncertainty, alongside just 25 incorrect samples among 164 generations.The caption describes this as a well-calibrated uncertainty profile supporting effective self-correction strategies.Source: Figure 4 · p. 11
  • Figure 5 shows Qwen2.5-3B-Instruct P(True) assigning high confidence to 77 correct samples and low confidence to most of 23 incorrect samples.The analysis uses pass@3 verification and indicates well-calibrated estimates.Source: Figure 5 · p. 11
  • Figure 6 shows Llama-3.2-3B-Instruct P(True) assigning consistently higher confidence to 55 correct generations than to 45 incorrect ones.The caption characterizes this as effective uncertainty separation under pass@3 verification.Source: Figure 6 · p. 11
  • Figure 7 reports only 6 correct and 94 incorrect DeepSeek-R1-Distill-Qwen-1.5B generations, while P(True) assigns near-zero confidence to most incorrect solutions.Despite low overall accuracy, the verifier provides informative uncertainty estimates using pass@3 verification.Source: Figure 7 · p. 12
  • Figure 8 shows Llama-3.2-3B-Instruct correct and incorrect generations overlapping at low predicted failure probability, limiting threshold-based self-correction.The caption identifies the estimates as overconfident and poorly separable.Source: Figure 8 · p. 12
  • Figure 9 shows DeepSeek-Coder-1.3B-Instruct correct generations at lower uncertainty, while incorrect outputs have a broader, overlapping distribution.As in Figure 2, careful threshold calibration is needed.Source: Figure 9 · p. 12

A.2. Prompt Templates

The appendix documents prompts for uncertainty estimation methods whose wording can influence model behavior, including code generation, self-evaluation, and optional uncertainty-guided regeneration.

A.2.1. MEAN TOKEN ENTROPY (MTE)

  • MTE reused baseline greedy-decoding prompts, with strict chat formatting for Llama and DeepSeek and plain-text formatting for Qwen.The chat instruction requested valid Python code only; both formats appended a code-start marker to the HumanEval problem.Source: §A.2.1 · p. 13

A.2.2. VERBALIZED CONFIDENCE PROMPTS

  • Verbalized confidence asked whether a candidate would pass all tests and required a response from 0 to 100.The self-evaluator prompt defined 0 as certainty the code was wrong and 100 as certainty it was correct.Source: §A.2.2 · p. 13
  • Verbalized-confidence generations used a Python coding-assistant instruction, while the user prompt supplied the HumanEval problem and function signature.Candidates were evaluated by executing the task prompt, generated code, and benchmark tests together under a timeout.Source: §A.2.2 · p. 13

A.2.3. P(TRUE) PROMPTS

  • P(True) prompted the model to judge whether a candidate passed all tests, requiring exactly True or False; benchmark tests were run under a timeout.The evaluation prompt included the HumanEval problem and candidate solution.Source: §A.2.3 · p. 14
  • Optional P(True)-guided regeneration appended “The previous attempt failed. Try a different approach” to the original HumanEval prompt.The regenerated program was externally verified, and a new verifier-based P(True) estimate was obtained from stochastic rollouts.Source: §A.2.3 · p. 14

B. APPENDIX B · B.1. Probe Design Choices

Semantic entropy probes use design choices intended to improve uncertainty estimation accuracy. The appendix describes multi-layer features, a nonlinear classifier, feature standardization, and model-specific thresholds.

B.1. Probe Design Choices

  • The reported probe design choices are intended to improve uncertainty estimation accuracy.Source: §B.1 · p. 14
  • SEP concatenates features from the last three transformer layers into 6144–9216 dimensional vectors.The design draws on syntactic information from earlier layers and semantic information from later layers.Source: §B.1 · p. 14
  • Multi-layer feature concatenation improves probe accuracy compared with single-layer features.Source: §B.1 · p. 14
  • SEP uses a 3-layer MLP classifier (256 →128 →64) with early stopping to prevent overfitting.The reported test AUROC is 0.62–0.89.Source: §B.1 · p. 14
  • SEP standardizes features so that all layers contribute equally.Source: §B.1 · p. 14
  • Model-specific F1-optimized thresholds of 0.30–0.70 account for differing uncertainty distributions and improve precision-recall trade-offs.Source: §B.1 · p. 14

B.2. Threshold Tuning

The section describes F1-based threshold tuning for uncertainty-guided decoding and reports that the selected thresholds vary by model and feature method.

B.2. Threshold Tuning

  • Candidate thresholds from 0.3 to 0.7 are evaluated on held-out test data, and the threshold maximizing F1 is selected for each model and feature method.The procedure also measures precision, recall, accuracy, and trigger rate; F1 balances avoiding unnecessary corrections with capturing uncertain generations.Source: §B.2 · pp. 14–15
  • Qwen2.5-Coder-3B-Instruct with SLT uses threshold 0.7 (F1: 0.96, accuracy: 0.96), while DeepSeek-Coder-1.3B-Instruct with TBG uses 0.3 (F1: 0.78, accuracy: 0.80).The F1-optimized thresholds outperform a naive median-based approach and are automatically loaded during inference.Source: §B.2 · p. 15
Loading 2608.14659v1…