PaperlayerView exact revision on arXiv

Source-linked AI summary

Characterizing Overconfident Failure in LLM-Based Code Generation

Ravishka Rathnasuriya, Wei Yang

arXiv:2610.11300v1cs.SEcs.AI

TL;DR

Generated code can look plausible yet fail execution, and existing validation may be costly, incomplete, or delayed. The paper evaluates uncertainty and overconfident failure across code models and benchmarks, finding that uncertainty and common mitigations do not reliably identify or resolve failing generations.

  • Problem

    Generated programs can appear plausible while failing execution, and testing or program analysis may be incomplete, costly, or applied only after generation.

    Source: §1 · p. 1

  • Method

    The paper characterizes overconfident failure in code LLMs by studying uncertainty signals, selective generation, mitigation strategies, and latent representations.

    Source: §1 · p. 3

  • Results

    Model-derived uncertainty often fails to identify incorrect programs, even under calibration, sampling, and abstention prompting.

    Source: §12 · p. 18

  • Takeaways & Limitations

    The findings motivate behavior-aware evaluation of high-confidence failures, generate-or-defer behavior, and reliability under limited validation budgets.

    Source: §12 · p. 18

    The study covers four open-source 7–8B code models, single-generation Python synthesis, and execution-based benchmarks, which do not establish complete semantic correctness or cover other development settings.

    Source: §9 · p. 17

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used for automated code generation, but generated programs can appear syntactically plausible while still failing execution-based correctness checks. Existing validation methods, such as testing and program analysis, remain essential but are often incomplete, costly, or applied only after generation. Model-derived uncertainty is therefore a natural early reliability signal. This paper studies the dilemma of overconfidence in code LLMs where incorrect programs are often generated with token-level confidence comparable to correct programs. We study this dilemma across four open-source code models and three execution-based benchmarks. Our analysis begins by investigating whether existing uncertainty metrics provide reliable proxies for execution correctness in code generation. We then characterize overconfidence at both global and local token levels, asking whether incorrect programs remain indistinguishable from correct ones under confidence and entropy summaries, including selective generation and the limits of instruction tuning. Finally, we evaluate whether common mitigation strategies reduce this failure mode. Our study yields four findings. First, existing uncertainty signals provide only partial and model-dependent evidence of execution failure. Second, overconfidence persists at both program and token levels, and uncertainty-based selection does not consistently improve accepted-set accuracy. Third, instruction tuning can increase certainty on failing generations without consistently improving correctness discrimination. Fourth, common mitigation techniques improve specific aspects of reliability but do not reliably resolve overconfident failure. Our exploratory latent analysis suggests that hidden representations may encode correctness-related signals that output confidence does not expose.

1 Introduction

Execution-based validation can be costly, incomplete, or delayed, while model-derived uncertainty offers an early reliability proxy whose connection to code correctness remains uncertain. The paper characterizes overconfident failure and finds that existing signals and common mitigations do not reliably distinguish failing from passing generations.

1 Introduction

  • Generated code can be syntactically plausible yet violate its specification, while execution testing and program analysis may be incomplete, costly, or applied only after generation.These constraints motivate interest in model-intrinsic reliability signals during code generation.Source: §1 · p. 1
  • Prior uncertainty research evaluates estimators and decision rules, but leaves open how reliably uncertainty separates passing from failing generations and where the mismatch arises.The paper examines this reliability limit across generation stages, conditions, and mitigation strategies.Source: §1 · p. 2
  • Across 17 generation-stage and six prefill-stage signals, no uncertainty measure provides stable failure ranking across models and benchmarks.The introduction frames this as partial, model-dependent evidence of execution failure.Source: §1 · p. 2
  • Failing programs often remain highly certain, including in their least-confident token regions, and uncertainty-based selection does not consistently improve accepted-set accuracy.Instruction tuning can increase certainty on failing generations without consistently improving correctness discrimination.Source: §1 · p. 3
  • Calibration improves probability alignment, but calibrated scores add little predictive skill beyond benchmark pass rate; sampling gains diminish, and abstention prompting does not reliably defer incorrect generations.An exploratory latent analysis finds pass–fail differences in latent-state geometry not reflected in output-token confidence.Source: §1 · p. 3

2 Problem Setup and Scope

The section frames code generation as a selective decision problem: a system estimates whether a candidate will pass, although execution correctness is unavailable a priori. It defines the signal sources and the overconfident-failure mismatch between estimated reliability and execution behavior.

2.1 Code Generation as a Selective Decision Problem

  • Generation capability and reliability estimation are distinct: pass@k measures whether generated solutions pass, while a selective system uses a reliability score to decide whether to accept a candidate.The operational correctness signal is the task-specific test suite's pass/fail outcome.Source: §2.1 · p. 3

2.2 Uncertainty-Aware Code Generation

  • Because correctness cannot be known a priori, practical systems use an estimated score as a proxy; overconfident failure occurs when a failing candidate receives a high reliability score.The proxy assumption is that higher-scoring candidates are more likely to pass.Source: §2.2 · p. 4

2.3 Sources of Uncertainty Signals

  • Reliability signals include white-box model information, variation across sampled generations, or external judge estimates; prefill signals precede code tokens, while decoding signals use generated outputs or internal states.Prefill signals cannot observe implementation-specific errors, while decoding signals may reflect fluency or memorized patterns rather than functional correctness.Source: §2.3 · pp. 4–5

2.4 The Problem: Overconfident Failure

  • Overconfident failure is a failing generation in a high-reliability region, without a universal threshold, because score distributions vary across models and metrics.Token confidence concerns decoding decisions, whereas execution evaluates completed-program semantics; passing and failing implementations can have comparable token confidence.Source: §2.4 · p. 5

3 Empirical Overview

The empirical overview describes the study's models, execution-based benchmarks, and research questions. The design spans controlled function-level tasks and a broader benchmark with richer contexts and library use.

3 Empirical Overview

  • The study selects open-source, code-specialized instruction-tuned models in the 7–8B parameter range, enabling access to token-level statistics, prefill-stage signals, and internal representations while reducing scale-attributable variation.Source: §3.1 · p. 5

3.1 Benchmarks

  • Evaluation uses HumanEval+, MBPP+, and BigCodeBench, with the first two providing function-level settings and BigCodeBench covering richer contexts, complex instructions, and library APIs.The benchmarks contain 164, 378, and 1,140 Python tasks, respectively.Source: §3.1 · p. 5

3.2 Research Questions

  • The study investigates whether uncertainty signals reflect correctness, how overconfident failure manifests and persists, whether mitigation reduces it, and whether latent representations encode correctness information.These are framed as four research questions covering uncertainty, persistence, mitigation, and latent representations.Source: §3.2 · p. 6

4 Reliability of Standard Uncertainty Signals for Failure Detection

The section tests whether prefill- and generation-stage uncertainty scores can rank code failures, finding that their failure discrimination is limited and varies across models and benchmarks.

4 Reliability of Standard Uncertainty Signals for Failure Detection

  • The evaluation computes 17 generation-stage and six prefill-stage scores, including six specified prefill scores, and samples up to 10 candidates when metrics require multiple generations.Failure is the positive class for AUC; BigCodeBench is excluded because low pass rates leave too few passing generations for stable discrimination estimates.Source: §4 · p. 6, Table 2 · p. 7
  • Across 17 generation-stage and six prefill-stage signals, no uncertainty measure provides stable failure ranking across models and benchmarks.Generation-stage best AUCs range from 0.598 to 0.696; prefill AUC reaches 0.70 for CodeLlama on HumanEval+, while most settings fall between 0.52 and 0.62.Source: §4 · p. 6, Table 3 · p. 7
  • Uncertainty thresholds should be re-evaluated for each deployment setting because the same signal can provide different failure discrimination across models and workloads.The results do not support a model- and task-independent mapping from standard uncertainty scores to code correctness.Source: §4 · p. 7

5 Characterizing Overconfidence During Code Generation

The section characterizes overconfidence through program- and token-level confidence and entropy, selective generation, and instruction tuning. Failing programs can remain highly certain, while uncertainty-based selection and post-training do not consistently improve correctness discrimination.

5.1 Program-Level Confidence Remains High for Failing Generations

  • On BigCodeBench, CodeLlama and QwenCoder assign higher average confidence to incorrect than correct generations; confidence gaps are small on controlled benchmarks.For CodeLlama, mean confidence is 0.956 for incorrect versus 0.943 for correct generations; for QwenCoder, it is 0.920 versus 0.897.Source: §5.1 · p. 8, Table 4 · p. 8

5.2 Local Token Confidence Provides Limited Failure Evidence

  • Bottom-10% token confidence does not reliably separate correct and incorrect programs, and on BigCodeBench it is higher for incorrect generations from CodeLlama and QwenCoder.On MBPP+, the difference is weak for several models and statistically insignificant for OpenCoder.Source: §5.2 · pp. 8–9

5.3 Predictive Entropy Provides Limited Pass-Fail Separation

  • For OpenCoder on MBPP+, average and top-10% entropy distributions substantially overlap, so entropy does not reliably identify failing programs.Incorrect outputs have marginally higher mean entropy, but the distributions are not clearly separable.Source: Figure 2 · p. 9, §5.3 · p. 9

5.4 Uncertainty-Based Selective Generation

  • Uncertainty-based filtering improves accepted-set accuracy in some settings but can reduce it, including when the most-certain BigCodeBench generations contain no passing solutions.On MBPP+, CodeLlama confidence rises from 46.0% at 100% coverage to 65.79% at 10%; for OpenCoder, it falls from 71.2% to 71.05% at 10% coverage.Source: Figure 3 · p. 9, §5.4 · p. 10

5.5 Instruction Tuning Increases Certainty in Failing Generations

  • Instruction tuning increases certainty on failing generations, but its effect on pass–fail discrimination differs by model and is not consistently positive.For OpenCoder, confidence AUC changes from 0.5230 to 0.5094; for DeepSeekCoder, confidence AUC rises from 0.4391 to 0.5654, while remaining below 0.57.Source: Figure 4 · p. 10, §5.5 · p. 11

5.6 Implications of Overconfidence Characterization

  • Reliability evaluation should assess functional accuracy separately from uncertainty-based correctness discrimination, including after post-training and model updates.Higher Pass@1 does not necessarily improve identification of correct generations, and post-training can alter both uncertainty magnitude and score ordering.Source: §5.6 · pp. 11–12

6 Limits of Mitigating Overconfident Failure

The section evaluates calibration, repeated sampling, and abstention prompting as responses to overconfident failure. Each improves a specific aspect of reliability, but none reliably resolves the problem.

6.1 Calibration Improves Probability Alignment but Provides Limited Predictive Skill

  • Post-hoc calibration improves probability alignment but provides little predictive skill beyond benchmark pass rates and does not reliably distinguish passing from failing generations.On MBPP+, Platt scaling reduces CodeLlama’s average token-confidence Brier score from 0.459 to 0.248 while Brier Skill Score remains approximately 0.001; on BigCodeBench, it reduces that score from 0.856 to 0.061 with Brier Skill Score approximately 0.001.Source: §6.1 · pp. 12–13, Table 5 · p. 13

6.2 Reliability Under Finite Sampling Budgets

  • Repeated sampling quickly reduces unsolved tasks at small budgets, but the gains flatten and many tasks remain unsolved after 50 attempts.Roughly 90 tasks remain for CodeLlama, about 57 for DeepSeekCoder, and around 45–50 for OpenCoder and QwenCoder; sampling expands candidates but does not solve candidate selection or persistent failure.Source: §6.2 · pp. 13–14, Figure 5 · p. 14

6.3 Explicit Abstention Prompting Does Not Reliably Induce Deferral

  • Explicit abstention prompting does not reliably induce deferral; models can still generate incorrect code when an “I don’t know” response is allowed.The penalty-aware prompt instructed OpenCoder to answer only above 95% confidence, yet it produced code for a task asking for the difference of two squares instead of deferring.Source: §6.3 · p. 14, Figure 6 · p. 15

6.4 Implications

  • Generation quality and correctness-sensitive confidence are distinct reliability properties, so improved pass rates should not be treated as evidence of reduced overconfidence.The paper frames correctness-sensitive confidence as an explicit learning objective: reduce confidence on incorrect generations while preserving it on correct ones.Source: §6.4 · pp. 14–15

7 Latent-Space Evidence of Overconfident Failure

The latent-space analysis finds pass–fail differences in hidden-state geometry for CodeLlama and QwenCoder on HumanEval+, despite unreliable output confidence. The authors suggest this may indicate correctness-related signals that token probabilities do not expose.

7 Latent-Space Evidence of Overconfident Failure

  • The analysis computes effective rank from the normalized singular-value spectrum of hidden states over generated solution tokens.Higher effective rank indicates a more dispersed representation; lower effective rank indicates a more compact one.Source: §7 · p. 15
  • For CodeLlama and QwenCoder on HumanEval+, failing generations have higher effective rank than passing generations across many middle and later layers.The authors interpret higher effective rank as less compact hidden-state geometry and suggest that correctness-related variation may be encoded internally without appearing in token probabilities.Source: §7 · p. 15

8 Discussion

The discussion proposes training, inference-time control, and reliability evaluation as distinct responses to overconfident failure. It argues that reliability depends on reducing false acceptance without indiscriminate deferral or unbounded validation.

8 Discussion

  • Post-training should recognize likely failures as well as generate correct code, since instruction tuning increases certainty on failures without consistently improving pass–fail discrimination.Execution-grounded data linking tasks and candidates to outcomes, especially matched passing and failing candidates, could support objectives that penalize confident failures and reward deferral; latent representations may also distinguish outcomes when output confidence overlaps.Source: §8 · p. 16
  • Inference-time systems should use confidence alongside additional evidence, rather than map confidence directly to acceptance or deferral.Evidence may come from developer clarification, retrieval or verification, testing, analysis, or an independent evaluator; repeated generation alone leaves persistent failures.Source: §8 · p. 16
  • Reliability evaluation should test system decisions, not only score alignment, including whether policies reduce false acceptance under varying costs and validation budgets.Calibration can substantially reduce BS while BSS remains near zero in many settings, and explicit IDK prompting does not reliably induce deferral.Source: §8 · p. 16
  • Reliability interventions should reduce failing programs eligible for acceptance without relying on indiscriminate deferral or unbounded validation.The three directions address failure recognition, responses to evidence, and whether those responses reduce false acceptance.Source: §8 · pp. 16–17

9 Limitations

The study's conclusions are bounded by its model, task, and benchmark scope.

9 Limitations

  • The study evaluates four open-source code LLMs in the 7–8B range; extension to larger proprietary models remains open.The analyses require token probabilities and hidden representations.Source: §9 · p. 17
  • The evaluation covers single-generation code synthesis, not repository-scale or agentic workflows where tool use and iterative feedback may change overconfidence.Source: §9 · p. 17
  • Python execution-based test suites provide practical correctness labels but do not establish complete semantic correctness or cover other languages and development settings.Source: §9 · p. 17

10 Threats to Validity

The paper considers internal, external, construct, and conclusion validity, and limits its conclusion to the evaluated model-derived signals and settings.

10 Threats to Validity

  • Metric implementation details, including token filtering, sequence truncation, and score orientation, may affect individual values despite shared pipelines and standard definitions.Source: §10 · p. 17
  • Findings may not generalize to larger proprietary models, project-level tasks, other languages, or agentic workflows; token-probability and hidden-state metrics assume white-box access.Source: §10 · p. 17
  • Benchmark test outcomes approximate functional correctness rather than proving semantic equivalence, while uncertainty and related signals are proxies for reliability evidence.Source: §10 · p. 17
  • Exact effect sizes may vary with decoding parameters, calibration splits, thresholds, and benchmark composition, although trends span multiple models and settings.Source: §10 · p. 17
  • The evaluated model-derived signals are insufficient as standalone mechanisms for deciding whether generated code is likely to pass execution-based validation.The authors avoid claiming that uncertainty is useless.Source: §10 · pp. 17–18

11 Related Work

The related work distinguishes uncertainty for generated code from uncertainty in classification and natural-language generation, and positions this study around execution-based correctness and reliability decisions.

11 Related Work

  • Code correctness depends on executable behavior, unlike classification labels or textual and subjective response properties used in other uncertainty settings.Source: §11 · p. 18
  • Prior code-model work studies uncertainty, calibration, and reliability signals; this paper instead asks whether uncertainty supports decisions such as identifying failures, abstaining, or seeking validation.Source: §11 · p. 18
  • Sampling may yield a correct candidate without identifying which candidate to trust, and prompt-level abstention does not guarantee safe generate-or-defer behavior.Source: §11 · p. 18
  • The study characterizes overconfident failure as a deployment-facing software-engineering reliability problem and motivates connecting self-assessment to execution-grounded correctness.Source: §11 · p. 18

12 Conclusion

Model-derived uncertainty often fails to identify incorrect programs, even under several mitigation approaches. The authors call for behavior-aware evaluation focused on high-confidence failures, generate-or-defer behavior, and reliability under limited validation budgets.

12 Conclusion

  • Model-derived uncertainty often fails to identify incorrect programs, even under calibration, sampling, and abstention prompting.Source: §12 · p. 18
  • Behavior-aware evaluation should measure high-confidence failures and generate-or-defer behavior.Source: §12 · p. 18
  • Reliability should also be evaluated under limited validation budgets.Source: §12 · p. 18
Loading 2610.11300v1…