PaperlayerView exact revision on arXiv

Source-linked AI summary

Calibration and Correctness of Language Models for Code

Claudio Spiess, David Gros, Kunal Suresh Pai, Michael Pradel, Md Rafiqul Islam Rabin, Amin Alipour, Susmit Jha, Prem Devanbu, Toufique Ahmed

arXiv:2402.02047v4cs.SEcs.LG

TL;DR

Generated code can be buggy or vulnerable, making reliable confidence signals important for deciding how much to trust and review it. The paper evaluates calibration across code-generation settings and tests improvement approaches; models are generally poorly calibrated out of the box, though few-shot reflection improves a completion setting.

  • Problem

    Generated code can be buggy or vulnerable, creating a need for confidence signals that help users assess whether code is correct.

    Source: §I · p. 1, §VIII · p. 11

  • Method

    The paper studies intrinsic and reflective calibration across code-generation settings and evaluates correctness using exact matches and test outcomes.

    Source: A. Code Correctness · p. 3, §VIII · p. 11

  • Results

    Models are generally poorly calibrated out of the box; reflective few-shotting for GPT-3.5 completion reached a Brier skill score of 0.15, particularly with retrieval-augmented few-shotting.

    Source: §VIII · p. 11

  • Takeaways & Limitations

    Improved calibration could help manage the risks and benefits of LLM-generated code.

    Source: §V · p. 10

    HumanEval and Defects4J have only 164 and 120 samples, respectively, so findings there may not generalize; larger, more natural synthesis and repair datasets are recommended.

    Source: §VI · p. 10

Abstract

from arXiv · show

Machine learning models are widely used, but can also often be wrong. Users would benefit from a reliable indication of whether a given output from a given model should be trusted, so a rational decision can be made whether to use the output or not. For example, outputs can be associated with a confidence measure; if this confidence measure is strongly associated with likelihood of correctness, then the model is said to be well-calibrated. A well-calibrated confidence measure can serve as a basis for rational, graduated decision-making on how much review and care is needed when using generated code. Calibration has so far been studied in mostly non-generative (e.g. classification) settings, especially in software engineering. However, generated code can quite often be wrong: Given generated code, developers must decide whether to use directly, use after varying intensity of careful review, or discard model-generated code. Thus, calibration is vital in generative settings. We make several contributions. We develop a framework for evaluating the calibration of code-generating models. We consider several tasks, correctness criteria, datasets, and approaches, and find that, by and large, generative code models we test are not well-calibrated out of the box. We then show how calibration can be improved using standard methods, such as Platt scaling. Since Platt scaling relies on the prior availability of correctness data, we evaluate the applicability and generalizability of Platt scaling in software engineering, discuss settings where it has good potential for practical use, and settings where it does not. Our contributions will lead to better-calibrated decision-making in the current use of code generated by language models, and offers a framework for future research to further improve calibration methods for generative models in software engineering.

I. INTRODUCTION

LLM code suggestions can be buggy or vulnerable, so developers need confidence signals aligned with correctness to guide graduated review. This study evaluates calibration across code-generation tasks and tests methods for improving it.

I. INTRODUCTION

  • A well-calibrated confidence signal could help developers manage code correctness risks through graduated quality-control decisions.Source: §I · pp. 1–2
  • The study evaluates a calibration framework across code synthesis, completion, and repair, using exact-match and test-based correctness criteria.Source: §I · p. 2
  • ECE ranged from 0.09 to 0.73 across settings, indicating that intrinsic LLM confidence was a poor predictor of code correctness.Source: §I · p. 2
  • Confidence rescaling generally improved calibration, while reflective methods worked inconsistently across settings.Source: §I · p. 2
  • For GPT-3.5 code completion, reflective few-shotting raised the Brier skill score from 0 to 0.15.Source: §I · p. 2
  • The paper identifies BM25-aided few-shotting as a potentially practical approach for the widely used task of code completion.Source: §I · p. 2

A. Research Agenda

The research agenda asks whether confidence tracks code correctness and whether rescaling, reflection, or few-shotting can improve that alignment. It treats calibration as agreement between stated confidence and empirical correctness frequency.

A. Research Agenda

  • The agenda evaluates confidence alignment for synthesis, completion, and repair using exact-match and test-passing correctness criteria.Source: A. Research Agenda · p. 2
  • Intrinsic confidence was generally poorly aligned with correctness, motivating investigation of calibration improvements.Source: A. Research Agenda · p. 2
  • The rescaling question includes a practical constraint: fitting its parameters requires correctness data.Source: A. Research Agenda · p. 2
  • The agenda tests whether reflective confidence helps and whether few-shotting improves calibration for instruction-tuned code completion.Source: A. Research Agenda · p. 2

A. Calibration

  • Calibration means that outputs assigned confidence p are correct at empirical frequency p, supporting confidence-sensitive decisions such as choosing a hat or umbrella.Source: A. Calibration · pp. 2–3

B. Why Calibration Matters for Code

Calibration can help developers use confidence signals to manage code-quality risks through more nuanced review decisions. The methodology evaluates three generative tasks, correctness by tests, and intrinsic and reflective confidence measures.

B. Why Calibration Matters for Code

  • Well-calibrated confidence can support graduated quality control, while poor calibration risks both rejecting likely-correct code and accepting likely-incorrect code.Source: B. Why Calibration Matters for Code · p. 3

III. RESEARCH METHODOLOGY

  • The methodology covers function synthesis, line-level completion, and program repair, evaluating correctness with hidden tests rather than placing tests in prompts.Source: §III · p. 3, B. Confidence Measures · p. 4

A. Code Correctness

  • The study prefers test-passing correctness over exact match because semantically equivalent code may differ, while test suites can be incomplete, incorrect, or flaky.Source: A. Code Correctness · p. 3

B. Confidence Measures

  • Confidence measures include intrinsic token and sequence probabilities, reflective self-evaluation and QA logits, plus generated-sequence length as a baseline.Source: B. Confidence Measures · p. 3
  • For QA-logit measures, the study also normalizes probability mass across TRUE and FALSE expressions to estimate confidence.Source: B. Confidence Measures · p. 3

C. Measures of Calibration

Calibration is assessed by comparing confidence with observed correctness across confidence buckets. ECE summarizes calibration deviations, while the Brier score also captures discrimination and avoids one misleading ECE case.

C. Measures of Calibration

  • Reliability plots group predictions into confidence buckets and compare each bucket’s observed correctness with its average predicted confidence.Bars closer to the diagonal indicate better calibration.Source: C. Measures of Calibration · p. 4
  • CODEX is well-calibrated and generally correct for token-level completion, while GPT-3.5 verbalized confidence is poorly calibrated for line completion.Calibration may differ when correctness is measured beyond the token-level training objective.Source: C. Measures of Calibration · p. 4
  • ECE is the weighted average of bucket-level confidence–correctness deviations, but a base-rate predictor can yield a deceptively low ECE.The paper defines bucket correctness as the fraction of correct predictions and bucket confidence as their average estimated probability.Source: C. Measures of Calibration · p. 4
  • The Brier score measures both discrimination and calibration, whereas ECE measures calibration alone and can be affected by binning.A Brier score near zero is possible when correct outputs receive probabilities near 1 and incorrect outputs near 0.Source: C. Measures of Calibration · pp. 4–5

D. Rescaling Approaches

The paper evaluates Platt scaling and other rescaling approaches for calibration. It uses five-fold fitting to reduce overfitting risk, while noting that more parameterized alternatives may be risky with limited data.

D. Rescaling Approaches

  • Platt scaling fits logistic regression to confidence logits, learning a multiplier and intercept to rescale probabilities.The paper reports that it improves calibration, with caveats.Source: D. Rescaling Approaches · p. 5
  • Five-fold rescaling fits on four-fifths of the data and applies the fit to the held-out fifth to reduce overfitting and skew.The folds are rotated so each combination of four-fifths is used for fitting.Source: D. Rescaling Approaches · p. 5
  • Temperature rescaling lacks Platt scaling’s intercept, while more parameterized alternatives risk overfitting when datasets contain only a few hundred examples.The paper also names histogram binning and isotonic regression as alternatives.Source: D. Rescaling Approaches · p. 5

E. Tasks & Dataset

The evaluation covers function synthesis, line-level completion, and single-line program repair using datasets with different correctness measures. The study also evaluates three code-generation models at temperature zero.

E. Tasks & Dataset

1) Function Synthesis:

  • Function synthesis generates Python functions from docstrings and uses functional testing to determine correctness.HUMANEVAL and MBPP are used, but their largely artificial problems may not generalize to real-world development.Source: 1) Function Synthesis · p. 5
  • For synthesis, MBPP problems are reformatted as docstring prompts and about 75 problems are excluded because their reference solutions fail the provided tests.The synthesis datasets provide a datapoint for calibration, though their artificial problems limit generalization to real-world development.Source: 1) Function Synthesis · p. 5

2) Line-level Code Completion:

  • Line-level completion is evaluated on DYPYBENCH, using both exact-match correctness and test correctness on real-world Python projects.DYPYBENCH contains 50 projects totaling over 2.2 million lines; 1,988 functions were selected with at least three body lines, full test coverage, and a docstring line.Source: 2) Line-level Code Completion · p. 5

3) Program Repair:

  • Program repair uses 120 single-line Defects4J bugs and 3,000 sampled SSTUBS examples; SSTUBS evaluation is exact-match only because it lacks test sets.The study focuses on small, pre-localized single-line bugs.Source: 3) Program Repair · pp. 5–6

F. The Models

  • The study evaluates GPT-3.5, CODEX, and CODEGEN2-16B with temperature zero.For synthesis, temperature zero is described as standard practice for pass@1 when generating one solution.Source: F. The Models · p. 6

IV. RESULTS

Across the evaluated tasks, raw confidence measures generally align poorly with code correctness, with calibration varying by task and correctness criterion. The results also show that test-passing and exact-match capture different aspects of generated code.

1) Overall Correctness:

  • GPT-3.5 performs well on function synthesis and line-level completion, while CODEX generally performs well on program repair.DYPYBENCH represents the widely used code-completion task.Source: 1) Overall Correctness · p. 6

2) Correctness:

  • On DYPYBENCH, about half of test-passing generations are not exact matches, and 6.89% of exact matches fail tests.The authors attribute the latter discrepancy to flaky tests affected by network conditions, execution order, and other environment conditions.Source: 2) Correctness · p. 6

3) Confidence Measures:

4) Calibration without Rescaling:

  • For line completion, ptotal is slightly worse than the baseline and has ECE ∼0.15, while average token probability is overconfident.The average token probability exceeds the roughly 30% overall success rate.Source: 4) Calibration without Rescaling · p. 6
  • Raw intrinsic measures are inconsistent for synthesis and remain below the base rate for both program-repair models.For synthesis, the better-performing intrinsic measure varies by model and dataset; repair findings may not generalize from small DEFECTS4J, and CODEGEN2 has too few correct outputs for reliable confidence estimates.Source: 4) Calibration without Rescaling · pp. 6–7
  • Raw confidence is well calibrated only for exact-match code completion; for test-passing correctness, it is poorly calibrated.The raw baseline is hard to beat, with the best Skill Score around 0.05.Source: 4) Calibration without Rescaling · pp. 6–7

B. RQ 2: Can alignment between LLM confidence in gen-

The results examine Platt scaling as a way to improve calibration across code-generation tasks. Scaling generally improves calibration, but its apparent gains can be misleading and its cross-task performance varies.

erated code, and its correctness, be improved by confidence rescaling?

  • Bucket collapse can produce near-zero ECE by squashing confidence values to the base rate, so Skill Score should be considered alongside ECE and Brier Score.Source: erated code, and its correctness, be… · p. 7
  • ECE improves from an average of 0.32 to 0.03 after Platt scaling, but low ECE can reflect bucket collapse rather than useful confidence.For measures with post-scaling SS of at least 0.05, ECE improves from 0.38 to 0.05; the reliability plots show improved calibration for DYPYBENCH line completion.Source: Figure 2 · p. 7, erated code, and its correctness, be… · p. 7, Table IV · p. 8
  • After rescaling, intrinsic measures improve skill for line completion and function synthesis, but no measure improves program-repair skill.The ptotal measure reaches SS 0.15 on HUMANEVAL; reflective prompts also show improvement, with SS and ECE brought to similar values.Source: erated code, and its correctness, be… · p. 7
  • Rescaling can require over 64 correctness examples for positive skill and lower ECE, and cross-task transfer is viable for similar domains and base rates but can fail badly otherwise.For GPT-3.5, transfer between function-synthesis tasks changed average SS from 0.14 to 0.12 and ECE from 0.07 to 0.05; DYPYBENCH-to-synthesis transfer changed SS from 0.12 to -1.28 and ECE from 0.08 to 0.54.Source: erated code, and its correctness, be… · p. 8
  • Rescaling may offer a 10–20% improvement over baseline rates and good calibration, but results are inconsistent, including poor CODEGEN2 skill on function synthesis.Metric improvements can be misleading when calibration matches the base rate through bucket collapse or fails to generalize.Source: erated code, and its correctness, be… · p. 8

C. RQ 3: Is confidence obtained by reflection better aligned with correctness?

Reflective confidence is inconsistently calibrated and does not reliably outperform intrinsic measures across the tested tasks. For GPT-3.5 line completion, retrieval-based few-shot examples improve reflective calibration, with rescaling producing the strongest reported skill score.

C. RQ 3: Is confidence obtained by reflection better aligned with correctness?

  • The two logit-based reflective measures were usually strongly correlated, while verbalized self-ask confidence had no consistent relationship with them.Source: C. RQ 3: Is confidence obtained by… · p. 8
  • Reflective measures were inconsistently calibrated: some settings showed positive SS and ECE below 0.2, but repair results did not indicate good calibration overall.For repair, GPT-3.5 and Codex showed positive SS and low ECE for pB, but the results were inconsistent after normalization.Source: C. RQ 3: Is confidence obtained by… · pp. 8–9
  • Reflective approaches may be best calibrated without rescaling in some settings, but did not significantly improve over intrinsic measures in the tested settings.The reflective verbalized probability was not well calibrated, and intrinsic and reflective confidence values had no consistent relationship.Source: C. RQ 3: Is confidence obtained by… · pp. 8–9

D. RQ 4: Can we use few-shot techniques to achieve bet-

  • 0.15 skill score was reached for GPT-3.5 line completion with rescaled, BM25-selected few-shot examples, exceeding other DYPYBENCH measures.Randomly selected examples did not improve calibration over baseline pNB; BM25 selection yielded positive SS of 0.08 before rescaling.Source: D. RQ 4: Can we use few-shot… · p. 9
  • BM25-based few-shot reflection improved calibration, but it depends on examples with known correctness labels.The study used five examples, selected randomly or by similarity; BM25 retrieval was more similar to rescaling because it relied on a larger set of ground truths.Source: D. RQ 4: Can we use few-shot… · p. 9
  • The authors identify alternative prompting, fine-tuning, and chain-of-thought as approaches that could alter these findings and merit future study.Source: D. RQ 4: Can we use few-shot… · p. 9

V. DISCUSSION

The discussion presents calibration as a basis for graduated review of generated code and considers extensions beyond binary correctness. The authors also note limits to generalizability from small datasets, selected tasks and models, and prompt choices.

V. DISCUSSION

  • A calibrated confidence measure can support graduated review decisions, unlike a single high-confidence acceptance threshold that may reduce both risk and useful LLM assistance.The example uses GPT-3.5 code completion with test-passing correctness and few-shot average-token-probability confidence; the base correctness rate is about 33%, with skill score 0.15.Source: §V · p. 9
  • The paper proposes extending correctness prediction to multiple defect perspectives, including severity, defect type, and repair complexity.Source: §V · p. 10
  • Using geometric means to normalize for sequence length marginally but consistently improved Brier and skill scores in the authors’ experiments.Source: §V · p. 10

VI. THREATS TO VALIDITY

  • Results may not generalize from HUMANEVAL and DEFECTS4J, which contain 164 and 120 samples, respectively; the authors call for larger, more natural synthesis and repair datasets.Three other datasets have more than 800 samples each, and the study notes that line-level completion has a large, natural dataset.Source: §VI · p. 10
  • The exploration does not cover every software-engineering task, and findings may vary with model selection, prompts, problem phrasing, or dataset realism.HUMANEVAL and MBPP use small Python programs that may not represent real-world development; the authors also note possible flakiness in DyPyBench tests.Source: §VI · p. 10

VII. RELATED WORK

Prior calibration research largely examined non-code tasks or classification, while this study evaluates intrinsic and reflective confidence across software-engineering tasks and datasets. The paper reports generally poor out-of-the-box calibration, some improvement from Platt scaling, and a stronger result for retrieval-augmented few-shot completion.

VII. RELATED WORK

  • Compared with prior natural-language calibration studies using intrinsic measures, this work evaluates three coding tasks with artificial and natural code datasets and intrinsic and reflective measures.Source: §VII · p. 11
  • Related work spans calibration in vision and language models, uncertainty estimation, localized code-edit uncertainty, and trained correctness estimators; some studies use correlation rather than calibration metrics.Source: §VII · p. 11

VIII. CONCLUSION

  • Across several practical settings, models were generally poorly calibrated out of the box, while Platt scaling generally improved calibration somewhat.Source: §VIII · p. 11
  • GPT-3.5 code completion reached a skill score of 0.15 with reflective in-context learning, particularly with retrieval-augmented few-shotting.Source: §VIII · p. 11
  • The authors describe retrieval-augmented few-shotting as encouraging, while noting that most models with simple reflection or no intervention did not provide reliable confidence measures.Source: §VIII · p. 11

APPENDIX

The appendix examines calibration plots, rescaling reuse and sample-size effects, and the sensitivity of verbalized-confidence results to retry failures. Its analyses show that calibration metrics can diverge and that excluding retry-failure values affects unscaled results more than scaled results.

APPENDIX

  • 0.73, 0.71, and 0.73 AUC values accompanied ECE values of 0.11, 0.06, and 0.37 across three GPT-3.5 examples.The examples were scaled HumanEval Ask T/F, scaled MBPP Ask T/F, and nonscaled HumanEval Ask T/F, respectively.Source: Figure A4 · p. 18
  • AUC had a limited relationship with ECE but a strong relationship with scaled skill score across GPT-3.5 confidence measures and tasks.Source: Figure A5 · p. 18
  • Dataset-measure pairs with less data or concentrated values showed greater variation across Platt-scaling curves.The figure compares measurement-to-confidence curves across tasks and measures, showing fold curves alongside a curve fit to all data.Source: Figure A6 · p. 19
  • Reusing a rescaling showed the most potential for datasets with similar tasks and base rates, but could still produce sizable changes in skill score or ECE.The analysis suggests reflective measures may be more robust across rescalings.Source: Figure A7 · p. 20
  • More examples used to fit a Platt rescaling improved skill score and ECE in the bootstrap analysis.Each simulation sampled examples for fitting and applied the rescaling to the remaining data; the figure reports the median and 90% interval across 500 simulations.Source: Figure A8 · p. 21

UNDERSTANDING EFFECTS OF VERBALIZED RETRY FAILURES

  • After excluding all these instances, scaled Skill Score was unchanged on average (mean difference 0.00), while nonscaled results showed mean Skill Score change of -0.11 and mean ECE change of 0.02.The most extreme changes occurred in areas of already poor calibration.Source: UNDERSTANDING EFFECTS OF VERBALIZED… · p. 22
Loading 2402.02047v4…