PaperlayerView exact revision on arXiv

Source-linked AI summary

Calibration of Large Language Models on Code Summarization

Yuvraj Virk, Premkumar Devanbu, Toufique Ahmed

arXiv:2404.19318v3cs.SEcs.CL

TL;DR

Without a human-written reference, it is difficult to judge whether an LLM-generated code summary is sufficiently similar to one a human might write. The paper studies confidence-based calibration for this task, reporting that earlier tokens are better calibrated than later ones and that in-context learning with modern rescaling can improve calibration.

  • Problem

    Without a gold human-written summary, users lack a direct way to judge whether an LLM-generated code summary is sufficiently similar to what a human might write.

    Source: §1 · p. 2, §2.3 · p. 7

  • Method

    The paper treats sufficient similarity to a human-written summary as correctness and examines confidence calibration across similarity thresholds and LLM settings.

    Source: §1 · p. 3, §2.3 · p. 7

  • Results

    The authors report improved calibration with in-context learning and modern rescaling, while BM25-retrieved few-shot summaries have Brier scores of 0.25 to 0.57, worse than random.

    Source: §1 · p. 3, §5.1 · p. 11

  • Takeaways & Limitations

    A confidence signal can help users judge the reliability of a generated summary when considered alongside other summary-quality metrics.

    Source: §1 · pp. 2–3

    Similarity metrics do not explicitly capture developers’ different summary intents, and integrating intent into evaluation metrics and confidence scores remains future work.

    Source: §6.2 · p. 16

Abstract

from arXiv · show

A brief, fluent, and relevant summary can be helpful during program comprehension; however, such a summary does require significant human effort to produce. Often, good summaries are unavailable in software projects, which makes maintenance more difficult. There has been a considerable body of research into automated AI-based methods, using Large Language models (LLMs), to generate summaries of code; there also has been quite a bit of work on ways to measure the performance of such summarization methods, with special attention paid to how closely these AI-generated summaries resemble a summary a human might have produced. Measures such as BERTScore and BLEU have been suggested and evaluated with human-subject studies. However, LLM-generated summaries can be inaccurate, incomplete, etc.: generally, too dissimilar to one that a good developer might write. Given an LLM-generated code summary, how can a user rationally judge if a summary is sufficiently good and reliable? Given just some input source code, and an LLM-generated summary, existing approaches can help judge brevity, fluency and relevance of the summary; however, it's difficult to gauge whether an LLM-generated summary sufficiently resembles what a human might produce, without a "golden" human-produced summary to compare against. We study this resemblance question as calibration problem: given just the code & the summary from an LLM, can we compute a confidence measure, that provides a reliable indication of whether the summary sufficiently resembles what a human would have produced in this situation? We examine this question using several LLMs, for several languages, and in several different settings. Our investigation suggests approaches to provide reliable predictions of the likelihood that an LLM-generated summary would sufficiently resemble a summary a human might write for the same code.

1 Introduction

The introduction argues that code summaries can aid program comprehension but that LLM summaries often fail to match developer-written summaries. It frames calibration as a way to estimate whether a generated summary is acceptably similar to a human-written one, without a reference summary.

1 Introduction

  • 70–80% of the time, LLMs produce summaries humans may not find acceptably similar to human summaries, motivating a way to judge generated-summary quality.The introduction notes that even a small rate of inaccurate or incomplete summaries may matter when summaries are generated at scale and relied upon.Source: §1 · p. 2
  • Raw model confidence may not reliably indicate whether a summary is acceptable when no reference summary is available.The paper asks whether confidence derived directly from model output can provide that indication.Source: §1 · p. 2
  • Calibration compares predicted confidence with actual outcomes, and reliable confidence could help users decide whether to accept, reject, or inspect a summary further.The paper illustrates poor calibration with a high-confidence, low-quality summary.Source: §1 · p. 2
  • The paper frames summary correctness as sufficient similarity to human-written summaries and studies confidence measures intended to predict that correctness.It presents the approach as supporting better decisions about summary reliability.Source: §1 · pp. 2–3
  • The authors introduce calibration for code summarization, examine multiple similarity thresholds, and report improved calibration with in-context learning and modern rescaling techniques.They also report that earlier summary tokens are better calibrated than later tokens, which can be overconfident.Source: §1 · p. 3
  • The authors say their findings can help developers make well-justified decisions about using LLM-generated summaries, including some that are low quality.Source: §1 · p. 3

2 Background & Motivation

The paper frames confidence calibration as a way to assess whether code summaries are likely to be useful, especially when no human-written reference is available. It reviews calibration measures, ways to summarize token confidence, and similarity to human summaries as a correctness criterion.

2 Background & Motivation

  • LLM code summarizers use in-context learning and semantic augmentation, but generated code summaries remain imperfect despite progress.The section situates this work amid earlier neural summarization approaches and current LLM-based methods.Source: §2 · p. 3

2.1 Calibration: How to use unreliable outputs from flawed models

  • Calibration means predicted confidence should align with empirical correctness rates, making it useful for decisions about whether to rely on a model output.A well-calibrated model is correct about 60% of the time when it reports 60% confidence.Source: §2.1 · pp. 3–4

2.1.1 Measuring Calibration.

  • Brier Score summarizes probability-outcome error, while ECE measures confidence-accuracy gaps across bins; ECE can mislead when binning varies.Brier scores should be interpreted relative to a base-rate reference, and ECE requires careful binning.Source: §2.1.1 · pp. 4–5

2.2 Confidence in a generated summary

  • A generated summary is a token sequence with per-token probabilities, so the paper considers arithmetic or geometric means to obtain one confidence value.The token probabilities come from the model’s output distribution, regardless of the decoding method.Source: §2.2 · p. 6

2.3 Correctness of a generated summary

  • The paper defines summary correctness as sufficient similarity to human-written summaries, a criterion difficult to assess in use without a human reference.Prior work found embedding-based similarity metrics more aligned with human judgments than lexical measures; the authors reanalyze BERTScore and SentenceBERT for binary prediction.Source: §2.3 · pp. 6–7

3 Research Questions

The research questions examine whether LLM confidence predicts code-summary similarity, how rescaling affects calibration, and whether calibration varies across summary position. The authors report that averaged per-token probabilities are not well-calibrated and describe the study as the first detailed examination of this task.

3 Research Questions

  • RQ 1 asks whether confidence measures over generated-summary tokens are well-calibrated; the study tests LLM log-probabilities against thresholded BERTScore similarity.The authors examine several ways to aggregate per-token probabilities and set similarity thresholds.Source: §3 · p. 7
  • Averaged per-token probabilities are not well-calibrated for predicting summary correctness, and the authors try Platt rescaling to improve alignment with observed frequencies.Correctness is evaluated using BERTScore similarity thresholded into a binary sufficient-similarity outcome.Source: §3 · pp. 7–8
  • RQ 2 asks how rescaling affects calibration across LLMs, while RQ 3 asks whether earlier summary tokens are better calibrated than later ones.The authors examine per-token probabilities across the generated sequence after observing that they change from early to later tokens.Source: §3 · p. 8
  • The authors characterize these questions together as the first detailed examination, to their knowledge, of LLM calibration for code summarization.Source: §3 · p. 8

4 Methodology

The methodology describes the models and prompting strategies, then explains how the study computes summary confidence, defines correctness, and evaluates calibration. It uses token probabilities, human-similarity thresholds, and rescaling or token-selection procedures to assess confidence.

4.1 Models

  • The experiments used GPT-3.5-Turbo, Code-Llama-70b, and DeepSeek-Coder-33b Instruct.Code-Llama-70b and DeepSeek-Coder-33b Instruct were the coding models described for the study.Source: §4.1 · p. 8

4.2 Prompting Methods for Code Summaries

  • Prompts combined three BM25-retrieved method-comment examples with ASAP-based chain-of-thought extracted using static analysis.The approach paired retrieval with intermediate reasoning steps informed by repository information, identifier types and scope, and dataflow.Source: §4.2 · pp. 8–9

4.3 How to measure models’ confidence in a token sequence? (RQ1)

  • Under an independence assumption, the study used the geometric mean of per-token probabilities as sequence confidence because it was less sensitive to high token probabilities than the arithmetic mean.The authors report that the geometric mean consistently calibrated better empirically.Source: §4.3 · p. 9

4.5 Experimental Dataset

  • Calibration experiments used CodeXGLUE Java and Python datasets, with 5,000 randomly selected test samples per language and training-only BM25 retrieval.The datasets contained 164,923 Java and 251,820 Python training samples; CodeXGLUE was deduplicated.Source: §4.5 · p. 10

4.6 Rescaling (RQ2)

  • Platt scaling fitted a logistic regression rescaler to predicted probabilities and outcomes, with five-fold cross-validation separating repositories across folds.Brier and Skill scores were calculated cumulatively across the folds.Source: §4.6 · p. 11

4.7 Using the first tokens to compute confidences (RQ3)

  • For RQ3, the study tuned the number of initial summary tokens to maximize Skill score in cross-validation, then tested Brier-score differences against using all tokens.The significance analysis used paired t-tests with Benjamini-Hochberg-adjusted p-values.Source: §4.7 · p. 11

5 Result

Raw token-probability confidence was poorly calibrated against summary correctness, while Platt rescaling improved calibration across configurations. Token position also mattered: later tokens tended to have higher probabilities, and the value of truncating confidence calculations depended on the setting.

5.1 RQ1 Calibration: LLM Confidence vs. Summary Correctness

  • Raw calibration was poor: Brier scores ranged from 0.25 to 0.57, skill scores were negative, yet reliability diagrams still showed some signal.The best reported confidence–BERTScore correlation was 0.45 for CodeLlama-70b Python summaries.Source: §5.1 · pp. 11–12, Table 1 · p. 12, Figure 4 · p. 12

5.2 RQ2: Platt Rescaled vs. Raw Calibration

  • Platt rescaling significantly improved calibration in every model-language configuration, yielding positive skill scores, Brier scores of 0.11–0.17, and ECE of 0.01–0.03.DeepSeek-Coder-33B on Java and CodeLlama-70b on Python each achieved a skill score of 0.11.Source: §5.2 · p. 12, Figure 5 · p. 13, Table 2 · p. 13
  • ASAP increased success rates for Python and Java, but Java skill scores fell while Python skill rose for CodeLlama-70B and DeepSeek-Coder-33B.The authors suggest extra prompt information may be redundant for verbose, strictly typed Java but may supply missing information, such as variable types, for Python.Source: §5.2 · pp. 12–13, Table 3 · p. 13
  • Threshold choice substantially affected calibration: at high precision, success was very low and GPT-3.5-Turbo Python summaries were virtually never accepted; at high recall, calibration remained good except for that configuration.High-precision correctness used BERTScore threshold 0.89; high-recall correctness used SentenceBERT threshold 0.80.Source: §5.2 · p. 14, Table 4 · p. 14

5.3 RQ3: Effect of Token Position on Confidence

  • Token probabilities generally increased with summary position, with CodeLlama-70b Java medians exceeding 0.9 after 20 tokens; similar late-token trends appeared across configurations.In GPT-3.5-Turbo Python summaries, median probabilities at positions 1–5 were already over 0.8, limiting the effect of excluding later tokens.Source: §5.3 · p. 14, Figure 6 · p. 15
  • Using only early tokens significantly reduced Brier scores in some, not all, configurations; the useful cutoff depended on language, model, prompting strategy, and summary length.For Python, significant reductions occurred only for CodeLlama-70b and DeepSeek-Coder-33B with few-shot prompting; for Java, they occurred in all but DeepSeek-Coder-33B few-shot prompting. Only 9.41% of Python summaries exceeded 15 tokens, limiting the overall effect of a 15-token cutoff.Source: §5.3 · pp. 14–16, Figure 7 · p. 15

6 Discussion: Threats & Limitations

The discussion identifies limits in the study’s comparison with open-source summaries and in capturing summary intent. It also reports API constraints on temperature scaling and poor calibration from the self-reflective measures studied.

6.1 Do LLMs surpass code summaries found in open-source projects?

  • The study uses open-source human-written summaries as references, but does not establish whether LLM summaries improve developers’ on-task programming performance.The authors note that their human ratings select a similarity threshold; they are not intended to assess completeness or accuracy against code.Source: §6.1 · p. 16

6.2 Limitation: Summary Intent Capture

  • Similarity metrics do not capture developers’ different how, what, and why intents, leaving intent-aware evaluation and confidence scores for future work.A user may request a summary for a particular intent, but those intents are not explicitly represented by the metrics discussed.Source: §6.2 · p. 16

6.3 Limitation: Platt scaling

  • Temperature scaling was unavailable because the APIs used did not provide access to complete softmax layers.The authors identify incorporating temperature scaling as future work if complete softmax access becomes available.Source: §6.3 · p. 16

6.4 Self-Reflection Calibration

  • GPT-3.5-Turbo’s logit-based self-reflection was severely overconfident, while verbalized self-reflection was poorly calibrated in both prompting setups.GPT-3.5-Turbo assigned confidence above 0.9 to 94% of predictions and had a negative skill score; higher temperatures and majority voting did not improve verbalized-measure calibration.Source: §6.4 · p. 17

7 Related Work

Related work situates this study at the intersection of calibration and code summarization, where similarity-based correctness presents distinct challenges. It also describes reference-free content-adequacy evaluation as complementary to calibrated similarity.

Related work

  • Calibration research spans NLP and software engineering, but the cited SE work concerns tasks such as program repair rather than code-summary calibration.The passage notes established NLP work using log-probabilities for classification and generative settings.Source: §7 · p. 17
  • The authors report no prior study of calibration for generative code summarization models, whose summary correctness is continuous and differs from correctness in repair or synthesis.They also identify programming-language effects, static-analysis prompt products, and earlier-token uncertainty as SE-specific topics or contributions.Source: §7 · p. 17
  • SIDE measures content adequacy without a human reference, but can score relevant generic summaries highly and is presented as complementary to similarity-based confidence.The authors propose presenting SIDE and calibrated confidence together because they indicate different aspects of summary quality.Source: §7 · pp. 17–18
  • The authors anticipate further calibration studies across software-engineering problems to improve the reliability of model outputs.Source: §7 · p. 18

8 Conclusion & Acknowledgements

The conclusion presents calibration as a way to estimate whether generated code summaries are sufficiently similar to human-written ones. It reports promising prompting and token-position approaches, but not improved calibration from reflection-based methods.

8 Conclusion & Acknowledgements

  • The study finds that first-token confidence, retrieval-augmented few-shot prompting, and ASAP can be advantageous, while reflection-based approaches do not improve calibration.Source: §8 · p. 18
  • The paper frames code-summary calibration as estimating whether a generated summary is sufficiently similar to a human-written summary, using a thresholded similarity metric.Source: §8 · p. 18
  • The authors present their work as the first evaluation of whether LLM-based code summarization approaches are well calibrated, contributing a methodology and approaches to the question.Source: §8 · p. 18
Loading 2404.19318v3…