Automatic Code Summarization via ChatGPT: How Far Are We?

1 INTRODUCTION

• Code summarization automatically generates natural-language comments for code snippets, addressing the labor-intensive creation and maintenance of high-quality comments.High-quality comments support program comprehension and software maintenance, yet are often absent, unmatched, or outdated during software evolution.Source: §1 · p. 1
• ChatGPT accepts longer inputs, uses conversation history, and can process source code, but its use for code summarization remains insufficiently explored.Source: §1 · pp. 1–2
• The study designs heuristic questions to identify an appropriate prompt, then generates comments for all 14,918 CSN-Python test snippets.It evaluates BLEU, METEOR, and ROUGE-L against ground-truth comments and compares ChatGPT with NCS, CodeBERT, and CodeT5.Source: §1 · p. 2
• The study’s overview is presented in Figure 1, which frames prompt exploration, large-scale evaluation, model comparison, and subsequent analysis.Source: §1 · p. 2, Figure 1 · p. 2
• The paper investigates ChatGPT’s code summarization performance, evaluates it against three SOTA models, and outlines challenges and opportunities for future research.The stated contributions include prompt exploration, benchmark evaluation, comparative analysis, and identification of open research directions.Source: §1 · p. 2

2.1 Q1: Can ChatGPT perform code summarization tasks?

• ChatGPT can generate comments for programming-language functions, although automated code summarization is not its primary purpose.Source: §2.1 · pp. 2–3

2.2 Q2: What does the comment generated by ChatGPT look like?

• ChatGPT’s sample comments are much more detailed than the ground-truth comment and can recognize the programming language without it being specified.Source: §2.2 · p. 3

2.3 Q3: How to use ChatGPT to generate concise comments?

• Prompts containing “short,” “succinct,” or “concise” shorten comments, but they remain longer than the 11-word ground truth; “short” is closest.Source: §2.3 · p. 3

2.4 Q4: What kind of prompt does ChatGPT suggest for generating short comments?

• ChatGPT’s suggested prompts include examples, but some suggestions incorrectly ask users to provide the code description that ChatGPT should generate.Source: §2.4 · p. 3

2.5 Q5: Which one performs better, the ChatGPT-suggested prompts in Q4 or the prompt proposed in Q3?

• About 90% of ground-truth comments contain one sentence and about 99% contain fewer than 40 words, motivating sentence and word constraints.Source: §2.5 · p. 4
• ChatGPT (one sentence) achieves BLEU 10.66, METEOR 13.36, and ROUGE-L 20.37, outperforming ChatGPT (short) and ChatGPT (within 40 words).Source: §2.5 · pp. 5–6, Table 5 · p. 7
• The selected prompt is “Please generate a short comment in one sentence for the following function:<code>”, while ChatGPT (short) comments remain out-of-distribution.Source: §2.5 · pp. 4, 6

3.1 Dataset

• 251,820/13,914/14,918 samples comprise the clean CSN-Python training/validation/test sets used in the experiments.The dataset follows prior cleaning that removed comments unrelated to code snippets.Source: §3.1 · p. 7

3.1.1 Evaluation Metrics.

• BLEU, METEOR, and ROUGE-L evaluate generated comments against references, with higher scores indicating better summarization performance.BLEU uses n-gram precision with a brevity penalty; METEOR uses word alignment; ROUGE-L uses longest common subsequence.Source: §3.1.1 · p. 7

3.2 Baseline

• NCS, CodeBERT, and CodeT5 serve as the deep-learning code-summarization baselines compared with ChatGPT.NCS uses a Transformer encoder-decoder with copying, CodeBERT uses masked-language modeling and replaced-token detection, and CodeT5 uses four pre-training tasks with a pre-trained decoder.Source: §3.2 · pp. 7–8

4.1 Result Analysis

• CodeT5 has higher median, first-quartile, and third-quartile scores than ChatGPT for BLEU and ROUGE-L, and a higher third quartile for METEOR.These distributions were computed over all CSN-Python test-set samples.Source: §4.1 · p. 8, Figure 9 · p. 8
• CodeT5 comments are significantly better than ChatGPT comments across BLEU, METEOR, and ROUGE-L, with differences statistically significant at the 5% level and all three p-values below 0.05.Source: §4.1 · p. 8

4.2.1 successful cases.

• ChatGPT can capture core code semantics and describe detailed behavior, such as establishing a Druid-broker connection and returning the connection object.In the successful case, CodeT5 instead used “Get” rather than the reference verb “Establish.”Source: §4.2.1 · p. 8, Figure 10 · p. 8

4.2.2 moderate cases.

• Both systems identified logging, but ChatGPT omitted “to standard error” and added irrelevant color-coding information.CodeT5 included “to stderr,” suggesting greater sensitivity to the programming-language feature sys.stderr.write.Source: §4.2.2 · p. 8, Figure 11 · p. 9

4.2.3 weak cases.

• Neither model recognized that the weak-case function executes SQL statements, instead misinterpreting it as a copy operation.The authors attribute this misunderstanding partly to ignoring conn.commit() and note that ChatGPT may struggle with deep code logic.Source: §4.2.3 · p. 9, Figure 12 · p. 9

Neural code summarization

• NMT-based summarizers use encoder-decoder networks and increasingly incorporate ASTs, code property graphs, similar snippets, file context, and pre-training.Examples include CodeBERT and CodeT5, which pre-train on general language-modeling tasks before fine-tuning for code summarization.Source: §5 · p. 9

ChatGPT for software engineering

• Recent ChatGPT studies examine automated program repair, deep-learning debugging, and code generation, repair, and summarization.Reported concerns include verification costs, prompt effects, poor generalization to unseen problems, negative effects of long prompts, and limited attention span.Source: §5 · p. 9

Threats to External Validity

• ChatGPT can generate varied answers for identical code under different prompts, so conclusions based on random outputs may be misleading.The study mitigates this by selecting a prompt in a pre-study and using a large dataset.Source: §6 · p. 9
• Results may vary across programming languages and benchmark sizes because the evaluation targets Python on the CSN corpus.The authors selected CSN and Python as a popular large benchmark setting and plan to evaluate other CodeSearchNet languages.Source: §6 · p. 9

Threats to Construct Validity

• Because the study evaluates GPT-3.5, its results may underestimate the latest ChatGPT based on GPT-4.The authors plan to compare GPT-4-based ChatGPT with previous versions.Source: §6 · p. 9

(1) A lot of room for improvement on existing datasets

• ChatGPT’s zero-shot summarization still has substantial room for improvement on CSN-Python, motivating automated prompt engineering beyond sentence and word limits.The paper identifies sentence limitation and word limitation as two directions for exploring prompts that generate in-distribution comments.Source: §7 · pp. 9–10

(2) Cropping and templating the rich but lengthy comments that ChatGPT generates by default

• Cropping and templating ChatGPT’s rich comments could extract concise, useful information while making outputs more developer-friendly.The authors observe that ChatGPT’s comments can contain more details than ground-truth comments, while cropped and templated versions are useful for developers.Source: §7 · p. 10

(3) Creating a new benchmark dataset

• Low-quality ground-truth comments in existing datasets motivate using ChatGPT to help construct higher-quality code-summarization benchmarks.One example shows ChatGPT producing a more informative summary than a ground-truth comment that does not capture the code’s functionality.Source: §7 · p. 10, Figure 14 · p. 10

(4) Designing new metrics to evaluate code summarization models

• Traditional metrics can rank CodeT5 above ChatGPT despite ChatGPT covering important semantics that CodeT5 omits, motivating metrics based on semantic quality.For the illustrated snippet, CodeT5 scores higher on BLEU, METEOR, and ROUGE-L, while ChatGPT captures the meanings of converting a string to a valid filename.Source: §7 · p. 11, Figure 15 · p. 11

• Overall, ChatGPT is inferior to NCS, CodeBERT, and CodeT5 on CSN-Python in BLEU, METEOR, and ROUGE-L.Source: §8 · p. 11
• The study uses heuristic questions to explore a prompt that guides ChatGPT toward in-distribution comments before comparing it with three dedicated summarization models.Source: §8 · p. 11
• The findings outline four open challenges and opportunities for applying ChatGPT to code summarization.Source: §8 · p. 11