Source-linked AI summary
Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learning
Mingyang Geng, Shangwen Wang, Dezun Dong, Haotian Wang, Ge Li, Zhi Jin, Xiaoguang Mao, Xiangke Liao
TL;DR
Existing comment generators usually produce one perspective, although developers need comments covering diverse intents. The study tests few-shot in-context learning with LLMs, alongside demonstration selection and output reranking, and finds that adequately prompted Codex outperforms supervised learning while both strategies further improve performance.
Problem
Existing approaches generally generate one aspect of a code snippet, while developers often need comments expressing multiple intents and perspectives.
Method
The study evaluates Codex for multi-intent comment generation using few-shot in-context learning, similarity-based demonstration selection, and comment-based output reranking.
Results
Adequate prompting lets Codex exceed DOME, while similarity-based selection and reranking further improve its performance on two large-scale datasets.
Takeaways & Limitations
LLMs can potentially support multi-intent comment generation, making demonstration selection and reranking promising directions for future evaluation and research.
Takeaways & Limitations
Possible Codex pretraining data leakage and sampling or demonstration-selection randomness remain threats that the experiments do not fully eliminate.
Abstract
from arXiv · showhide
Code comment generation aims at generating natural language descriptions for a code snippet to facilitate developers' program comprehension activities. Despite being studied for a long time, a bottleneck for existing approaches is that given a code snippet, they can only generate one comment while developers usually need to know information from diverse perspectives such as what is the functionality of this code snippet and how to use it. To tackle this limitation, this study empirically investigates the feasibility of utilizing large language models (LLMs) to generate comments that can fulfill developers' diverse intents. Our intuition is based on the facts that (1) the code and its pairwise comment are used during the pre-training process of LLMs to build the semantic connection between the natural language and programming language, and (2) comments in the real-world projects, which are collected for the pre-training, usually contain different developers' intents. We thus postulate that the LLMs can already understand the code from different perspectives after the pre-training. Indeed, experiments on two large-scale datasets demonstrate the rationale of our insights: by adopting the in-context learning paradigm and giving adequate prompts to the LLM (e.g., providing it with ten or more examples), the LLM can significantly outperform a state-of-the-art supervised learning approach on generating comments with multiple intents. Results also show that customized strategies for constructing the prompts and post-processing strategies for reranking the results can both boost the LLM's performances, which shed light on future research directions for using LLMs to achieve comment generation.
1 INTRODUCTION
Existing code comment generators typically describe one aspect, whereas developers need multiple perspectives. This study evaluates LLM-based in-context learning and finds that adequate demonstrations, similarity-based selection, and reranking improve multi-intent comment generation.
- Motivation: Existing approaches generate one-aspect comments, but developers often need multiple intents, including functionality and usage.Around 67% of methods in top-starred Java projects reportedly contain comments with more than one intent.
- Approach: The study investigates in-context learning for multi-intent comment generation using task instructions, demonstrations, and a query.Experiments use Codex on two large-scale Java datasets, Funcom and TLC.
- Findings: Fewer than 10 demonstrations leave Codex below DOME, whereas 10 demonstrations make its performance exceed the supervised baseline.The finding concerns the effect of prompt adequacy on LLM effectiveness.
- Findings: 97% and 131% BLEU gains result when the 10 demonstrations are selected by code similarity rather than randomly, across the two datasets.The comparison uses the same number of examples and Codex as the model.
2 BACKGROUND AND RELATED WORKS
Code comment generation supports program comprehension, but conventional methods generally summarize only one aspect despite developers expressing multiple intents. The section introduces LLMs and in-context learning as alternatives that use large-scale pretraining knowledge without task-specific parameter updates.
- 2.1 Comment Generation: Automatic comment generation summarizes code in natural language to facilitate program comprehension.Earlier approaches include manually defined rules and neural comment generation models.
- 2.1 Comment Generation: Developers express six comment intents: what, why, how-to-use, how-it-is-done, property, and others.The taxonomy is presented in Table 1.
- 2.2 Large Language Models: LLMs are pretrained on massive unlabeled corpora using self-supervision, capturing domain knowledge without task-specific training data.The background describes applications across natural language and code-related tasks.
- 2.2 Large Language Models: Codex is a GPT-3-based LLM trained autoregressively on code-comment pairs from many programming languages.The study uses Codex as its representative software-engineering LLM.
- 2.3 In-Context Learning: In-context learning sends task descriptions and a few examples with a query to an LLM without updating model parameters.Prompts align downstream inputs with the model’s pretrained knowledge.
3 STUDY DESIGN
The study evaluates Codex for multi-intent comment generation through in-context learning, using prompt demonstrations, retrieval-based selection, and comment reranking on Java datasets.
- The study investigates multi-intent comment generation with large language models using the in-context learning paradigm.
- 3.2 The Prompt Template for Multi-Intent Comment Generation: In-context learning prompts combine natural-language task instructions, code demonstrations with comments, and a developer query within the model's context window.Zero-shot, one-shot, and few-shot settings differ by the number of code demonstrations.
- 3.2 The Prompt Template for Multi-Intent Comment Generation: The prompt specifies the programming language and desired comment intent, including what, why, how-to-use, how-it-is-done, and property descriptions.
- 3.3 Demonstration Retrieval: Retrieval-based demonstration selection chooses examples similar to the target code, using token-based and semantic-based strategies.Token similarity uses processed sub-token overlap and Jaccard similarity, while semantic retrieval uses sentence-transformer embeddings and cosine similarity.
- 3.4 Reranking Strategy: Generated comments are reranked by comparing them with the comment associated with the most similar retrieved code, using token-based and semantic-based comment similarities.
- 3.5 Datasets: Experiments use the Funcom and TLC Java datasets and evaluate Codex with BLEU, ROUGE-L, and METEOR.The datasets contain code-comment pairs from thousands of Java projects.
4 STUDY RESULTS
Codex performance improves as prompts include more demonstrations, with ten-shot learning consistently outperforming DOME across metrics and datasets. Retrieval-based selection and reranking provide additional gains, although no single strategy consistently dominates.
- RQ1: Vanilla In-Context Learning: Zero-shot and one-shot Codex remain weaker than DOME because insufficient demonstrations do not fully leverage the model’s potential.Zero-shot average BLEU is 21.2% and 18.8% on the two datasets, versus DOME’s 31.8% and 22.2%.
- RQ1: Vanilla In-Context Learning: Ten demonstrations make Codex outperform DOME consistently across all three metrics and both datasets.With ten examples, Codex averages 33.4%/76.1%/24.1% on Funcom and 27.2%/66.7%/19.2% on TLC, using BLEU/ROUGE-L/METEOR.
- RQ2: Demonstration Selection: Token- and semantic-similarity selection substantially improve ten-shot BLEU over random selection, with relative gains reaching 97% on Funcom and 131% on TLC.Semantic selection raises BLEU from 33.4% to 65.9% on Funcom and from 27.2% to 62.8% on TLC.
- RQ2: Demonstration Selection: Similar demonstrations can make one-shot learning outperform vanilla ten-shot learning, highlighting demonstration quality as an important prompt factor.On Funcom, similar-example one-shot BLEU is 39.2%, compared with 33.4% for vanilla ten-shot learning.
- RQ2/RQ3: Strategy Dependence: Neither demonstration selection nor reranking has a universally superior strategy across settings.Performance depends on factors including the number of examples, intent category, and whether demonstrations are selected.
- RQ3: Reranking: Both token-based and semantic-based reranking further improve Codex, but their relative effectiveness depends on the selection setting and intent.Token-based reranking generally works better with demonstration selection, whereas semantic-based reranking generally works better without it.
5 DISCUSSION
The discussion highlights strong human-evaluation performance and practical implications, while identifying threats involving data leakage, randomness, demonstration availability, and language scope.
- 5.1 Human Evaluation: 4.3, 4.1, and 3.8 were the highest participant scores for the best-performing model variant across the three evaluated aspects.The variant used 10-shot learning with semantic-based demonstration selection and token-based reranking.
- 5.1 Human Evaluation: All model variants scored above 4 for naturalness, whereas every usefulness score remained below 4.The results indicate stronger fluency than usefulness in the human evaluation.
- 5.2 Implications: Prompt quality affects generated comments, and similar demonstrations can improve results, but no single demonstration-selection strategy is universally best.The discussion calls for further attention to demonstration selection.
- 5.2 Implications: Because sampling may not produce the best comment first, developers can query LLMs repeatedly and researchers should investigate automatic reranking.The paper links repeated generation to the need for better result-selection methods.
- 5.3 Threats to Validity: Codex may have seen test comments during pre-training, but zero-shot weakness suggests its outputs are not generated solely through memorization.Fully resolving this leakage threat would require retraining the model from scratch, which the authors consider infeasible with available computation.
- 5.3 Threats to Validity: The study mitigates sampling and demonstration-selection randomness with 100 repetitions, but acknowledges that this may not fully eliminate the threat.More experiments are left for future work.
- 5.3 Threats to Validity: The practical availability of similar code for prompt construction is unclear, although 10-shot Codex still exceeds DOME with randomly selected demonstrations.The study also focuses only on Java because multi-intent comment datasets in the literature are available for that language.
6 CONCLUSION
The conclusion reports that few-shot in-context learning enables Codex to address multi-intent comment generation and that prompt selection and output reranking improve its effectiveness.
- 6 CONCLUSION: Few-shot in-context learning enables Codex to exceed the state-of-the-art supervised learning approach on multi-intent comment generation.The study establishes new baselines for the task.
- 6 CONCLUSION: Demonstration selection and result reranking both boost Codex’s performance.The paper identifies both strategies as research directions for deeper investigation.
- 6 CONCLUSION: The study establishes new baselines for multi-intent comment generation and points to research directions for further investigation.
7 DATA AVAILABILITY
The study states that all code and data are publicly available.
- 7 DATA AVAILABILITY: All code and data in the study are publicly available.