Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learning

• Prior approaches generate one comment per code snippet, although developers need multiple comments covering diverse intents; around 67% of methods in top-starred Java projects have comments with multiple intents [47].Zhai et al. identified six comment-intent categories in real-world projects.Source: §1 · p. 2
• The study tests in-context learning with Codex on Funcom and TLC, motivated by LLM pretraining on code-comment pairs and real-world comments that often contain multiple intents.It also investigates customized demonstration selection and post-processing strategies; DOME's supervised-learning basis is limited by available training data.Source: §1 · p. 2
• With ten demonstration examples, Codex outperforms the supervised DOME approach, whereas fewer than ten examples yield suboptimal performance.The comparison is reported for experiments on the Funcom and TLC Java datasets.Source: §1 · p. 2
• With ten examples selected by code similarity, Codex BLEU increases by 97% and 131% on the two datasets, respectively, compared with random selection.Source: §1 · p. 2
• Using the comment of a corpus code similar to the target to guide reranking improves Codex BLEU by 9.9% and 9.6% on the two datasets, respectively.These gains are compared with the preceding experiment setting.Source: §1 · p. 2

2.1 Comment Generation

• Existing comment-generation methods generally describe one aspect, although developers express six intents; DOME generates comments conditioned on a specified intent [47].The six categories are what, why, how-to-use, how-it-is-done, property, and others. Earlier approaches include template-based rules and learned code-to-comment models.Source: §2.1 · p. 3

2.2 Large Language Models

• Codex, the study’s representative LLM, is based on GPT-3 and pretrained autoregressively on code-comment examples from many programming languages.During pretraining, it predicts the next token in a sequence and feeds that prediction into the next step; the paper selects it as a widely studied software-engineering LLM.Source: §2.2 · p. 3

2.3 In-Context Learning

• In-context learning supplies a prompt with examples and/or task descriptions plus a query, enabling an LLM to respond without updating its parameters.The study’s rationale is that code-comment pretraining exposes LLMs to real-world comments expressing different intents, potentially enabling diverse-perspective comment generation without fine-tuning.Source: §2.3 · pp. 3–4

3.1 Research Questions

• RQ1 compares zero-, one-, and few-shot Codex with DOME [47], while RQ2 tests retrieval-based demonstration selection and RQ3 tests reranking sampled outputs.The questions examine vanilla in-context learning with randomly selected examples, whether similar-code demonstrations improve performance, and whether reranking multiple outputs can produce more qualified results.Source: §3.1 · p. 4

3.2 The Prompt Template for Multi-Intent Comment Generation

• Prompts combine the query code, code-comment demonstrations, and natural-language instructions specifying the programming language and desired comment intent.The intent-specific instructions cover functionality (what), rationale (why), usage (how-to-use), implementation (how-it-is-done), or method properties; demonstrations pair code with a comment and are separated by “###”. Zero-shot uses no demonstrations, one-shot uses one, few-shot uses multiple, and the prompt must fit the model’s context window.Source: §3.2 · p. 4

3.3 Demonstration Retrieval

• Retrieval selects demonstrations with comments in the desired intent category, using either token-level or semantic similarity to the target code.Token-based retrieval removes Java keywords, splits identifiers using camel case and underscores, lowercases subtokens, and ranks candidates with Jaccard similarity; semantic retrieval embeds code with a pretrained sentence transformer and ranks by cosine similarity.Source: §3.3 · pp. 4–6

3.4 Reranking Strategy

• Reranking uses the comment for the code snippet most similar to the target as a reference, then scores generated comments by token-based or semantic similarity.Semantic similarity uses a pretrained sentence transformer to embed each whole comment and cosine similarity to compare it with the reference.Source: §3.4 · p. 6

3.5 Datasets

• Evaluation uses the Funcom and TLC Java datasets, reusing intent labels predicted by a fine-tuned CodeBERT classifier with an F1-score around 90%.Funcom contains 2.1M code-comment pairs from 29K projects; TLC contains 87,136 pairs from over 9K projects. The study retrieves demonstrations from training sets, evaluates on test sets, and excludes the “others” intent because its comments are unspecified or ambiguous.Source: §3.5 · p. 6

3.6 Evaluation Metrics

• Performance is measured with BLEU, ROUGE-L, and METEOR, which assess n-gram precision, longest-common-subsequence overlap, and unigram matching, respectively.METEOR aligns the generated summary with the reference; ROUGE-L is based on the longest common subsequence.Source: §3.6 · p. 6

3.7 Experiment Settings

• Experiments use code-davinci-002 with temperature 0.5, up to ten demonstrations, and DOME [47] as the comparison baseline.Zero-shot, one-shot, five-shot, and ten-shot settings are evaluated; RQ1 and RQ2 settings are repeated 100 times and averaged to account for randomness.Source: §3.7 · p. 6

4.1 RQ1: the Effectiveness of Vanilla In-Context Learning

• With ten code demonstrations, Codex consistently outperforms DOME across all three metrics on Funcom and TLC, averaging 33.4%/76.1%/24.1% and 27.2%/66.7%/19.2%, respectively.At five demonstrations, Codex’s ROUGE-L and METEOR values are higher than DOME’s, while BLEU is slightly lower. Across intent categories, all approaches generally have relatively low performance on the “how-it-is-done” category.Source: §4.1 · p. 7

4.2 RQ2: the Effectiveness of Demonstration Selection

• At ten shots, token- and semantic-similarity selection raises Codex BLEU on Funcom from 33.4% to 64.5%/65.9% and on TLC from 27.2% to 60.7%/62.8%, respectively.Source: §4.2 · p. 7
• No demonstration-selection strategy consistently wins: token-based selection is better on average at one shot, while semantic selection is better at five or ten shots, with intent-level exceptions.For the “what” intent, token-based selection scores BLEU 50.5%/44.8% on the two datasets, exceeding semantic selection’s 40.4%/40.2%.Source: §4.2 · p. 8

4.3 RQ3: the Effectiveness of Reranking

• With 10-shot semantic selection and token reranking, Codex scores BLEU 72.4%/68.8% on the two datasets, beating DOME by 128%/210%, respectively.These are the average scores of the best-performing model variant.Source: Figure 3 · p. 10
• For 1-shot reranking, BLEU rises from 23.1%/21.1% to 29.1%/26.8% with token reranking and 30.2%/27.2% with semantic reranking on Funcom/TLC.In another case, token reranking selected an oracle-equivalent comment and raised BLEU from 23.4% to 68.6%.Source: Figure 3 · p. 10, §4.3 · p. 10, Figure 4 · p. 10
• Reranking strategies also have no consistent winner: token-based reranking generally works better with demonstration selection, while semantic-based reranking generally works better without it.For the “what” intent, semantic reranking can perform better even when combined with demonstration selection.Source: §4.3 · p. 10

5.1 Human Evaluation

• In the user study, the best variant scored 4.3, 4.1, and 3.8 across the three aspects; all variants exceeded 4 for naturalness but scored below 4 for usefulness.Six experienced Java developers rated 400 generated comments from 100 code snippets without knowing which approach produced them.Source: §5.1 · pp. 10–11, Table 6 · p. 11

5.2 Implications

• Codex-10-shot with semantic-based selection and token-based reranking outperformed DOME by 128%/210% on BLEU for Funcom/TLC.The authors say LLMs could help developers generate multi-intent comments and that new code summarization approaches should be compared with LLMs.Source: §5.2 · p. 11
• Examples similar to the target code may improve prompts, and token-based and semantic-based demonstration selection complement each other rather than offering a single best strategy.The authors call for more research on demonstration selection strategies.Source: §5.2 · p. 11
• Because sampling may not produce an oracle-like comment initially, developers can query LLMs again, and automatic reranking warrants further study.The authors report that two simple reranking heuristics yielded promising results.Source: §5.2 · p. 11

5.3 Threats to Validity

• Possible pre-training data leakage remains an internal-validity threat, although weak zero-shot performance suggests outputs are not simply memorized test comments.Fully addressing leakage would require retraining the model from scratch, which the authors say is currently infeasible given computational-resource limits.Source: §5.3 · p. 11
• Experiments were repeated 100 times to mitigate randomness from model sampling and demonstration selection, but that may not eliminate the threat.The authors leave further experiments for future work.Source: §5.3 · p. 11
• Although finding similar demonstrations may be difficult, Codex with 10 randomly selected examples still outperformed DOME.Source: §5.3 · p. 11
• The study covers only Java because multi-intent datasets are available for that language; the authors cite two large-scale datasets and Java’s prominence in comment-generation research as mitigating factors [36] [47].Source: §5.3 · p. 11

• Few-shot in-context learning enables Codex to outperform the state-of-the-art supervised approach on multi-intent comment generation.Source: §6 · p. 11
• Demonstration selection and result reranking both help boost Codex's performance on the task.Source: §6 · p. 11
• The study establishes new baselines for multi-intent comment generation and identifies research directions for further investigation.Source: §6 · p. 11