Source-linked AI summary
Few-shot training LLMs for project-specific code-summarization
Toufique Ahmed, Premkumar Devanbu
TL;DR
The paper asks whether few-shot LLM learning can address project-specific code summarization despite limited local data. It evaluates Codex with prompted examples in cross-project and same-project settings, finding evidence that few-shot and same-project training can outperform comparison models, while noting possible pretraining overlap with the test data.
Problem
Project-specific code phenomena matter for software engineering, but relevant project data can be limited, motivating study of few-shot learning for code summarization and same-project learning.
Method
The paper evaluates Codex for code summarization using prompted examples from cross-project or same-project data, including historically ordered samples for same-project experiments.
Results
Codex few-shot code summarization significantly outperformed a fine-tuned model trained with thousands of samples using just ten samples, and same-project few-shot performance exceeded cross-project performance overall.
Takeaways & Limitations
Project-specific few-shot training appears promising and feasible for code summarization because a task can use ten samples from the same project.
Takeaways & Limitations
The authors could not account for the risk that Codex had already seen test data during its large-scale pretraining because the pretraining dataset was unavailable.
Abstract
from arXiv · showhide
Very large language models (LLMs), such as GPT-3 and Codex have achieved state-of-the-art performance on several natural-language tasks, and show great promise also for code. A particularly exciting aspect of LLMs is their knack for few-shot and zero-shot learning: they can learn to perform a task with very few examples. Few-shotting has particular synergies in software engineering, where there are a lot of phenomena (identifier names, APIs, terminology, coding patterns) that are known to be highly project-specific. However, project-specific data can be quite limited, especially early in the history of a project; thus the few-shot learning capacity of LLMs might be very relevant. In this paper, we investigate the use few-shot training with the very large GPT (Generative Pre-trained Transformer) Codex model, and find evidence suggesting that one can significantly surpass state-of-the-art models for code-summarization, leveraging project-specific training.
1 INTRODUCTION
The introduction motivates few-shot learning for project-specific code summarization, where local terminology and coding patterns matter but project data may be scarce. It frames the study around whether LLM few-shot learning extends to code summarization and same-project settings, and how it compares with state-of-the-art models.
- LLMs and few-shot learning: LLMs such as GPT-3 and Codex use autoregressive next-token training and can support code-related tasks after suitable fine-tuning.GPT-3 is described as having over 150 billion parameters, while Codex has 12 billion and is trained on code.
- LLMs and few-shot learning: Few-shot learning supplies task examples through a prompt rather than conventional weight-adjusting fine-tuning.The prompt contains a handful of input-output pairs followed by a query input, making the regime highly sample-efficient.
- Project specificity: Project-specific terminology, identifiers, APIs, and coding patterns create local phenomena that code-learning systems must handle.These conventions appear across documentation, issue reports, identifiers, and other project artifacts.
- Project specificity: Project-specific examples may be scarce, whereas older deep-learning systems can require O(10^4) or O(10^5) local samples.This scarcity is especially emphasized early in a project’s history.
- Research questions: The paper uses comment synthesis as a well-studied test bed to ask whether LLM few-shot learning transfers to code summarization and same-project learning.It also asks how performance in these settings compares with state-of-the-art models.
2 BACKGROUND AND RELATED WORK
The background positions code summarization as a neural translation-like task and reviews the transition from encoder-decoder models toward large decoder-only models. Prior work shows strong zero-shot results from InCoder, while fine-tuned models can still outperform that setting.
- Code summarization: Automated code summarization can support comprehension, development, maintenance, and updating misaligned or outdated comments.Developers reportedly spend around 59% of their time comprehending work.
- Modeling approaches: Code summarization resembles neural machine translation and has commonly used neural encoder-decoder architectures.The related work spans RNN-based systems, Transformers, and foundation models.
- Modeling approaches: Large language models including Codex have been applied to code summarization, also called docstring generation.This establishes the task’s connection to decoder-only language-model applications.
- Prior results: InCoder achieved impressive zero-shot results on CodeXGLUE, but fine-tuned CodeT5, CodeBERT, and PLBART models could still outperform that setting.Prior Codex-D work used a small human-evaluation dataset and omitted BLEU-4, which CodeXGLUE recommends.
3 METHODOLOGY
The methodology prompts Codex with a small set of code-comment examples before a query function, evaluating cross-project and same-project code summarization on sampled CodeXGLUE data. The study controls prompt size and generation settings to fit the private Codex interface.
- Few-shot pipeline: The approach prepends n function-summary pairs and then a query function, prompting Codex to generate the query’s comment.The prompt is structured as f_1,s_1, f_2,s_2, …, f_n,s_n, f_q, with comment boundary symbols.
- Dataset: The evaluation uses 1,000 randomly selected CodeXGLUE test examples across Ruby, JavaScript, Java, Go, PHP, and Python.Ten training samples are randomly selected for Codex few-shot prompting, while comparisons use the same sampled test collection.
- Same-project setting: Same-project experiments use four Python and four Java projects, with historical samples selected by creation date to prevent future-data leakage.The design targets shared project identifiers and vocabulary while remaining within the CodeXGLUE test set.
- Model and prompt budget: The study uses code-davinci-002, whose 4000-token prompt limit permits approximately 10–15 sequences for few-shot prompting.The interface supports prompting but not conventional weight-adjusting fine-tuning in this study.
- Generation settings: Generation uses temperature 0, Top_p 1.0, and a 50-token maximum, then clips output at the comment-ending symbol.These settings aim for well-defined answers and limit continued generation after the summary.
4 RESULT
Codex few-shot training substantially outperformed fine-tuned foundation-model baselines in cross-project code summarization, while same-project examples improved performance further. Statistical testing found overall significant improvements, but zero- and one-shot settings performed poorly.
- Cross-project few-shot: Codex achieved more than +2 BLEU-4 for JavaScript and Go over competitive models.The authors describe improvements above +2 BLEU-4 as reasonable proxies for human-perceptible preference.
- Cross-project few-shot: With 10 samples, Codex outperformed all listed fine-tuned foundation models across all six programming languages.The baselines used thousands of training examples, whereas Codex used only 10 samples.
- Same-project few-shot: 21.65 BLEU-4 increased to 24.37 BLEU-4, a 12.56% improvement, when Codex used same-project rather than cross-project examples.Same-project Codex also outperformed the other models considered for every project.
- Same-project few-shot: Same-project few-shot training improved Codex performance for all 8 evaluated projects.The comparison used time-series ordering to avoid leakage between training and test examples.
- Zero-shot and one-shot training: Zero-shot and one-shot Codex training did not work for code summarization, achieving 2.96 and 6.22 BLEU-4 on average.The authors attribute the difficulty to teaching Codex to generate docstrings after code when its training commonly presents docstrings before code.
5 THREATS
The paper identifies limited direct safety risk from Codex-based code summarization, but warns that misleading comments could harm programmers and that test-data contamination cannot be ruled out.
- Code summarization has less direct safety and security risk than code generation because docstrings and comments are not executed.However, misleading comments could still create problems for programmers.
- The test data may have appeared in Codex’s large-scale pre-training, creating a contamination risk the authors could not assess.The pre-training dataset was unavailable to the authors.
- Abysmal zero- and one-shot performance and smooth improvement through roughly ten samples provide suggestive evidence against simple memorization.The authors interpret this pattern as evidence of few-shot generalization, while retaining the contamination concern.
6 CONCLUSION
The study finds that Codex few-shot training can outperform fine-tuned models with thousands of examples using ten samples, with same-project examples performing better than cross-project examples.
- Codex few-shot training significantly outperformed a fine-tuned model trained with thousands of samples using just ten samples.The paper frames this as evidence of strong sample efficiency for code summarization.
- Same-project Codex few-shot training performed better than cross-project training, with statistically significant overall improvement.The authors report same-project improvement as promising for project-specific software-engineering settings.
- Ten summarization samples can be generated within a few hours of development, supporting the feasibility of same-project few-shot training.The paper also makes the code-summarization dataset anonymously available.