Source-linked AI summary
Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)
Toufique Ahmed, Kunal Suresh Pai, Premkumar Devanbu, Earl T. Barr
TL;DR
Code summarization prompts may benefit from explicitly exposing semantic facts that developers consider when performing coding tasks. ASAP automatically extracts and adds those facts to prompts, improving several evaluated settings while leaving developer-performance effects unvalidated.
Problem
ASAP addresses whether explicitly adding automatically extracted semantic facts to LLM prompts can improve code summarization, a task where further progress is still needed.
Method
ASAP augments BM25-retrieved few-shot exemplars and input code with automatically extracted semantic facts, then queries an LLM with the resulting prompt.
Results
12% overall improvement was observed with ASAP using the Codex model, while tagged identifiers increased BLEU by 0.41 for Java and 1.22 for Python.
Takeaways & Limitations
Semantic augmentation provides a beneficial way to use limited prompt-length budget, and richer prompts can outperform adding more unaugmented BM25 shots.
Takeaways & Limitations
The study did not conduct a user study, so metric improvements may not translate into improved developer performance.
Abstract
from arXiv · showhide
Large Language Models (LLM) are a new class of computation engines, "programmed" via prompt engineering. We are still learning how to best "program" these LLMs to help developers. We start with the intuition that developers tend to consciously and unconsciously have a collection of semantics facts in mind when working on coding tasks. Mostly these are shallow, simple facts arising from a quick read. For a function, examples of facts might include parameter and local variable names, return expressions, simple pre- and post-conditions, and basic control and data flow, etc. One might assume that the powerful multi-layer architecture of transformer-style LLMs makes them inherently capable of doing this simple level of "code analysis" and extracting such information, implicitly, while processing code: but are they, really? If they aren't, could explicitly adding this information help? Our goal here is to investigate this question, using the code summarization task and evaluate whether automatically augmenting an LLM's prompt with semantic facts explicitly, actually helps. Prior work shows that LLM performance on code summarization benefits from few-shot samples drawn either from the same-project or from examples found via information retrieval methods (such as BM25). While summarization performance has steadily increased since the early days, there is still room for improvement: LLM performance on code summarization still lags its performance on natural-language tasks like translation and text summarization. We find that adding semantic facts actually does help! This approach improves performance in several different settings suggested by prior work, including for two different Large Language Models. In most cases, improvement nears or exceeds 2 BLEU; for the PHP language in the challenging CodeSearchNet dataset, this augmentation actually yields performance surpassing 30 BLEU.
1 INTRODUCTION
The paper proposes Automatic Semantic Augmentation of Prompts (ASAP), which adds automatically extracted semantic code facts to few-shot LLM prompts. Applied to code summarization, the approach improves performance across models and settings, often by about 2 BLEU or more.
- Contribution: ASAP constructs software-engineering prompts from syntactic and semantic facts automatically extracted from source code.The method is motivated by the facts developers consider when manually performing coding tasks.
- Results: 2 BLEU is approximately the improvement reached or exceeded in almost all languages, while Go shows a slightly smaller gain.The paper relates this threshold to prior evidence about BLEU and human-perceived improvement.
- Results: 4.6 BLEU is the improvement reported for PHP, reaching 32.73 BLEU on the de-duplicated CodeSearchNet dataset.This is described as a state-of-the-art high point for PHP in that dataset.
- Evaluation: ASAP is evaluated for code summarization using code-davinci-002, text-davinci-003, and GPT-3.5-turbo against a vanilla-BM25 few-shot baseline.The evaluation also includes the code-completion task.
- Results: ASAP also produces statistically significant improvements on code completion.The reported completion metrics are exact match and edit similarity.
2 BACKGROUND & MOTIVATION
The paper situates ASAP within prompt engineering, few-shot learning, code summarization, and externally supplied reasoning information. It argues that code-analysis products can improve prompts beyond ordinary retrieval or model-generated step-by-step reasoning.
- Background: Prompt engineering treats large language models as computation engines programmed through prompts rather than parameter-changing training.Few-shot prompts provide input-output exemplars before a new test input.
- Prior Prompting Approaches: A few-shot chain-of-thought prompting variant underperformed, reaching about 20.25 BLEU versus 24.97 BLEU for vanilla BM25.The zero-shot step-by-step variant performed even worse.
- Semantic Augmentation: ASAP embeds analysis products derived by mature code-analysis tools into prompts instead of expecting models to derive intermediate information during the task.These products are intended to represent simple facts developers use when understanding code.
- Motivation: ASAP improves on previous prompting approaches by augmenting prompts with code-analysis products.The paper presents this as evidence that explicitly providing such information improves LLM performance.
- Code Summarization: Code summarization supports code understanding, requirements traceability, and maintenance, but progress remains incremental.Automated summaries can also become outdated as projects evolve.
3 DATASET & METHODOLOGY
The study evaluates ASAP on de-duplicated CodeSearchNet samples using few-shot LLM prompting and automatically derived code facts. Its prompts combine retrieved exemplars, analysis products, and the target function, producing improvements in summarization and completion.
- Dataset: CodeSearchNet provides de-duplicated, cross-project samples from six programming languages for the experiments.The languages are Java, Python, JavaScript, Ruby, Go, and PHP.
- Exemplar Retrieval: BM25 retrieves relevant input-output exemplars from the training set for few-shot prompting.The selected exemplars are paired with a target function for completion.
- Automatic Semantic Prompt Augmentation: The analysis products include repository information, tagged identifiers, and data-flow information derived from source code.Repository information uses GitHub, while data-flow information and identifier labels come from Treesitter-based analysis.
- Automatic Semantic Prompt Augmentation: Each ASAP exemplar is a triple of source input, analysis product, and output comment.The final prompt concatenates three exemplars with the target code and its analysis product.
- Results: ASAP achieves significant overall improvements on code completion as well as code summarization.Completion is evaluated with exact match and edit similarity.
4 RESULTS
Across six languages and multiple evaluation settings, A𝑆𝐴𝑃-enhanced prompts improved code summarization over vanilla BM25 few-shot prompting, with statistically significant gains and broader model applicability.
- BM25 selection alone improved BLEU-4 by 3.00 (15.10%) for Java and 1.12 (5.42%) for Python relative to earlier baseline sampling.The paper treats BM25-based few-shot learning as a baseline rather than a contribution.
- BLEU improvements ranged from 1.84 (8.12%) to 4.58 (16.27%) across six programming languages after combining A𝑆𝐴𝑃 prompting components.Most improvements exceeded the 2.0 BLEU threshold associated with likely human-perceived improvement.
- All final A𝑆𝐴𝑃 prompts significantly outperformed vanilla BM25 few-shot learning after false-discovery-risk adjustment.The comparison used a pairwise one-sided Wilcoxon signed-rank test.
- 14 of 16 project-level comparisons favored prompt-enhanced few-shot learning over vanilla BM25, covering both cross-project and same-project scenarios.These comparisons involved eight projects and yielded an 87.5% success rate.
- A𝑆𝐴𝑃 improved performance for text-davinci-003 by 13.08% to 18.69% and for gpt-3.5-turbo by 1.68% to 9.13% across Java, Python, and PHP.The findings suggest stronger effectiveness for completion models than for the chat model.
- 5.79% Exact Match and 5.11% Edit Similarity gains were observed for zero-shot Java and Python line completion with semantic facts.For Python, statistical significance was found only for Edit Similarity, not Exact Match.
5 DISCUSSION AND ABLATION STUDY
The discussion examines which semantic augmentations matter, how they compare with adding more vanilla shots, and how they affect output specificity, cost, and errors. Ablations and examples indicate that repository information and tagged identifiers contribute substantially, while three augmented shots can outperform five unaugmented BM25 shots.
- Ablation study: Repository information contributes most to A𝑆𝐴𝑃’s performance among the evaluated semantic components.The paper separately ablates repository information, tagged identifiers, and data-flow graphs.
- Ablation study: Tagged identifiers improve BLEU by 0.41 for Java and 1.22 for Python over a tag-free identifier list.The comparison supports a benefit from explicitly representing identifier roles.
- More shots or A𝑆𝐴𝑃?: Three A𝑆𝐴𝑃-augmented shots outperform five vanilla BM25 shots, while adding more vanilla shots does not consistently improve performance.Only Java improves with both four and five vanilla shots relative to baseline, and unrelated retrieved samples can hurt performance.
- Output differences: A𝑆𝐴𝑃 consistently produces more specific information than the baseline, including terms such as “gradient,” “replace,” and “datarootext.”These differences were observed across multiple manually examined samples.
- Error analysis: A𝑆𝐴𝑃 can become overly specific, causing BLEU-4 to drop when developer-written summaries are more generic.The examined examples include “Andrew’s monotone chain algorithm,” “deployable unit,” and “column vector.”
- Performance and cost: 44% of samples improve with A𝑆𝐴𝑃, while performance declines for 30% and remains unchanged for the rest.The approach also requires more tokens, although the additional per-query monetary and performance cost is described as modest.
6 RELATED WORK
Related work spans traditional and neural code summarization, few-shot LLM prompting, and program-analysis-assisted software-engineering systems. The paper positions semantic augmentation of retrieved exemplars as a way to improve code summarization and code completion, while leaving other tasks for future work.
- Code summarization: Earlier code-summarization methods ranged from template matching and retrieval to pretrained transformer models such as CodeBERT, PLBART, and CodeT5.Pretrained models improve performance but generally require extensive, data-hungry fine-tuning and separate models for different languages.
- LLM prompting: LLMs enable few-shot code summarization by generating outputs from prompts containing problem-solution exemplars without task-specific fine-tuning.Few-shot learning has already been reported as beneficial for code summarization.
- LLMs in software engineering: Prior software-engineering research also combines LLMs with program analysis, synthesis, search-based testing, and few-shot learning for tasks including generation, repair, and testing.These approaches address a broader set of software-engineering problems than the paper’s code-summarization evaluation.
- Retrieval-based prompting: Retrieval-based exemplar selection has improved several software-engineering tasks, but its performance can be constrained by the availability of similar examples.The paper follows this line of work by using BM25 to select relevant exemplars.
- Scope: The paper reports that semantic hints help code summarization and code completion, while their value for other tasks remains future work.This defines the supported task boundary of the reported findings.
7 THREATS & LIMITATIONS
The paper identifies potential test-data exposure and the absence of user validation as limitations, while noting that fine-tuning could improve the approach at higher cost.
- Potential test-data exposure cannot be directly checked because the models’ training dataset is inaccessible.
- Random few-shotting performed worse, while relevant information improved performance as its amount and quality increased.
- The sample-size analysis found that the required sample size was always below the 1000 examples used in the primary studies.
- No user study was conducted, so metric improvements may not translate into improved developer performance.
- Fine-tuning large language models to use derived semantic facts may improve augmented prompting but would be costly.
8 CONCLUSION
The paper explores automatically augmenting few-shot prompts with semantically derived code facts for code summarization and completion. Its findings indicate that this approach is generally helpful and may surpass state-of-the-art performance.
- The paper proposes enhancing few-shot LLM samples with tagged facts automatically derived through semantic analysis.
- The approach is motivated by developers’ implicit extraction of code facts during comprehension before writing summaries.
- The authors evaluate semantic prompt augmentation on code summarization and code completion using the challenging, de-duplicated CodeSearchNet dataset.
- The findings indicate that Automatic Semantic Augmentation of Prompts is generally helpful, with estimates suggesting it can surpass state-of-the-art performance.