Code Generation Tools (Almost) for Free? A Study of Few-Shot, Pre-Trained Language Models on Code

1 INTRODUCTION

• Building code-generation tools traditionally requires substantial human effort, often repeated for each task and target programming language.Traditional approaches combine program analysis with hand-coded rules and heuristics, while supervised learning typically requires large amounts of training data.Source: §1 · p. 1
• The proposed framework uses an existing FSLM with an extracted input, a designed prompt, and lightweight post-processing rather than a task-specific generation technique built from scratch.The paper presents this as its first systematic study of FSLM-based code-generation tools.Source: §1 · p. 2
• The study compares few-shot tools with Major, MeMo, and Randoop across code mutation, documentation-based oracle generation, and test-case generation.The tasks range from changing a few code tokens to generating entire test cases, and use either existing code or natural-language documentation as input.Source: §1 · p. 2
• FSLM-based tools complement traditional tools in code mutation, match MeMo for oracle generation, and exceed Randoop for test generation.The reported oracle-generation F1 scores are 0.59 for MeMo and 0.60 for the FSLM-based tool.Source: §1 · p. 2
• 14% coverage for a simple FSLM-based test generator versus Randoop’s 10%; combining Randoop- and FSLM-generated tests yields 16% coverage.The paper also reports that oracle generation scores are 0.59 for MeMo and 0.60 for the FSLM-based tool, while the mutation tool produces mutants Major cannot generate.Source: §1 · p. 2

2 BACKGROUND

• A generative language model predicts the next token from preceding tokens, and repeated next-token sampling can generate text.The initial training for downstream tasks different from next-token prediction is referred to as pre-training.Source: §2 · p. 2
• Few-shot learning uses a model for a task without fine-tuning, typically by placing between one and ten examples in its query.The paper calls its use of generative language models as few-shot learners few-shot learning with large-scale, pre-trained language models (FSLM), and uses Codex.Source: §2 · p. 2
• Prompts can include example input-output pairs followed by a new input, and their contents can strongly influence model results.The user can set temperature: higher values yield more diverse responses but make them less factual and precise, and repeated queries may differ.Source: §2 · pp. 2–3
• Table 1 presents representative input-output examples produced by the FSLM-based tools.The examples use the format x ==> y, where x is the prompt input and y is its output.Source: Table 1 · p. 3

3 METHODOLOGY

• The framework takes textual software artifacts, such as source code or documentation, and produces code snippets such as modified code, executable specifications, or test cases.Its three stages are instance extraction, prompt design, and post-processing.Source: §3 · p. 3
• Instance extraction selects artifact parts relevant to the task; this can keep inputs within model limits and reduce processing time.For code mutation, the framework extracts lines for which mutants are needed; the cited Codex models have a maximum input size of 4,096 tokens.Source: §3 · p. 3
• Prompt design combines an extracted instance with contextual information, such as task descriptions or examples, before the model generates a completion.The prompt includes invariant material and instance-specific material; post-processing can filter outputs that fail to compile or place predicted code in a task-specific template.Source: §3 · p. 3

3.1 Research Questions

• RQ1 asks how accurate model predictions are versus existing tools, RQ2 asks which prompts produce accurate results, and RQ3 asks how model size affects effectiveness.The questions address the human costs of hand-built tools, prompt influence on predictions, and computational resources used to train large models.Source: §3.1 · p. 3
• The prompt question matters because prompts are the main control for prediction quality, while model-size findings can inform computational-resource allocation.The study also motivates accuracy comparisons by the human costs of building traditional tools by hand.Source: §3.1 · pp. 3–4

3.2 Task 1: Code Mutation

• Code mutation injects programming mistakes into existing code to assess test-suite quality; the study compares its model-based tool with Major.Major applies built-in mutation operators and ensures its mutants compile.Source: §3.2 · p. 4, §3.2.1 · p. 4

3.2.1 Baseline Tool.

• For mutation, the extractor returns individual Java lines, and experiments focus on lines where Major applies a mutation.The model is asked to modify one line at a time and suggest multiple mutants.Source: §3.2.2 · p. 4, §3.2.3 · p. 4

3.2.2 Instance Extraction.

• The mutation prompt combines a task description with examples of code lines and their mutants; non-compiling suggestions are filtered out.Regular expressions extract proposed mutations from model completions.Source: §3.2.3 · p. 4, §3.2.4 · p. 4

3.3 Task 2: Generating Oracles from Natural Language Documentation

• Oracle generation targets metamorphic test oracles extracted from Javadoc, where related inputs are expected to produce outputs in a corresponding relationship.The example documentation describes equivalence between Arrays.toString(a) and Arrays.asList(a).toString() when a is not null.Source: §3.3 · pp. 4–5
• For oracle extraction, MeMo identifies likely oracle-bearing sentences, then translates them into code using a dependency parser and hand-coded rules; the study tests replacing that translation step with model queries.The FSLM receives the full comment containing a likely oracle-bearing sentence, identified using MeMo for comparison.Source: §3.3.1 · p. 5, §3.3.2 · p. 5

3.3.2 Instance Extraction.

• An oracle-generation instance is a method whose description probably contains an oracle; the prompt includes the entire comment containing a likely sentence.The instance extraction uses MeMo to identify those sentences for a fair baseline comparison.Source: §3.3.2 · p. 5

3.3.3 Prompt.

• Oracle examples pair a method signature and Javadoc with an analysis and executable equivalence; the model completes the analysis and oracle for the target instance.The default uses four examples, and temperature is set to zero because higher temperatures produced imprecise predictions.Source: §3.3.3 · p. 5

3.3.4 Post-processing.

• Oracle predictions are extracted with regular expressions, checked for compilation, and have class names expanded; evaluation uses 299 manually extracted oracles from 5,000 methods across nine Java projects.Oracle descriptions range from 3 to 500 words, and oracle code from 3 to 81 tokens.Source: §3.3.4 · p. 5, §3.3.5 · p. 5

3.4 Task 3: Test Case Generation

• The study generates unit tests for methods under test, comparing the model with Randoop while having each generate 100 test cases per method.Randoop is invoked with the target method and selected helper methods.Source: §3.4.1 · pp. 5–6

3.4.2 Instance Extraction.

• A test-generation instance is a randomly selected public method from a Java class.The extractor takes the class as input and produces its list of public methods.Source: §3.4.2 · p. 6
• The default prompt gives a brief task description and one example containing helper methods, the method under test, and a test case.One example is used because state-of-the-art FSLMs support bounded prompt sizes.Source: §3.4.3 · p. 6

3.4.3 Prompt.

• The model receives the target method and generates 100 candidates through repeated queries at temperatures 0.0 to 0.9 in 0.1 steps, with 10 queries per temperature.The single example may affect the generated test, so the study compares example-selection strategies.Source: §3.4.3 · p. 6

3.4.4 Post-processing.

• Generated candidates are inserted into a test-class template, then non-compiling tests and duplicates are discarded; benchmarks use 18 methods, two randomly selected from each of nine projects.The 18 methods are listed in Table 5, while the projects are listed in Table 2.Source: §3.4.4 · p. 6, §3.4.5 · p. 6

• Section 4 presents answers to the paper’s three research questions, with broader results and impact discussed in Section 5.Source: §4 · p. 6

4.1.1 Code Mutation.

• 2,721 mutants were generated by the FSLM tool versus 2,810 by Major; 62.5% of model-generated mutants compiled, and 90% of a 30-mutant sample changed behavior.About 18% of FSLM-generated mutants overlapped with Major's; the manual sample's remaining 10% either preserved semantics or had unclear effects.Source: §4.1.1 · p. 6, Table 3 · p. 7
• The model generated a more diverse set of mutants than Major, including edits outside Major's built-in operators; 7/30 sampled mutants replaced an expression with null.The authors conclude that the FSLM tool generated many useful mutants with minimal effort, despite not reproducing exactly the same mutants as Major.Source: §4.1.1 · p. 7

4.1.2 Generating Oracles from Natural Language Documentation.

• The oracle prompt produced only one oracle per method, limiting cases where multiple oracles were valid; some model misses involved invented code.The authors suggest prompting for a list or querying multiple times; two of six cases where MeMo succeeded involved the model inventing code.Source: §4.1.2 · p. 7
• FSLM oracle generation achieved an F1-score of 0.60 versus MeMo's 0.59, with precision of 0.82 versus 0.64.The comparison used a ground truth of 299 oracles across the studied libraries.Source: Table 4 · p. 7, §4.1.2 · p. 7

4.1.3 Test Case Generation.

• 14% vs. 10% line coverage: the FSLM test generator exceeded Randoop while generating fewer tests (202 vs. 682) averaging 11 vs. 31 lines.Randoop generated tests for all 18 methods, while the model generated compilable tests for 16; the two methods where Randoop achieved higher coverage were those with no compilable model tests.Source: Table 5 · p. 8, §4.1.3 · p. 8
• 16% line coverage: combining FSLM and Randoop tests exceeded either tool alone; for Colt's quantiles method, the combination reached 39%.For that method, FSLM tests achieved 29% line coverage and Randoop tests 26% individually.Source: §4.1.3 · p. 8

4.1 RQ1: Accuracy

• Across the three tasks, FSLM tools were on par with or complemented traditional tools, and performed better for test generation.The authors summarize the tools as complementary for mutation, on par for oracle generation, and better for test generation.Source: §4.1.3 · p. 8

4.2.1 Code Mutation.

• Mutation prompts with only a natural-language description produced inconsistent formatting and no usable results, even after adding output-format instructions.Examples therefore played a large part in solving this task with an FSLM.Source: §4.2.1 · p. 8
• Mutation Ex-only prompts had 5% fewer compilable mutants than the default, while adding a brief task description was an easy way to improve performance.The description in this task was a single sentence.Source: §4.2.1 · p. 9
• Bad-ex mutation prompts slightly increased overlap with Major and the percentage of compilable mutants compared with the default prompt.The authors describe this result as surprising because the examples were deliberately chosen to worsen predictions.Source: §4.2.1 · p. 9

4.2.2 Generating Oracles from Natural Language Documentation.

• For oracle generation, NL-only prompts produced incorrectly formatted responses and no usable results; other prompt styles did not significantly affect accuracy.Examples appeared necessary, while removing the natural-language description had negligible effect; bad examples performed worse than other styles.Source: §4.2.2 · p. 9, Figure 5 · p. 9

4.2.3 Test Case Generation.

• 14% line coverage: the default test-generation prompt exceeded NL-only and Ex-only at 12% each, and Bad-ex at 8%.The supplied Table 6 reports test-generation prompt variants and a smaller model.Source: Table 6 · p. 9, §4.2.3 · p. 9

4.2.2 Generating Oracles from Natural Language Documentation.

• A general task description yielded an F1 score of 0.54 for oracle generation, substantially worse than the default prompt.The authors suggest example quality matters more than the natural-language description for this task.Source: §4.2.2 · p. 9

4.2.3 Test Case Generation.

• For test generation, a natural-language description was more important than an input-output example (12% vs. 9%), and a same-class example outperformed an unrelated one (14% vs. 8%).The prompt always included the method under test and helper constructors and methods.Source: §4.2.3 · p. 9

4.2 RQ2: Impact of Prompt

• Across tasks, adding a brief natural-language description helped or did not hurt, while suitable examples were crucial for effective predictions.Source: §4.2.3 · p. 9

4.3 RQ3: Impact of Model Size

• The experiments use Codex Davinci by default and repeat them with the smaller Cushman model, motivated by the computational price of larger models.Source: §4.3 · p. 9

4.3.1 Code Mutation.

• 2,721 to 2,487 generated mutants and 62.5% to 52.8% compiling mutants show lower code-mutation effectiveness with the smaller model.The authors conclude that a larger model is beneficial for code mutation.Source: §4.3.1 · pp. 9–10

4.3.2 Generating Oracles from Natural Language Documentation.

• 0.58 versus 0.60 F1 shows that the smaller model performs nearly identically to the larger model for oracle generation.The authors note that some tasks can be handled almost as well by smaller models.Source: §4.3.2 · p. 10

4.3.3 Test Case Generation.

• 10% line coverage with the smaller model is four percentage points below the larger model and about the same as Randoop.Source: §4.3.3 · p. 10

4.3 RQ3: Impact of Model Size

• Larger models improve effectiveness, or at least do not reduce it, across all three tasks; the oracle-generation effect is small, so model size should be selected with computational cost in mind.The authors recommend carefully selecting model size for each task because large models are computationally costly.Source: §4.3.3 · p. 10

Prompt design

• Prompt design still has open questions about mining examples, alternating them across queries, and sensitivity to data format.Source: §5 · p. 10

Model size

• Large models can cost hundreds or millions of dollars to train, may be hard to use or unavailable publicly, and more accessible tools would benefit from improved smaller open models.Source: §5 · p. 10

Integrating FSLM-based and traditional tools

• Different precision-recall tradeoffs in oracle generation motivate exploring blends of FSLM-based and traditional techniques.Source: §5 · p. 10

Threats to Validity

• Findings may not generalize beyond the evaluated models, three tasks, or Java-based tools.The authors note that many other code-generation tasks remain unstudied.Source: §5 · p. 10

Studies of neural models of code

• Prior neural-model studies evaluate related models [55], compare sizes within one architecture [15], or fine-tune transformers for similar tasks [40].Source: §6 · p. 10

Language models in software engineering

• Related software-engineering work includes masked-token mutation [18], code generation from natural-language descriptions [7] [15], and program synthesis using a context bank of examples [32].The paper contrasts its end-to-end generative mutant approach with single-token masking and notes that the cited code-generation studies do not compare against traditional tools.Source: §6 · p. 10

Generative language models in general

• Although generative language models have been studied across tasks, the authors know of no other systematic study of few-shot models across different code-generation tasks [13] [50].Source: §6 · p. 10

Neural software analysis

• Unlike neural software-analysis approaches that target specific problems, this study tests a general-purpose generative model against non-neural code-manipulation tools.Source: §6 · p. 10

7 CONCLUSIONS

• Codex-based tools complement, match, or exceed three traditional tools across three code-generation tasks, and the methodology for creating them is relatively simple.Source: §7 · p. 11
• The study compares Codex [15] against traditionally built tools for code mutation, oracle generation, and test-case generation.Source: §7 · p. 11
• The authors describe these results as first steps toward applying few-shot learning to software-engineering problems.Source: §7 · p. 11