AceCoder: Utilizing Existing Code to Enhance Code Generation

1 INTRODUCTION

• AceCoder pairs an intermediate preliminary, such as test cases, with similar retrieved programs to address requirement understanding and code implementation.LLMs first produce the preliminary and then generate code; retrieved examples supply implementation content.Source: §1 · pp. 1–2
• The paper frames code generation as requiring both requirement understanding and implementation, including interpreting underspecified requests and mastering grammar, algorithms, and libraries.Requirements may omit input-output details and exceptions, while implementation remains difficult even for human developers.Source: §1 · pp. 1–2
• Pass@1 gains over few-shot prompting reach 56.4% in MBPP, 70.7% in MBJP, and 88.4% in MBJSP.The paper also reports that AceCoder outperforms retrieval-based baselines and that human developers prefer its programs.Source: §1 · p. 2
• Pass@1 gains over retrieval-based baselines reach 13.1% in MBPP, 23.44% in MBJP, and 15.8% in MBJSP.AceCoder is evaluated with CodeGeeX, CodeGen, and InCoder, which range from 6B to 13B parameters.Source: §1 · p. 2
• Human developers prefer AceCoder-generated programs, with reported advantages in correctness, code smell, and maintainability.The paper reports gains across CodeGeeX-13B, CodeGen-6B, and InCoder-6B.Source: §1 · p. 2
• Experiments report that all three AceCoder modules are necessary and that the proposed designs outperform multiple alternatives.The study also compares AceCoder with prompting and retrieval-based baselines across Python, Java, and JavaScript benchmarks.Source: §1 · p. 2

2 MOTIVATING EXAMPLES

• Few-shot prompting misses an essential repeated-character scenario, while AceCoder's test cases cover boundary inputs and accompany a correct program.The example uses generated test cases to clarify the requirement before code generation.Source: §2 · pp. 2–3
• A few-shot program uses a wrong condition for identifying lowercase letters joined by an underscore, illustrating the implementation challenge.The example is based on a real-world benchmark requirement.Source: §2 · p. 3
• Retrieved programs ranked by BM25 include relevant regular-expression content such as re.search, which can inform implementation.The motivating example sorts retrieved programs in descending BM25 score.Source: §2 · p. 3, Figure 2 · p. 3
• With only two example slots, redundant top matches can crowd out a more informative pattern example; the selector instead chooses complementary programs.The example contrasts repeated re.search examples with a program showing a relevant regular expression.Source: §2 · p. 3

3.1 An Overview

• AceCoder retrieves similar requirement-code pairs, converts them into requirement-preliminary-code examples, and prompts LLMs to generate a preliminary before the final code.A preliminary is a software artifact, such as test cases, that clarifies the requirement.Source: §3.1 · p. 3

3.2 Example Retrieval

• Because LLM input lengths and inference budgets limit examples, AceCoder selects a few programs from the retrieved set.The retrieval stage separates finding similar programs from selecting a smaller set for the prompt.Source: §3.2 · p. 3, §3.2.2 · p. 4

3.2.1 Retriever.

• The retriever uses BM25 over training-data requirements to return programs associated with lexically similar requirements.The paper uses Lucene and describes BM25 as a lightweight bag-of-words retrieval function.Source: §3.2.1 · pp. 3–4

3.2.2 Selector.

• The selector reduces redundancy by choosing examples with informative matched n-grams and decaying matches already covered by selected examples.It repeats selection until the example limit is reached; the paper notes that the selector is practical when the candidate and example counts are small.Source: §3.2.2 · pp. 4–5

3.3 Prompt Construction

• Prompt construction uses triple examples of requirements, preliminaries, and code to encourage the model to produce an intermediate preliminary before its answer.The preliminary is a software artifact for clarifying the requirement.Source: §3.1 · p. 3, §3.3 · p. 5

3.4 Code Generation

• In code generation, LLMs use the prompt examples to produce a preliminary and then generate code based on that preliminary and the input requirement.The illustrated output first generates test cases, which provide information such as input-output formats and invalid inputs.Source: §3.4 · p. 5

4.1 Research Questions

• The study asks how AceCoder compares with prompting and retrieval-based methods, whether developers prefer its code, and how its modules and design choices contribute to performance.The study describes six research questions, while the supplied list specifies comparison, human review, module contribution, and design questions.Source: §4 · p. 5, §4.1 · pp. 5–6

4.2 Evaluation Datasets and Metrics

• The evaluation uses MBPP in Python and MBJP and MBJSP in Java and JavaScript, respectively.The benchmarks contain crowd-sourced programming problems; Table 1 presents dataset statistics.Source: §4.2.1 · p. 6, Table 1 · p. 6

4.2.2 Metrics.

• Pass@k is the percentage of requirements for which at least one of k generated programs passes all test cases, with k set to 1, 3, or 5.The study omits match-based metrics such as BLEU because they are poor measures of program functionality.Source: §4.2.2 · p. 6

4.3 Comparison Baselines

• The prompting baselines are zero-shot, few-shot, and chain-of-thought prompting.Few-shot prompting uses randomly selected requirement-code examples, while chain-of-thought adds intermediate natural-language reasoning.Source: §4.3 · p. 6
• The retrieval-based comparisons include REDCODER, which fine-tunes PLBART using retrieved programs, and Jigsaw, which inserts retrieved programs from API documentation into prompts.Both are compared as approaches that augment code generation with retrieved information.Source: §4.3 · p. 6

4.4 Base Large Language Models

• The base models are CodeGeeX with 13 billion parameters, CodeGen-Multi-6.1B, and InCoder with 6.7 billion parameters.The authors leave closed-source GPT models to future work because dynamic updates could affect experimental fairness and reproducibility.Source: §4.4 · p. 6

4.5 Implementation Details

• AceCoder retrieves up to 20 programs and selects three examples, matching the example count used by baselines.The retrieval corpus is each dataset's training data, and ground truths are excluded from retriever outputs.Source: §4.5 · p. 6
• The analyzer converts retrieved training examples into requirement-test-case-code triples, and generation uses nucleus sampling with temperature 0.8 and top-p 0.95.Baseline sampling settings match AceCoder's; the supplied passage reports maximum generation lengths of 400, 500, and 500.Source: §4.5 · pp. 6–7

RQ1: Performance against prompting techniques

• Pass@1 gains over the SOTA baseline reached 56.4% in MBPP, 70.7% in MBJP, and 88.4% in MBJSP.AceCoder also improves CodeGeeX-13B by up to 88.4%, CodeGen-6B by up to 65.5%, and InCoder-6B by up to 57.5%, and is effective across Python, Java, and JavaScript.Source: §5 · p. 7

RQ2: Performance against retrieval-based models

• Pass@1 gains over Jigsaw reached 13.1% in MBPP, 23.44% in MBJP, and 15.8% in MBJSP.AceCoder outperforms the retrieval-based baselines across all three benchmarks.Source: §5 · p. 7, Table 3 · p. 8

RQ4: Contributions of the modules

• All three modules are necessary for AceCoder to achieve its best performance, and adding modules improves CodeGeeX across the benchmarks.The retriever alone brings Pass@1 improvements of 17.6% in MBPP, 40.4% in MBJP, and 67.2% in MBJSP; adding the analyzer yields improvements of 31.1%, 70.7%, and 88.4%, respectively.Source: Table 4 · p. 8, §5 · pp. 8–9

RQ3: Human evaluation

• Human evaluation found AceCoder better than few-shot prompting by 61.8% in correctness, 33.7% in code smell, and 13.8% in maintainability.The evaluation reports these improvements as statistically significant, with all p-values substantially below 0.05.Source: Table 5 · p. 8, §5 · p. 8

RQ5: Alternative module designs

• Across design alternatives, AceCoder's choices perform better: dense retrieval slightly drops performance, BLEU favors shorter examples, and test cases capture details signatures miss.Test cases are presented as more suitable preliminaries than APIs or method signatures because they cover edge cases as well as input-output formats and functionality.Source: §5 · p. 9

6.1 AceCoder vs. CoT prompting

• CoT prompting slightly improves on few-shot prompting but performs worse than zero-shot prompting; the authors argue AceCoder is more promising because test cases are less ambiguous.AceCoder uses software artifacts such as test cases as intermediate preliminaries, unlike CoT’s natural-language steps.Source: §6.1 · pp. 9–10

6.2 AceCoder vs. Rank Techniques

• AceCoder and rank techniques address different goals, so the authors omit direct comparisons and suggest using ranking after AceCoder generates multiple programs.AceCoder targets generation accuracy; rank techniques select the best among multiple outputs.Source: §6.2 · p. 10

6.3 Threats to Validity

• To address generalizability concerns, experiments use three real-world benchmarks spanning Python, Java, and JavaScript, Pass@k, and 6B–13B models, averaging three runs.The authors identify generalizability of experimental results as a threat to validity.Source: §6.3 · p. 10
• Less relevant retrieved programs may hurt performance; the authors say AceCoder degrades at worst to few-shot prompting and is usually superior.Source: §6.3 · p. 10

• Code LLMs include standard language models trained with next-token prediction and instruction-tuned models adapted to follow user instructions.The section gives examples of both model categories, including CodeGen, CodeGeeX, InCoder, ChatGPT, and Code Alpaca.Source: §7 · p. 10
• Zero-shot prompting combines an instruction with a requirement, while few-shot prompting adds requirement-code examples to the prompt.Prompting techniques use a special prompt to leverage LLMs for code generation.Source: §7 · p. 10
• CoT prompting elicits natural-language reasoning before code, while prompting methods designed for natural-language generation bring only slight improvements in code generation.The section also names self-consistency and least-to-most prompting as follow-up methods.Source: §7 · pp. 10–11

• Across three benchmarks, AceCoder significantly outperforms state-of-the-art baselines with guided code generation and example retrieval.Source: §8 · p. 11
• Guided code generation has LLMs produce a preliminary, such as test cases, before code to clarify requirements and guide generation.Source: §8 · p. 11
• Future work will explore teaching LLMs to use unseen frameworks without retraining.Source: §8 · p. 11