PaperlayerView exact revision on arXiv

Source-linked AI summary

What Makes Good In-context Demonstrations for Code Intelligence Tasks with LLMs?

Shuzheng Gao, Xin-Cheng Wen, Cuiyun Gao, Wenxuan Wang, Hongyu Zhang, Michael R. Lyu

arXiv:2304.07575v2cs.SE

TL;DR

The paper addresses the lack of systematic evidence about how to construct effective demonstrations for code-intelligence ICL. It experimentally studies selection, ordering, and demonstration count across three tasks, finding that all three substantially affect performance and that carefully designed demonstrations yield large gains over common construction methods.

  • Problem

    The paper investigates how to construct good demonstrations when ICL performance strongly depends on demonstration quality.

  • Method

    The study compares demonstration selection, ordering, and quantity across code summarization, bug fixing, and program synthesis, including diversity, retrieval, ordering, and generalizability analyses.

  • Results

    Carefully designed demonstrations improve BLEU-4 by at least 9.90% for code summarization, EM by 175.96% for bug fixing, and EM by 50.81% for program synthesis over widely used methods.

  • Takeaways & Limitations

    Effective demonstrations should balance similarity and diversity, place similar examples near the prompt end, and increase example count without triggering truncation.

  • Takeaways & Limitations

    The study uses closed-source OpenAI models, creating potential data-leakage concerns, and evaluates only three representative tasks.

Abstract

from arXiv · show

Pre-trained models of source code have gained widespread popularity in many code intelligence tasks. Recently, with the scaling of the model and corpus size, large language models have shown the ability of in-context learning (ICL). ICL employs task instructions and a few examples as demonstrations, and then inputs the demonstrations to the language models for making predictions. This new learning paradigm is training-free and has shown impressive performance in various natural language processing and code intelligence tasks. However, the performance of ICL heavily relies on the quality of demonstrations, e.g., the selected examples. It is important to systematically investigate how to construct a good demonstration for code-related tasks. In this paper, we empirically explore the impact of three key factors on the performance of ICL in code intelligence tasks: the selection, order, and number of demonstration examples. We conduct extensive experiments on three code intelligence tasks including code summarization, bug fixing, and program synthesis. Our experimental results demonstrate that all the above three factors dramatically impact the performance of ICL in code intelligence tasks. Additionally, we summarize our findings and provide takeaway suggestions on how to construct effective demonstrations, taking into account these three perspectives. We also show that a carefully-designed demonstration based on our findings can lead to substantial improvements over widely-used demonstration construction methods, e.g., improving BLEU-4, EM, and EM by at least 9.90%, 175.96%, and 50.81% on code summarization, bug fixing, and program synthesis, respectively

I. INTRODUCTION

The paper studies how to construct effective in-context demonstrations for code intelligence, focusing on example selection, ordering, and quantity. Experiments across code summarization, bug fixing, and program synthesis show that demonstration design strongly affects performance.

  • The study systematically examines demonstration selection, ordering, and number across code summarization, bug fixing, and program synthesis.
  • Similarity and diversity in demonstration selection improve performance and make ICL predictions more stable.
  • Placing similar examples at the end of a prompt usually produces better ICL results.
  • Increasing demonstration count can help until input-length limits truncate examples, which is especially important for longer code.
  • A carefully designed demonstration improves BLEU-4 by at least 9.90% for code summarization, EM by 175.96% for bug fixing, and EM by 50.81% for program synthesis.
  • The paper presents a first systematic study of effective demonstration construction for code intelligence tasks.

II. BACKGROUND

This section introduces large language models and in-context learning as training-free approaches for code intelligence. It frames demonstration quality as a central design issue involving selection, ordering, and quantity.

  • A. Large Language Models: Large language models learn from a few examples placed in context, enabling in-context learning without parameter updates.
  • B. In-context Learning: Few-shot ICL typically uses fewer than 50 examples, while zero-shot ICL uses no demonstration examples.
  • B. In-context Learning: An ICL prompt concatenates reconstructed demonstration examples with a test input, which the language model uses to predict the output.
  • B. In-context Learning: Prior work identifies demonstration selection, ordering, and number as the main design factors affecting ICL performance.
  • B. In-context Learning: Task-level demonstrations are shared across test samples, whereas instance-level demonstrations select different examples for different test samples.

III. EXPERIMENTAL EVALUATION

The evaluation is organized around research questions about which demonstrations to select, how to order them, how many to use, and whether the findings generalize across models.

  • A. Research Questions: The experiments ask whether similar and diverse examples improve ICL and whether retrieval methods affect similarity-based selection.
  • A. Research Questions: The study compares random ordering with similarity-based ordering to assess how example arrangement influences ICL.
  • A. Research Questions: The number-of-examples question tests whether adding demonstrations improves performance or instead creates input-length problems.
  • A. Research Questions: A fourth question evaluates whether findings from selection, ordering, and quantity transfer to different large language models.

B. Evaluation tasks

The evaluation covers code summarization, bug fixing, and program synthesis using established datasets and task-specific metrics. These tasks span generation, repair, and synthesis settings.

  • The study evaluates three code intelligence tasks: code summarization, bug fixing, and program synthesis.
  • Code summarization generates useful comments for a given code snippet and is evaluated with BLEU-4, ROUGE-L, and METEOR.
  • Table I reports statistics for the benchmark datasets used in the evaluation.
  • The summarization evaluation uses CodeSearchNet and TLCodeSum datasets.
  • Bug fixing predicts corrected code from buggy code and natural-language fix guidance using the B2F dataset.
  • Bug-fixing performance is measured with Exact Match and BLEU-4.

3) Program Synthesis:

The program synthesis evaluation uses CoNaLa and assesses generated code with exact-match, code-similarity, syntax, and dataflow metrics under deterministic API settings.

  • CoNaLa provides 2,889 Python intent–code pairs, with 2,389 training samples and 500 testing samples.
  • Program synthesis performance is evaluated using Exact Match, CodeBLEU, Syntax Match, and Dataflow Match.Exact Match requires generated code to be identical to the target, while CodeBLEU incorporates syntactic and semantic information.
  • The experiments use OpenAI Codex for the first three research questions and additionally GPT-3.5 and ChatGPT for the fourth.API temperature, frequency penalty, and presence penalty are set to 0 for deterministic outputs.
  • The study uses four demonstrations in the first two research questions and examines demonstration-count effects separately in the third.

IV. EXPERIMENTAL RESULTS

The experimental study compares task-level and instance-level demonstration selection across code intelligence tasks, including random, similarity-based, and diversity-based strategies.

  • Experimental design: The study evaluates demonstration selection across three code intelligence tasks using task-level and instance-level settings.Task-level methods select a shared group for the test set, whereas instance-level methods retrieve examples for each test sample.
  • Task-level demonstration: Task-level selection compares randomly selected groups with diversity-oriented selection to assess performance and variation across example groups.
  • Instance-level demonstration: Instance-level selection treats demonstration construction as retrieval and compares BM-25, SBERT, UniXcoder, and CoCoSoDa.Dense methods use pretrained code or text representations and cosine similarity to retrieve training samples.
  • Evaluation protocol: Results are averaged over three demonstration orders, while coefficient of variation measures sensitivity to ordering.The coefficient of variation is calculated as σ/µ, with lower values indicating smaller variation.

2) Analysis:

The analysis finds that diverse and instance-specific demonstrations improve ICL performance and stability, while BM-25 is a strong retrieval baseline but remains below an oracle.

  • Demonstration selection: Diversity improves task-level demonstration performance and stability; on CSN, KmeansRND exceeds Random by 5.45% BLEU-4, 7.25% ROUGE-L, and 6.80% METEOR.On one code-summarization distribution, the best-to-worst BLEU-4 gap falls from about 2.5 for Random to about 0.6 for KmeansRND.
  • Demonstration selection: BM-25 achieves an average program-synthesis EM of 18.53, outperforming SBERT, UniXcoder, and CoCoSoDa by 14.88%, 15.81%, and 13.68%, respectively.The result supports BM-25 as a simple and effective instance-level baseline.
  • Demonstration selection: Instance-level demonstrations improve the best task-level exact-match results by at least 141.97% on B2Fmedium and 193.64% on B2Fsmall.The comparison is reported across the bug-fixing datasets.
  • Demonstration stability: Task-level demonstrations are generally more sensitive to example order than instance-level demonstrations.The reported comparison uses coefficient of variation across different example orders.
  • Limitations: Retrieval methods still leave a substantial gap to the Oracle, indicating room to improve semantic-similarity selection.

B. RQ2: Demonstration Order

The ordering analysis compares random, similarity-based, and reverse-similarity arrangements of demonstrations relative to the test sample.

  • B. RQ2: Demonstration Order: Similarity ordering achieves the best result in 62.96% of evaluated metric-task combinations.This corresponds to 17 of 27 metric-task cases.
  • B. RQ2: Demonstration Order: Different demonstration orders can materially change ICL performance, making ascending similarity the stronger option in most cases.Some cases still show both similarity-based strategies below the average result from random ordering.

C. RQ3: The Number of Demonstration Examples

The number of demonstration examples initially improves ICL performance, but excessive examples can trigger truncation and degrade results. The authors therefore recommend balancing performance gains against input length and cost, with four examples as a practical choice.

  • 2) Analysis:: Performance initially increases with more examples, but bug fixing peaks at 32 examples and drops sharply at 64 because of truncation.For B2Fsmall, truncation affects 2.33% of examples at 32 demonstrations and more than 80% at 64.
  • 2) Analysis:: Program synthesis continues improving and stabilizes beyond 32 examples because its shorter samples avoid truncation even at 64 examples.The cited sample lengths are 557, 492, and 101 characters for CSN, B2Fsmall, and CoNaLa, respectively.
  • 2) Analysis:: Four demonstration examples achieve 96.48%, 97.80%, and 94.80% of the best performance on the three tasks for EM, BLEU-4, and CodeBLEU, respectively.The authors recommend four examples as a cost-conscious balance between performance and prompt length.
  • 2) Analysis:: More examples do not always improve performance because code length makes truncation especially likely, while larger prompts also increase API cost and inference time.The paper frames example count as a trade-off between performance, truncation, cost, and latency.
  • 2) Analysis:: Finding 5 recommends using four examples to save costs while avoiding the truncation problem.

1) Experimental setup:

The generalization experiments test the findings across GPT-3.5 and ChatGPT using multiple selection, ordering, and demonstration-count settings. Results show that diversity, similarity-based retrieval, favorable ordering, and careful example counts consistently matter across these models.

  • 2) Analysis:: KmeansRND+Random outperforms Random+Random and produces a more stable prediction distribution across example groups.On GPT-3.5, it improves Random+Random by 6.24% for CB and 10.39% for EM.
  • 2) Analysis:: BM-25+Similarity improves BM-25+Random on all metrics and both LLMs, raising average EM by 5.56% on GPT-3.5 and 5.42% on ChatGPT.
  • 2) Analysis:: BM-25 matches or exceeds UniXcoder on GPT-3.5, improving CB by 2.50% and EM by 4.88%.The result supports BM-25 as a simple and effective selection method for these models.
  • 2) Analysis:: With 128 examples, 25.05% of examples suffer truncation and EM suddenly degrades, confirming that example count must be controlled.
  • 2) Analysis:: ChatGPT’s carefully designed demonstration beats the baseline by at least 10.59%, 294.57%, and 51.06% on code summarization BLEU-4, bug fixing EM, and program synthesis EM.The comparison is against the widely used baseline demonstration construction method.

A. Implications of Findings

The paper translates its findings into guidance for researchers and developers, emphasizing retrieval, diversity, ordering, prompt length, and task demonstrations. It also identifies data leakage, task selection, and model selection as threats to validity.

  • A. Implications of Findings: Researchers should study code representations and task-aware selection because current retrieval models remain far from the Oracle in selecting semantically similar examples.
  • A. Implications of Findings: Ordering similar examples at the end performs relatively better in many cases, but the improvement is inconsistent and motivates better automatic ordering methods.
  • A. Implications of Findings: Code length limits prompt capacity and raises computation and time costs, motivating program slicing and reduction techniques for ICL.
  • A. Implications of Findings: Developers should include demonstrations, use BM-25 retrieval when labeled training data exists, diversify task-level examples, and place similar samples near the prompt’s end.
  • B. Threats to Validity: The study’s validity is threatened by possible data leakage from closed-source models and by its selection of only three tasks and three LLMs.The authors plan to evaluate additional task types and models in future work.

VI. RELATED WORK

Related work traces the progression from pre-trained code models and large language models to in-context learning for code intelligence. The paper positions itself as a broader empirical study of demonstration construction across selection, ordering, and example count.

  • VI. RELATED WORK: Pre-trained code models including CodeBERT, CodeT5, and CodeGPT have supported software engineering tasks, while larger models such as Codex, ChatGPT, GPT-4, AlphaCode, CodeGen, and CodeGeeX extend programming capabilities.
  • VI. RELATED WORK: Large language models introduced emergent abilities such as learning from a few contextual examples without parameter updates.
  • VI. RELATED WORK: Prior code-intelligence studies applied ICL to program repair and other tasks, but mainly evaluated one or two tasks without fully examining demonstration construction.
  • VI. RELATED WORK: This paper experimentally studies demonstration selection, ordering, and number across code intelligence tasks and reports that carefully designed demonstrations substantially outperform simpler ones.
Loading 2304.07575v2…