PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generation

Zhiqiang Yuan, Junwei Liu, Qiancheng Zi, Mingwei Liu, Xin Peng, Yiling Lou

arXiv:2308.01240v1cs.CLcs.AI

TL;DR

Evaluations of open-source instructed LLMs on diverse software-engineering code tasks have been limited. This study evaluates 10 such models across four tasks and multiple settings, finding strong but variable performance, with fine-tuning yielding further gains and competitive results against baselines.

  • Problem

    Evaluations of instructed models in software engineering have largely focused on code generation, while commercial models lack transparency and reproducibility, leaving performance on diverse code tasks unclear.

    Source: §I · p. 1

  • Method

    The study evaluates 10 open-source instructed LLMs and five baselines on four code tasks in zero-shot, few-shot, and fine-tuning settings.

    Source: §I · p. 2

  • Results

    Zero-shot instructed LLMs sometimes beat small task-finetuned SOTA models; examples help most tasks but can hurt, while fine-tuning improves performance and beats small SOTA and similar-scaled untuned LLMs.

    Source: §I · p. 2

  • Takeaways & Limitations

    The findings inform model and usage recommendations, performance-cost trade-offs, and future research directions for instructed LLMs on code tasks.

    Source: §I · p. 2

    The findings may be limited by the prompts used, despite common design practices and refinement in a five-item pilot.

    Source: §VI · p. 11

Abstract

from arXiv · show

In this work, we evaluate 10 open-source instructed LLMs on four representative code comprehension and generation tasks. We have the following main findings. First, for the zero-shot setting, instructed LLMs are very competitive on code comprehension and generation tasks and sometimes even better than small SOTA models specifically fine-tuned on each downstream task. We also find that larger instructed LLMs are not always better on code-related tasks. Second, for the few-shot setting, we find that adding demonstration examples substantially helps instructed LLMs perform better on most code comprehension and generation tasks; however, the examples would sometimes induce unstable or even worse performance. Furthermore, we find widely-used BM25-based shot selection strategy significantly outperforms the basic random selection or fixed selection only on generation problems. Third, for the fine-tuning setting, we find that fine-tuning could further improve the model performance on downstream code comprehension and generation tasks compared to the zero-shot/one-shot performance. In addition, after being fine-tuned on the same downstream task dataset, instructed LLMs outperform both the small SOTA models and similar-scaled LLMs without instruction tuning. Based on our findings, we further present practical implications on model and usage recommendation, performance and cost trade-offs, and future direction.

I. INTRODUCTION

The paper addresses limited evaluation of open-source instruction-tuned LLMs across software-engineering tasks by studying them in zero-shot, few-shot, and fine-tuning settings. It reports strong but variable performance across settings and practical implications for model use and cost.

I. INTRODUCTION

  • Prior evaluations of instructed models in software engineering largely focus on code generation, and commercial models lack transparency and reproducibility, leaving their performance on diverse code tasks unclear.The paper presents its evaluation as a first attempt to address this gap for open-source instructed LLMs.Source: §I · p. 1
  • Zero-shot instructed LLMs are competitive on code tasks and sometimes outperform small task-fine-tuned SOTA models, while larger or code-domain-instructed models are not always better.The study also finds that general-domain-instructed models do not necessarily lose to code-domain-instructed models.Source: §I · p. 2
  • Demonstration examples improve performance on most tasks but can make results unstable or worse; BM25 shot selection significantly outperforms random or fixed selection only on generation problems.The BM25 advantage is not significant over either basic strategy on classification problems.Source: §I · p. 2
  • Fine-tuning improves instructed LLM performance over zero-shot or one-shot use, and fine-tuned instructed LLMs outperform small SOTA models and similar-scaled LLMs without instruction tuning.Similar-scaled instructed LLMs vary in memory and time costs; they do not necessarily use more memory than small SOTA models but take much longer for fine-tuning and inference.Source: §I · p. 2
  • The study evaluates 10 instructed LLMs against five baseline models on four representative code tasks across zero-shot, few-shot, and fine-tuning settings.The experiment used around 1,000 GPU-hours on one NVIDIA A800-80GB GPU.Source: §I · p. 2
  • The paper draws practical implications for model and usage recommendations, performance-cost trade-offs, and future directions.Its contribution includes comparisons with SOTA models across settings and recommendations for shot-selection strategies.Source: §I · p. 2

II. BACKGROUND

The background introduces instruction tuning, zero-shot and few-shot in-context learning, and parameter-efficient fine-tuning as concepts relevant to instructed LLMs.

II. BACKGROUND

  • Instruction tuning is supervised tuning of foundation models with instruction data to follow instructions and solve unseen tasks without demonstration examples.The paper depicts this process in Figure 1.Source: Figure 1 · p. 2, §II · p. 3
  • In-context learning uses task descriptions or demonstrations without updating model parameters; zero-shot uses instructions alone, while few-shot includes examples..Source: §II · p. 3
  • Parameter-efficient fine-tuning updates only a small number of parameters; LoRA applies a low-rank constraint to updated dense-layer matrices.The passage identifies adapter, prefix, prompt, and low-rank adaptation as PEFT techniques, and describes LoRA as popular in open-source LLM implementations.Source: §II · p. 3

III. EXPERIMENTAL SETUP

The experimental setup frames four research questions around zero-shot, few-shot, fine-tuned performance, and model costs, using four representative code comprehension and generation tasks.

III. EXPERIMENTAL SETUP

  • The study asks how instructed LLMs perform in zero-shot, few-shot, and further fine-tuning settings, and compares their inference and fine-tuning memory and time costs.The few-shot question also examines shot-selection strategies.Source: §III · p. 3

1) Tasks:

  • The four tasks are defect detection, clone detection, assertion generation, and code summarization.These are presented as representative code comprehension and generation tasks used in recent empirical studies.Source: 1) Tasks · p. 3
  • Defect detection tests code-correctness understanding by identifying security vulnerabilities, while clone detection tests code-similarity understanding.Examples of vulnerabilities include denial of service and injection.Source: 1) Tasks · p. 3
  • Assertion generation produces an assertion for a focal method and test prefix, while code summarization produces a natural-language description of a code snippet’s functionality.The tasks respectively reflect understanding test oracles and code intention.Source: 1) Tasks · p. 3

2) Metrics:

The metrics section specifies task-specific measures, uses ChatGPT to judge code summaries, and describes the sampled datasets used in the experiments.

2) Metrics:

  • Defect detection uses accuracy, clone detection uses F1, and assertion generation uses exact match.These metrics measure the ratio of defects identified, the harmonic mean of precision and recall, and the ratio of assertions identical to ground truth, respectively.Source: 2) Metrics · p. 3
  • For code summarization, ChatGPT compares generated summaries with ground truth and the preferred rate measures how often it prefers the generated summary.The prompt includes two manually crafted examples to mitigate potential position bias; Figure 2 shows a prompt example.Source: 2) Metrics · pp. 3–4, Figure 2 · p. 4

3) Datasets:

  • The datasets use sampled portions of the original task data, with 2,000 test items for each task except code summarization because ChatGPT API evaluation is costly.Table I summarizes the studied tasks, datasets, and metrics.Source: 3) Datasets · p. 4, Table I · p. 4

B. Studied Models

The study selects recent open-source instruction-tuned models across a range of sizes, foundations, and instruction domains, then describes their prompt components and task-specific instructions.

B. Studied Models

  • The study includes 10 recent open-source instruction-tuned LLMs released from March 1 to July 1, 2023, excluding closed-source models and models above 20B parameters.The exclusions reflect transparency, reproducibility, and resource constraints.Source: B. Studied Models · p. 4
  • The 10 models range from 6B to 16B parameters, use different foundation models, and include general-instruction and code-instruction tuning.The study also compares four small fine-tuned SOTA models and the similarly scaled, non-instruction-tuned CodeGen-6B.Source: B. Studied Models · p. 4

C. Prompt Design

  • Prompts comprise a system prompt, task instruction, and model-specific keywords that separate prompt parts.Not every model has a system prompt; the study includes one only when official documentation or repositories provide it.Source: C. Prompt Design · pp. 4–5
  • Task instructions specify the requested operation for defect detection, clone detection, assertion generation, or code summarization.The instructions request YES/NO classifications for detection tasks, an assertion using JUnit for assertion generation, and a method-level comment for summarization.Source: C. Prompt Design · p. 5

D. Experimental Procedure

The experiments compare zero-shot, one-shot, fine-tuned, and cost settings across code tasks, using shared datasets and specified model-specific training procedures.

D. Experimental Procedure

  • In zero-shot tests, the 10 instructed LLMs and CodeGen-6B receive prompts directly, while small SOTA models are fine-tuned because they are not applicable zero-shot.Their fine-tuned test performance is used for comparison.Source: D. Experimental Procedure · p. 5
  • The one-shot setting places one demonstration at the prompt beginning and compares fixed, random, and BM25-retrieved example selection.The study limits this comparison to one shot because two or more examples sometimes exceed model token limits.Source: D. Experimental Procedure · p. 5
  • For downstream fine-tuning, large models use LoRA with frozen original parameters, whereas all parameters of small SOTA models are tuned.All studied models are fine-tuned on the same training dataset for each task.Source: D. Experimental Procedure · p. 5
  • The cost comparison records each model’s memory and time costs during training and inference.The study runs 15 models across four tasks and the described settings, using around 1000 GPU-hours on one NVIDIA A800-80GB GPU.Source: D. Experimental Procedure · p. 5

E. Implementation Details

  • For fair comparison, the small SOTA models are retrained on the same training data used for instructed-model fine-tuning, following prior-work hyperparameters.LLM generation is made deterministic with sampling disabled, and the maximum input length is set to 2,048 tokens.Source: E. Implementation Details · pp. 5–6

IV. EXPERIMENT RESULTS

In zero-shot code tasks, instruction-tuned LLMs are competitive with fine-tuned small SOTA models, while instruction domain and model scale do not reliably predict stronger performance. The paper also frames its few-shot analysis around one-shot performance, instruction following, and shot selection.

A. Performance in Zero Shot (RQ1)

  • Instruction-tuned LLMs are competitive with, and sometimes outperform, small SOTA models fine-tuned for each task in zero-shot code comprehension and generation.Most instructed models outperform the SOTA models on clone detection and code summarization; on defect detection, WizardCoder-15B scores 54.4% versus 58.7% for SOTA.Source: Table IV · p. 6, A. Performance in Zero Shot (RQ1) · p. 6
  • CodeGen-6B performs extremely poorly in zero-shot tasks, generating snippets unrelated to the task, whereas instruction tuning improves zero-shot generalization.CodeGen-6B has a comparable parameter count but lacks instruction tuning.Source: A. Performance in Zero Shot (RQ1) · p. 6
  • General-instruction-tuned models can be competitive on code tasks, and code-specific instruction tuning does not necessarily produce better results.Alpaca-7B outperforms CodeAlpaca-7B on all studied code tasks except defect detection.Source: A. Performance in Zero Shot (RQ1) · p. 6
  • Increasing model scale sometimes yields marginal or negative zero-shot changes, with effects differing between generative and classification tasks.Dolly performs much worse on assertion generation and code summarization when scaled from 7B to 12B; Vicuna’s 7B-to-13B increase helps generation more than classification.Source: A. Performance in Zero Shot (RQ1) · p. 6

B. Performance in Few Shot (RQ2)

  • The few-shot analysis examines one-shot performance and instruction following against zero-shot results, as well as the effects of shot-selection strategies.The request describes this analysis but supplies no few-shot outcomes for this subsection.Source: B. Performance in Few Shot (RQ2) · p. 6

1) Comparison to zero-shot performance:

One demonstration improves performance in most evaluated cases, but examples can also produce unstable or worse outcomes. The reported input-length analysis shows a general decline in performance as inputs grow.

1) Comparison to zero-shot performance:

  • 25 of 40 model-task cases improve with one demonstration, and all studied models improve on assertion generation.For cases below 10% zero-shot performance, one shot consistently yields large gains; examples include ChatGLM-6B on defect detection, 7.1% to 54.2%, and Instruct-CodeGen-16B on code summarization, 9.0% to 41.0%.Source: Table V · p. 7, 1) Comparison to zero-shot performance · p. 7
  • One-shot examples sometimes cause unstable or worse performance; on clone detection, several models predict only yes or only no, making F1 invalid.The reported affected models are Vicuna-7B, Alpaca-7B, and StableLM-7B.Source: 1) Comparison to zero-shot performance · p. 7
  • For most models, performance trends downward as input length increases, with Figure 3 showing defect detection and assertion generation results.The x-axis is token count using each model’s tokenizer, and the y-axis is performance; the authors suggest limited ability to use longer contexts as one potential reason for few-shot instability.Source: 1) Comparison to zero-shot performance · p. 7, Figure 3 · p. 7

2) Instruction-following Capability:

In the one-shot setting, a demonstration substantially improves instruction-following ability, while BM25 selection is significantly better than fixed or random selection on generation tasks but not classification tasks.

2) Instruction-following Capability:

  • Here, instruction following means adhering to the requested response format, such as answering YES or NO or returning an assertion.The assessment focuses on format rather than response correctness.Source: 2) Instruction-following Capability · p. 7
  • Most instructed LLMs show reasonable instruction-following capability in zero-shot tasks, while CodeGen-6B performs extremely poorly.The authors report that instruction tuning improves instruction-following capability.Source: 2) Instruction-following Capability · p. 8, Table VII · p. 8
  • On defect detection, adding a demonstration example to the input improves ChatGLM-6B’s instruction-following capability from 16.2% in the zero-shot setting to 98.2% in the one-shot setting.Source: 2) Instruction-following Capability · p. 8
  • Adding a demonstration substantially improves instruction-following capability, potentially contributing to the one-shot performance gains reported elsewhere.The paper frames this connection as a possibility, not a demonstrated cause.Source: 2) Instruction-following Capability · p. 8

3) LLM Performance with different one-shot selection strategies:

  • The shot-selection comparison uses fixed, random, and BM25-similarity-based one-shot strategies and tests pairwise differences at a 0.05 significance level.The authors use paired t-tests and mark p-values below 0.05 as significant.Source: 3) LLM Performance with different… · p. 8, Table VIII · p. 8
  • BM25 selection significantly outperforms fixed and random selection on generation tasks, but shows no significant difference on classification tasks.The comparison covers assertion generation, code summarization, defect detection, and clone detection.Source: 3) LLM Performance with different… · p. 8, Table VI · p. 8

C. Fine-tuned Performance (RQ3)

Fine-tuning instructed LLMs improves task performance over zero-shot and one-shot settings, with the best performance on each task achieved after task-specific fine-tuning. The models adapt within a few epochs, and most outperform small SOTA models after fine-tuning.

C. Fine-tuned Performance (RQ3)

  • Task-specific fine-tuning gives instructed LLMs their best performance on each task, improving on zero-shot and one-shot settings.The comparison reports fine-tuned performance alongside zero-shot and one-shot results.Source: C. Fine-tuned Performance (RQ3) · p. 9
  • After fine-tuning on the same task dataset, most instructed LLMs outperform small SOTA models and CodeGen-6B, despite updating fewer parameters.The authors report fewer than 10M updated parameters for instructed LLMs versus more than 100M for small SOTA models; they suggest the instructed models’ existing capability may explain the result.Source: C. Fine-tuned Performance (RQ3) · p. 9, Table X · p. 9
  • Performance reaches a plateau after three fine-tuning epochs for the two illustrated models, ChatGLM-6B and CodeAlpaca-7B.The paper’s broader finding is that instructed LLMs adapt with a few epochs, such as three.Source: Figure 4 · p. 9, C. Fine-tuned Performance (RQ3) · p. 10

D. Costs (RQ4)

Instructed LLMs have varied memory and time costs, and larger models do not necessarily take longer. The paper recommends weighing these costs against task performance and available resources.

D. Costs (RQ4)

  • Similar-scaled instructed LLMs vary in memory and time costs, and larger models do not necessarily take longer for fine-tuning or inference.The comparison covers fine-tuning cost per epoch and zero-shot inference on 100 test items.Source: D. Costs (RQ4) · p. 10, Table XI · p. 11

V. IMPLICATIONS

  • The authors recommend WizardCoder, Vicuna, and ChatGLM as stable options across the studied tasks and settings.They present this as a practical recommendation, not a ranking of all instructed LLMs.Source: §V · p. 10
  • When data and resources are available, the authors recommend efficient fine-tuning because it can improve performance more consistently than in-context learning.They report adaptation with LoRA in only a few fine-tuning epochs.Source: §V · p. 10
  • For generation, the authors recommend similar demonstrations such as BM25-selected examples; for classification, they find no dominant selection strategy.They advise balancing example inclusion against longer prompts, which may reduce performance.Source: §V · p. 10
  • Small SOTA models may be safer when online latency or limited resources dominate, because instructed LLMs cost substantially more time and memory.The authors also note that instructed LLMs can perform better on code tasks when resources allow.Source: §V · p. 10, D. Costs (RQ4) · p. 10
  • Future work should improve evaluation, make tuning and inference more efficient, and build instructed LLMs for code comprehension and generation.Suggested evaluation improvements include practical benchmarks and sound metrics.Source: §V · p. 10

VI. THREATS TO VALIDITY

The paper’s findings are bounded by its prompts, tasks, datasets, and evaluation choices. It positions its study as a systematic evaluation of open-source instructed LLMs across representative code comprehension and generation tasks.

VI. THREATS TO VALIDITY

  • Results may depend on the prompts, tasks, datasets, and metrics used, limiting certainty about other prompts or generalization to other tasks and datasets.The authors report no overlap between released instruction datasets and test data, and use established metrics, including ChatGPT as judge for code summarization.Source: §VI · p. 11

VII. RELATED WORK

  • Earlier evaluations largely omitted software engineering or focused on small pre-trained models without instruction tuning.The paper distinguishes its scope as evaluation of open-source instructed LLMs across four representative code-related tasks.Source: §VII · p. 11

VIII. CONCLUSION

  • The study evaluates 10 recent open-source instructed LLMs and five baselines across four representative tasks, finding benefits from zero-shot, few-shot, and fine-tuning settings.The authors also draw practical implications for model use, performance-cost trade-offs, and future research.Source: §VIII · pp. 11–12
Loading 2308.01240v1…