PaperlayerView exact revision on arXiv

Source-linked AI summary

No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligence

Chaozheng Wang, Yuanhang Yang, Cuiyun Gao, Yun Peng, Hongyu Zhang, Michael R. Lyu

arXiv:2207.11680v1cs.SEcs.AI

TL;DR

The paper addresses gaps between pre-training and fine-tuning in code intelligence, including mismatched inputs and objectives and reliance on downstream data. It experimentally evaluates prompt tuning with CodeBERT and CodeT5 across three tasks and finds that prompt tuning outperforms fine-tuning, particularly in scarce-data settings. The authors conclude that prompt tuning can be adapted for code intelligence tasks, while noting limits in task and dataset coverage.

  • Problem

    Mismatched pre-training and downstream inputs and objectives make pre-trained knowledge difficult to fully exploit, while fine-tuning performance strongly depends on downstream data.

  • Method

    The paper experimentally evaluates prompt tuning with CodeBERT and CodeT5 on defect detection, code translation, and code summarization, including scarce-data settings and prompt designs.

  • Results

    Prompt tuning outperforms fine-tuning across full-data, data-scarcity, and cross-domain settings in the evaluated three-task study.

  • Takeaways & Limitations

    The findings suggest adapting prompt tuning for code intelligence tasks, especially when task-specific data are scarce.

  • Takeaways & Limitations

    The experiments cover a limited number of datasets and three representative tasks, leaving other code intelligence tasks for future evaluation.

Abstract

from arXiv · show

Pre-trained models have been shown effective in many code intelligence tasks. These models are pre-trained on large-scale unlabeled corpus and then fine-tuned in downstream tasks. However, as the inputs to pre-training and downstream tasks are in different forms, it is hard to fully explore the knowledge of pre-trained models. Besides, the performance of fine-tuning strongly relies on the amount of downstream data, while in practice, the scenarios with scarce data are common. Recent studies in the natural language processing (NLP) field show that prompt tuning, a new paradigm for tuning, alleviates the above issues and achieves promising results in various NLP tasks. In prompt tuning, the prompts inserted during tuning provide task-specific knowledge, which is especially beneficial for tasks with relatively scarce data. In this paper, we empirically evaluate the usage and effect of prompt tuning in code intelligence tasks. We conduct prompt tuning on popular pre-trained models CodeBERT and CodeT5 and experiment with three code intelligence tasks including defect prediction, code summarization, and code translation. Our experimental results show that prompt tuning consistently outperforms fine-tuning in all three tasks. In addition, prompt tuning shows great potential in low-resource scenarios, e.g., improving the BLEU scores of fine-tuning by more than 26\% on average for code summarization. Our results suggest that instead of fine-tuning, we could adapt prompt tuning for code intelligence tasks to achieve better performance, especially when lacking task-specific data.

1 INTRODUCTION

The paper investigates prompt tuning as an alternative to fine-tuning for code intelligence, addressing mismatches between pre-training and downstream tasks and data scarcity. Experiments examine its effectiveness across tasks, prompt designs, and scarce-data settings.

  • Motivation: Pre-training and fine-tuning use inconsistent inputs and objectives, making pre-trained knowledge difficult to fully exploit and making performance dependent on downstream data scale.Pre-training uses masked-token prediction over code and text, whereas fine-tuning may use source code with a classification objective.
  • Study scope: The study experimentally evaluates prompt tuning on defect detection, code translation, and code summarization.It uses CodeBERT and CodeT5 as the pre-trained models.
  • Research questions: The experiments test prompt tuning effectiveness, performance under low-resource and cross-domain scarcity, and the effects of prompt templates and verbalizers.These correspond to three research questions about effectiveness, data scarcity, and prompt design.
  • Findings: Prompt tuning brings non-trivial improvements to downstream code intelligence tasks, including classification and generation.The reported result covers both task types.
  • Findings: Prompt tuning can significantly outperform conventional fine-tuning, especially when training data are scarce.This finding motivates evaluating prompt tuning in data-scarcity scenarios.
  • Contributions: The paper presents its study as the first investigation of prompt tuning performance for code intelligence tasks and discusses implications for further research.The authors identify prompt effects and implications as contributions.

2 BACKGROUND

Fine-tuning adapts pre-trained models to downstream tasks but can mismatch their pre-training objectives and depend heavily on task-specific data. Prompt tuning reformulates inputs with prompts and label words to better align downstream tuning with pre-training.

  • Fine-tuning uses supervised downstream samples to adapt pre-trained model parameters for task-specific prediction.
  • Prompt tuning adds prompts so downstream inputs and objectives more closely resemble the masked-language-modeling pre-training stage.
  • Prompt templates reserve an input slot [X] for original text and an answer slot [Z] for predicted labels.
  • A verbalizer maps predicted label words to classes in the target class set.
  • Hard prompts use fixed natural-language instructions, whereas soft prompts use learnable continuous virtual-token vectors.
  • Hard-prompt template and verbalizer selection remain challenging, motivating vanilla and prefix soft-prompt alternatives.

3 EXPERIMENTAL EVALUATION

The experimental evaluation examines prompt tuning across code intelligence tasks, data-scarcity settings, and prompt-design choices. Its research questions compare effectiveness, assess low-resource and cross-domain capability, and study template and label-word effects.

  • The evaluation asks whether prompt tuning is effective for code intelligence tasks.
  • A second question examines prompt tuning under data scarcity, including low-resource and cross-domain settings.
  • A third question studies how prompt templates and label-word selection affect downstream performance.
  • The evaluation tests the hypothesis that aligning downstream and pre-training objectives may make prompt tuning more effective than fine-tuning.
  • The datasets used in the paper are summarized in Table 1.

3.2 Code Intelligence Tasks with Prompt Tuning

The study applies prompt tuning with CodeBERT and CodeT5 to defect detection, code summarization, and code translation. Task-specific hard, vanilla soft, and prefix soft prompts are designed for these settings.

  • 3.2 Code Intelligence Tasks with Prompt Tuning: The evaluation covers defect detection, code summarization, and code translation using CodeBERT and CodeT5.
  • 3.2.1 Pre-trained Models.: CodeBERT is an encoder-only RoBERTa-based model, while CodeT5 treats code-intelligence tasks as sequence-to-sequence problems.
  • 3.2.1 Pre-trained Models.: Defect detection identifies whether code is defect prone and is trained as classification with CodeBERT or generation with CodeT5.
  • 3.2.2 Defect Detection.: For defect detection, hard prompts use cloze templates whose answer slot receives label words mapped to classes by a verbalizer.
  • 3.2.2 Defect Detection.: Vanilla soft prompts replace hard-prompt natural-language tokens with virtual tokens, while prefix soft prompts append a learnable prefix.
  • 3.2.3 Code summarization.: Code summarization generates natural-language comments from code and uses CodeT5 because CodeBERT lacks a decoder.
  • 3.2.3 Code summarization.: Summarization hard prompts include programming-language, input, and generated-answer slots, with no verbalizer for generation.
  • 3.2.4 Code Translation.: Code translation migrates code between programming languages using task-specific prompts that map input code [X] to output code [Z].

3.3 Evaluation Datasets

The evaluation uses datasets from the CodeXGLUE benchmark for defect detection, code summarization, and code translation. These datasets cover open-source code, multilingual code-description pairs, and paired Java/C# translations.

  • The three task datasets are selected from the CodeXGLUE benchmark.
  • Defect detection uses 27K+ C snippets from QEMU and FFmpeg, with 45.0% labeled defective.
  • Code summarization uses CodeSearchNet code-comment pairs spanning Python, Java, JavaScript, Ruby, Go, and PHP.
  • Code translation uses Java and C# code from four public repositories and translates each into the corresponding other language.

3.4 Evaluation Metrics

The paper evaluates defect detection with Accuracy, generated comments with BLEU, and generated code with Accuracy and CodeBLEU.

  • Defect Detection: Accuracy measures defect detection performance, specifically the ability to identify insecure source code.
  • Code Summarization: BLEU evaluates generated comments by measuring their similarity to the ground-truth text.
  • Code Summarization: The experiments use smoothed BLEU-4, with n=4, for generation-task evaluation.
  • Code Translation: Code translation is evaluated with Accuracy, BLEU, and CodeBLEU, with Accuracy described as the strictest metric.
  • Code Translation: CodeBLEU accounts for code structure and semantics through n-gram, weighted n-gram, AST, and data-flow matching components.

3.5 Implementation Details

The implementation uses Hugging Face pre-trained models, PyTorch, and OpenPrompt, with task-specific training configurations and matched settings for fine-tuning and prompt tuning.

  • Framework and Tools: The experiments load models and tokenizers from Hugging Face, use PyTorch, and implement prompting with OpenPrompt.
  • Task Settings: CodeT5 is trained for 20 epochs on summarization, using maximum source and target lengths of 256 and 128.
  • Comparison Settings: Prompt tuning uses the same parameter configurations as fine-tuning for a fair comparison.
  • Fine-tuning Baselines: CodeBERT is fine-tuned only for defect detection, while CodeT5 is fine-tuned across defect detection, summarization, and translation.
  • Fine-tuning Baselines: For CodeBERT, the [CLS] representation feeds a feed-forward network; CodeT5 treats all tasks as generation.

4 EXPERIMENTAL RESULTS

Across defect detection, code summarization, and code translation, prompt tuning outperforms fine-tuning, with especially strong gains in low-resource settings. Prompt design choices also materially affect performance, particularly for classification tasks and prefix prompts.

  • Defect Detection: 3.30% improvement over fine-tuning is achieved on defect detection with CodeBERT, while CodeT5-small and CodeT5-base improve by 1.51% and 1.26%, respectively.The defect-detection advantage is statistically significant at the 0.05 level, with p-value 0.048.
  • Code Summarization: 6.46% and 2.91% improvement over fine-tuning is achieved in code summarization with CodeT5-small and CodeT5-base, respectively.Prompt tuning also performs better across programming languages, with the largest advancement in PHP: 9.11% and 5.01%.
  • Code Translation: 1.22%, 0.85%, and 0.87% average improvements are achieved for BLEU, Accuracy, and CodeBLEU, respectively, in code translation.Prompt tuning outperforms fine-tuning in both Java-C# and C#-Java translation directions.
  • Overall effectiveness: Prompt tuning consistently outperforms fine-tuning across defect detection, code summarization, and code translation.The finding holds across different pre-trained models and programming languages.
  • Low-resource scenarios: 28.08% and 26.86% average BLEU-4 improvements are achieved with CodeT5-small and CodeT5-base, respectively, in low-resource code summarization.Prompt tuning remains better across 100, 200, 300, 500, and 1000 training shots, although the improvement becomes less stark as training shots increase.
  • Prompt design: Prompt-template design, verbalizers, and prefix length substantially affect prompt-tuning performance.Hard prompts may be more effective for classification than corresponding vanilla soft prompts; prefix lengths between 100 and 200 perform best or nearly best across tasks.

5 DISCUSSION

Prompt tuning consistently outperforms fine-tuning across data regimes and benefits especially from scarce task-specific data. Prompt and verbalizer design still materially affects performance, while the study’s evidence is constrained by dataset, task, and prompt-design limitations.

  • Implication on the Utilization of Pre-trained Models: Prompt tuning consistently outperforms fine-tuning under full-data, data-scarcity, and cross-domain settings.
  • Implication on the Utilization of Pre-trained Models: Prompt tuning is especially advantageous when task-specific training data are scarce.
  • Implication on the Utilization of Prompts: Different prompt templates and verbalizers influence code-intelligence performance, even when templates have the same semantics.
  • Case Studies: Prompt tuning accurately preserves the intended update behavior in a Ruby summarization case where fine-tuning was misled by the word “modify”.
  • Case Studies: Prompt tuning produced Java code identical to the ground truth in a translation case where fine-tuning changed the original C# code’s semantics.
  • Threats to Validity: The study identifies limited datasets, limited downstream tasks, and potentially suboptimal prompt designs as threats to validity.

6 RELATED WORK

Related work traces code intelligence from pre-trained programming-language models to prompt-based adaptation. The paper applies prompt tuning to exploit knowledge from both natural language and programming language pre-training.

  • Code Intelligence: Code intelligence uses machine learning to learn program semantics for tasks such as code search, code summarization, and bug detection.
  • Pre-trained Models for Code Intelligence: CuBERT and CodeBERT pioneered pre-trained programming-language models using masked language modeling for source-code representation.
  • Pre-trained Models for Code Intelligence: PLBART supports both code-understanding and code-generation tasks through an encoder-decoder architecture.
  • Prompt Tuning: Prompt research developed from treating pre-trained language models as knowledge bases and adding learnable matrices or natural-language initializations to input embeddings.
  • Prompt Tuning: This work adapts prompt tuning to code intelligence to exploit knowledge about both natural language and programming languages.

7 CONCLUSION

The paper experimentally evaluates prompt tuning with two pre-trained models across three code-intelligence tasks. It reports that prompt tuning outperforms fine-tuning across full-data, scarce-data, and cross-domain settings.

  • Prompt tuning outperforms fine-tuning across full-data, data-scarcity, and cross-domain settings in three code-intelligence tasks with two pre-trained models.
Loading 2207.11680v1…