PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving ChatGPT Prompt for Code Generation

Chao Liu, Xuanlin Bao, Hongyu Zhang, Neng Zhang, Haibo Hu, Xiaohong Zhang, Meng Yan

arXiv:2305.08360v1cs.SE

TL;DR

Prompt quality is an open question for ChatGPT code generation, which this paper studies on CodeXGLUE’s text-to-code and code-to-code tasks. The authors manually optimize prompts through multi-step chain-of-thought, finding that specific prompt requirements improve CodeBLEU in both tasks.

  • Problem

    ChatGPT’s code-generation performance is not well understood and depends strongly on prompt quality.

    Source: §I · p. 1

  • Method

    The paper evaluates ChatGPT on CodeXGLUE’s text-to-code and code-to-code tasks using manually designed, multi-step chain-of-thought prompt optimizations.

    Source: §I · pp. 1–2

  • Results

    Adding specific requirements improved ChatGPT-task CodeBLEU by 73.58% for T2C and 3.45% for C2C.

    Source: §I · p. 1

  • Takeaways & Limitations

    The study reports that carefully designed prompts can improve ChatGPT’s code-generation performance on the two investigated tasks.

    Source: §I · p. 1, §IX · p. 10

    The evaluation uses one dataset, and the C2C task tests only C#-to-Java generation, so extension to other datasets, languages, and directions is uncertain.

    Source: §VIII · p. 10

Abstract

from arXiv · show

Automated code generation can be a powerful technique for software development, significantly reducing developers' efforts and time required to create new code by generating it automatically based on requirements. Recently, OpenAI's language model ChatGPT has emerged as a powerful tool for generating human-like responses to a wide range of textual inputs (i.e., prompts), including those related to code generation. However, the effectiveness of ChatGPT for code generation is not well understood, and the generation performance could be heavily influenced by the choice of prompt. To answer these questions, we conducted experiments using the CodeXGlue dataset to evaluate ChatGPT's capabilities for two code generation tasks, including text-to-code and code-to-code generation. We designed prompts by leveraging the chain-of-thought strategy with multi-step optimizations. Our results showed that by carefully designing prompts to guide ChatGPT, the generation performance can be improved substantially. We also analyzed the factors that influenced the prompt design and provided insights that could guide future research.

I. INTRODUCTION

This paper examines how prompt design affects ChatGPT’s text-to-code and code-to-code generation, evaluating both tasks on CodeXGlue. It uses manually constructed, multi-step prompt optimizations and examines generated-code performance, correctness, and quality.

I. INTRODUCTION

  • The study evaluates ChatGPT on two CodeXGlue tasks: generating code from natural-language requirements and translating code between programming languages.The introduction also identifies correctness and code quality analysis, alongside comparison with fine-tuned language models.Source: §I · pp. 1–2
  • Because ChatGPT’s code-generation performance depends strongly on prompt quality, the paper investigates prompt engineering as a way to guide its outputs.The authors use manually constructed prompts with chain-of-thought and multi-step optimization based on ChatGPT feedback.Source: §I · pp. 1–2
  • Adding specific prompt requirements improved ChatGPT-task CodeBLEU by 73.58% for T2C and 3.45% for C2C; concise-code prompting raised T2C to 50.18, and session sharing raised C2C to 48.80.The basic prompts scored 22.76 for T2C and 39.37 for C2C; prompt randomness had little effect under the specific instructions.Source: §I · p. 1
  • The paper identifies prompt-design methods for guiding code generation and releases a replication package for future exploration.The introduction presents the evaluation of ChatGPT on CodeXGlue as a contribution as well.Source: §I · p. 2

A. Large Language Model for Code Generation

The paper situates ChatGPT among language models used for code generation and frames prompt engineering as a way to shape model inputs. It uses manual prompt construction to study code-generation guidance and influential design factors.

A. Large Language Model for Code Generation

  • Code-generation models include BERT-, T5-, BART-, and GPT-based families, with approaches such as code-structure modeling, identifier awareness, and code-focused pretraining.Examples include CodeBERT, CodeT5, PLBART, CodeGPT, and CodeGen.Source: A. Large Language Model for Code… · p. 2

B. ChatGPT and Prompt Engineering

  • ChatGPT is a conversational language model built on the GPT-3.5 series and generates responses using prompt context and conversation history.The passage also notes optimization using reinforcement learning from human feedback.Source: B. ChatGPT and Prompt Engineering · p. 2
  • The study manually constructs prompts to explore guidance for code generation and influential prompt-design factors.The passage distinguishes manual construction from LM generation, retrieval-based prompts, and prompt learning.Source: B. ChatGPT and Prompt Engineering · p. 2

III. STUDY SUBJECTS

The study covers text-to-code and code-to-code generation, evaluates them using CodeXGLUE datasets, and uses CodeBLEU as its overall metric.

A. Code Generation Tasks

  • Text-to-code generates code from a natural-language functional specification, while code-to-code translates a code snippet into another language with the same functionality.The study uses Java as the target language for both tasks, with C# as the code-to-code input language.Source: A. Code Generation Tasks · p. 3

B. Datasets

  • The T2C dataset is CONCODE, with Java code, Javadoc-derived natural-language descriptions, and class-environment context; the C2C dataset pairs functionally equivalent C# and Java snippets.CONCODE has 100k training, 2k validation, and 2k test examples; the C2C dataset has 10k training, 0.5k validation, and 1k test examples.Source: B. Datasets · p. 3

C. Evaluation Metrics

  • CodeBLEU is the overall evaluation metric, combining lexical, abstract-syntax-tree, and data-flow matches between generated and reference code.The study also measures BLEU, which uses n-gram overlap and a brevity penalty.Source: C. Evaluation Metrics · p. 3

IV. METHODOLOGY

The method uses chain-of-thought prompting and iterative optimization to guide code generation. For text-to-code, prompts add dataset context, ground-truth formatting instructions, and behavioral requirements in successive steps.

A. Methods for Prompt Design

  • The design process first drafts a natural task prompt, asks ChatGPT how to improve it, then tests and iteratively refines it against training samples and ground truth.The general approach uses the chain-of-thought strategy to guide code generation.Source: A. Methods for Prompt Design · p. 3

B. Prompt Design for Text-to-Code Generation

  • The basic text-to-code prompt, “write a Java method that” plus the natural-language description, scored BLEU=5.29 and CodeBLEU=22.76 on 100 sampled training instances.The method section identifies Table I as the location of prompt variants and Figure 1 as an overview of prompt design and verification.Source: A. Methods for Prompt Design · p. 3, B. Prompt Design for Text-to-Code… · p. 4, Table I · p. 4
  • Adding class context to the basic text-to-code prompt raised sample accuracy to BLEU=10.42 and CodeBLEU=25.05.The context prompt supplies the class name, member variables, and member functions; it aims to prevent generation of an entire class when a method is required.Source: B. Prompt Design for Text-to-Code… · p. 4
  • Instructions to match ground-truth preprocessing raised text-to-code performance to BLEU=13.11 and CodeBLEU=36.00.The target formatting removes comments, throws, and modifiers, and renames the method, arguments, and local variables.Source: B. Prompt Design for Text-to-Code… · p. 4
  • The final text-to-code behavior prompt reached BLEU=27.48 and CodeBLEU=46.78, after API and exception-handling requirements were incorporated.API requirements alone yielded BLEU=22.14 and CodeBLEU=44.18.Source: B. Prompt Design for Text-to-Code… · p. 5

C. Prompt Design for Code-to-Code Generation

For code-to-code generation, the authors iteratively adjust prompts to reflect translation-specific data and formatting differences. Adding behavioral requirements improves CodeBLEU slightly in the sample tests but reduces BLEU, leading the authors to suspect negative effects.

C. Prompt Design for Code-to-Code Generation

  • The task prompt asks ChatGPT to translate a supplied C# function into Java; on 100 randomly selected training samples it scored BLEU=9.76 and CodeBLEU=39.37.The prompt template is “translate C# code into Java code” followed by the code.Source: C. Prompt Design for Code-to-Code… · p. 5
  • Formatting the input C# code with Markdown delimiters raised sample performance to BLEU=15.44 and CodeBLEU=45.00 after a no-annotation instruction had raised CodeBLEU but lowered BLEU.The no-annotation prompt scored BLEU=8.55 and CodeBLEU=45.28.Source: C. Prompt Design for Code-to-Code… · p. 5
  • The authors add API-usage and exception-handling requirements extracted from ground-truth code, but the full behavior prompt's results suggest these additions may be counterproductive for this task.API requirements alone scored BLEU=13.37 and CodeBLEU=46.17.Source: C. Prompt Design for Code-to-Code… · p. 5
  • The full behavior prompt yielded BLEU=8.90 and CodeBLEU=46.88 in sample tests, and the authors say it likely harms code-to-code generation.They report that additional requests may bring uncertainty despite ChatGPT's ability to understand translation context.Source: C. Prompt Design for Code-to-Code… · p. 5

D. Research Questions

The study asks whether its prompts improve ChatGPT code generation and examines three additional factors: requests for concise code, session context, and generation randomness.

D. Research Questions

  • RQ1 tests whether the designed prompts improve ChatGPT's performance on the corresponding test datasets.The prompts use chain-of-thought and multi-step optimization.Source: D. Research Questions · p. 5
  • RQ2 asks whether directly requesting concise code further improves generation performance.This question follows the observation that ChatGPT often generates code more detailed and complex than the ground truth.Source: D. Research Questions · p. 5
  • RQ3 tests whether ChatGPT generates better code when given a session containing multiple prompts rather than an individual session for each prompt.The motivation is that ChatGPT can learn from session context.Source: D. Research Questions · p. 5
  • RQ4 reruns guided ChatGPT multiple times to examine how generation randomness affects performance stability.The question addresses slight output differences that can occur with the same prompt.Source: D. Research Questions · p. 5

V. RESULTS

The designed prompts improve text-to-code performance substantially, while code-to-code results favor the detail prompt over the behavior prompt. The evaluation compares three prompt baselines across both tasks.

A. Effectiveness of the Designed Prompt (RQ1)

  • The three baselines are ChatGPT-task, ChatGPT-detail with task/context/processing prompts, and ChatGPT-behaviour, which adds the behavior prompt to the detail baseline.The detail baseline omits the updated-task prompt for T2C and the context prompt for C2C.Source: A. Effectiveness of the Designed… · pp. 5–6, Table III · p. 6
  • ChatGPT-behaviour reached T2C BLEU=21.59 and CodeBLEU=48.69, improving over ChatGPT-task by 283.48% and 73.58%, respectively.The authors conclude the designed prompts substantially improve text-to-code generation.Source: Table III · p. 6, A. Effectiveness of the Designed… · p. 6
  • ChatGPT-detail reached C2C BLEU=15.79 and CodeBLEU=47.71, while ChatGPT-behaviour scored BLEU=9.47 and CodeBLEU=47.38.The authors report that the designed prompt can also improve C2C performance, with ChatGPT-detail outperforming ChatGPT-behaviour.Source: Table III · p. 6, A. Effectiveness of the Designed… · p. 6

B. Impacts of Conciseness Request (RQ2)

RQ2 tests whether adding a conciseness request changes ChatGPT’s code-generation performance, while RQ3 examines whether using continuous sessions affects performance. The supplied passages describe their aims and setups but do not report their outcomes.

B. Impacts of Conciseness Request (RQ2)

  • RQ2 asks whether requesting concise code can improve on prompts that still produce code more complex than the ground truth.The authors added “concise” to the best T2C and C2C baseline prompts and tested the modified prompts on the testing data.Source: B. Impacts of Conciseness Request (RQ2) · p. 6

C. Impacts of Session Setting (RQ3)

  • RQ3 examines whether a continuous session lets ChatGPT use conversation context to improve code-generation performance.The authors note that earlier prompts in a session have less contextual information available.Source: C. Impacts of Session Setting (RQ3) · pp. 6–7
  • For RQ3, the authors generated multiple prompts in one session, then started a new session when the maximum prompt limit was reached.They describe this setup as fully utilizing each session and helping ChatGPT understand context.Source: C. Impacts of Session Setting (RQ3) · p. 7

D. Impacts of Generation Randomness (RQ4)

Across five runs, the designed prompts produced stable results for both code-generation tasks. The paper also compares the best prompts with fine-tuned models, finding different relative performance on T2C and C2C.

D. Impacts of Generation Randomness (RQ4)

  • The randomness analysis ran the best RQ3 baselines five times and calculated average and standard deviation performance.The authors used these measurements to assess generation stability and randomness effects.Source: D. Impacts of Generation Randomness… · p. 7
  • Across five runs, T2C CodeBLEU averaged 50.16 (STD=0.05), while C2C CodeBLEU averaged 48.86 (STD=0.33).The authors report that randomness had little effect and attribute this to the prompts’ specific instructions.Source: Table VI · p. 7, D. Impacts of Generation Randomness… · p. 7

A. Comparison with Fine-Tuned Models

  • ChatGPT-best ranked first among the compared fine-tuned models for T2C CodeBLEU, but fifth for C2C, outperforming only PBSMT.The authors suggest C2C’s poorer performance may result from limited contextual information in the prompt.Source: A. Comparison with Fine-Tuned Models · p. 8

B. Correctness of Code Generation

The paper assesses functional equivalence and code quality for ChatGPT-generated T2C and C2C outputs. It reports higher functional equivalence for the sampled C2C outputs, while generated code still contained code-quality issues.

B. Correctness of Code Generation

  • The authors relate lower T2C correctness to imprecise natural-language descriptions, while C2C translation supplies context for line-by-line generation.They also note that T2C generation works only for code using commonly used APIs.Source: B. Correctness of Code Generation · p. 8
  • Among 100 samples per task, 31 T2C and 59 C2C outputs were functionally equivalent to the ground truth.Correctness was measured by functional equivalence; the authors sampled 100 generated outputs from each task.Source: B. Correctness of Code Generation · p. 8, Table VIII · p. 9

C. Quality of Code Generation

  • T2C generated code had 29 critical code smells versus 35 in ground truth; C2C had 62 versus 17, with no severe bugs or vulnerabilities reported.The quality assessment used SonarQube and counted code with critical or blocker issues; one critical bug was also identified in T2C code.Source: C. Quality of Code Generation · pp. 8–9

VII. IMPLICATION

The implications section offers prompt-writing suggestions for developers and identifies future research directions. It also advises caution about generated-code quality based on the experiments.

VII. IMPLICATION

  • Developers are advised to provide rich programming context, request preprocessing or concise output as needed, and refine prompts step by step using ChatGPT’s feedback.The authors also suggest enclosing code snippets in triple quotes and using a continuous session to provide context.Source: §VII · pp. 9–10
  • Future research could automate prompt design, improve automatic correctness metrics, and further assess and improve generated-code quality.The authors frame these directions within the potential of the pre-train, prompt, predict paradigm for software engineering research.Source: §VII · p. 10

VIII. THREATS TO VALIDITY

The paper identifies limits in task and dataset coverage, manual prompt design, testing, and human evaluation. It concludes that its prompts were effective on two CodeXGlue generation tasks and points to future research directions.

VIII. THREATS TO VALIDITY

  • Findings may not extend beyond CodeXGlue, text-to-code, and C#-to-Java generation; other tasks, datasets, languages, and translation directions remain uncertain.The authors specifically name code completion and test-case generation as untested task types.Source: §VIII · p. 10
  • Prompt design and optimization relied on human understanding and observations, while choices were based on 100 randomly selected samples.The authors note designer knowledge and sample size or randomness may affect prompt performance and choices.Source: §VIII · p. 10
  • Testing covered a limited set of prompt and session choices because invoking the ChatGPT API across the whole dataset was costly.The tested choices were those considered likely to improve generation performance.Source: §VIII · p. 10
  • Correctness analysis relied on human judgments of 100 randomly selected samples, with three authors voting to mitigate subjective bias.The authors characterize the sample count as limited.Source: §VIII · p. 10

IX. CONCLUSION

  • The conclusion reports effective prompts on CodeXGlue for text-to-code and code-to-code generation, alongside prompt-factor analysis and comparisons with fine-tuned LLMs.It also says the study assessed generated-code correctness and quality and presents potential future research directions.Source: §IX · p. 10
Loading 2305.08360v1…