PaperlayerView exact revision on arXiv

Source-linked AI summary

ClarifyGPT: Empowering LLM-based Code Generation with Intention Clarification

Fangwen Mu, Lin Shi, Song Wang, Zhuohao Yu, Binquan Zhang, Chenxue Wang, Shichao Liu, Qing Wang

arXiv:2310.10996v1cs.SE

TL;DR

Ambiguous or insufficient requirements can lead code-generation LLMs to produce programs that diverge from users’ intent. ClarifyGPT checks for ambiguity, asks targeted questions, and uses responses to refine requirements before generating code; evaluations report improvements over default models across benchmarks.

  • Problem

    LLMs often generate code directly from ambiguous requirements without asking for clarification, risking programs that diverge from users’ intent.

    Source: §1 · p. 2

  • Method

    ClarifyGPT checks code consistency to detect ambiguity, prompts an LLM to reason over differing solutions and ask targeted questions, then refines the requirement using responses.

    Source: §1 · pp. 2–3

  • Results

    11.52% for GPT-4 and 15.07% for ChatGPT are the reported average improvements with ClarifyGPT across four benchmarks over the default models.

    Source: §1 · p. 3

  • Takeaways & Limitations

    ClarifyGPT’s evaluations show potential for applying interactive requirement clarification to code generation, and its user simulation supports automated evaluation without direct user participation.

    Source: §1 · p. 3

    Low-fidelity simulated user feedback may be unlike feedback in actual practice and can yield misleading evaluations.

    Source: §6.3 · p. 18

Abstract

from arXiv · show

We introduce a novel framework named ClarifyGPT, which aims to enhance code generation by empowering LLMs with the ability to identify ambiguous requirements and ask targeted clarifying questions. In particular, ClarifyGPT first detects whether a given requirement is ambiguous by performing a code consistency check. If it is ambiguous, ClarifyGPT prompts an LLM to generate targeted clarifying questions. After receiving question responses, ClarifyGPT refines the ambiguous requirement and inputs it into the same LLM to generate a final code solution. To evaluate our ClarifyGPT, we first conduct a human evaluation involving ten participants who use ClarifyGPT for code generation on two publicly available benchmarks: MBPP-sanitized and MBPP-ET. The results show that ClarifyGPT elevates the performance (Pass@1) of GPT-4 from 70.96% to 80.80% on MBPP-sanitized. Furthermore, to perform large-scale automated evaluations of ClarifyGPT across different LLMs and benchmarks without requiring user participation, we introduce a high-fidelity simulation method to simulate user responses. The automated evaluation results also demonstrate that ClarifyGPT can significantly enhance code generation performance compared to the baselines. In particular, ClarifyGPT improves the average performance of GPT-4 and ChatGPT across four benchmarks from 68.02% to 75.75% and from 58.55% to 67.22%, respectively. We believe that ClarifyGPT can effectively facilitate the practical application of LLMs in real-world development environments.

1 INTRODUCTION

ClarifyGPT addresses code-generation requirements whose ambiguity can lead LLM-generated programs away from users’ intent. It checks for ambiguity, generates targeted questions, refines the requirement using responses, and reports gains in human and automated evaluations.

Motivation

  • Ambiguous or insufficient requirements can lead LLMs to produce programs that diverge from users’ intent, while developers typically seek clarification.For example, a sorting requirement may not specify ascending or descending order.Source: §1 · p. 2

Framework

  • ClarifyGPT uses a two-step code consistency check to identify ambiguous requirements before asking questions and generating the final code.It generates test inputs, samples code solutions, and compares their outputs; differing outputs indicate a requirement needing clarification.Source: §1 · pp. 2–3

Evaluation

  • 11.52% for GPT-4 and 15.07% for ChatGPT are the reported average improvements across four benchmarks with ClarifyGPT over the default models.Human evaluation also raised GPT-4 Pass@1 on MBPP-sanitized from 70.96% to 80.8%.Source: §1 · p. 3

Contributions

  • ClarifyGPT also introduces simulated user feedback for automated evaluation and reports human evaluation on two benchmarks with ten participants.The framework, simulation method, evaluations, and publicly accessible dataset and source code are listed as contributions.Source: §1 · p. 3

2 BACKGROUND AND RELATED WORK

Prior code-generation work uses LLM architectures and prompting to produce or improve code, but the cited approaches do not directly resolve ambiguous user requirements. Clarifying-question research spans retrieval and dialogue, while code-generation work has addressed a narrower range of ambiguities.

2.1 LLM-based Code Generation

  • Code-generation LLMs include encoder-decoder models such as PLBART, CodeT5, and AlphaCode, and decoder-only models such as GPT-series models, PolyCoder, and InCoder.The passage describes encoder-decoder models as encoding input text and decoding it to a code solution, while decoder-only models generate code left to right.Source: §2.1 · p. 4
  • Prompting approaches such as CoT, Structured CoT, and self-planning elicit intermediate reasoning, but the paper says unclear intent can still mislead that reasoning.ClarifyGPT instead aims to detect ambiguous requirements and ask targeted questions.Source: §2.1 · p. 4

2.2 Clarifying Question Generation

  • Clarifying-question research in information retrieval and dialogue includes template-guided, rule-based, and seq2seq methods [9].These studies address ambiguous queries or dialogue rather than the code-generation setting described here.Source: §2.2 · pp. 4–5
  • Code ClarQA retrieves questions and answers from a constructed dataset, focusing on operational-level ambiguities and depending on similar requirements being available.ClarifyGPT is presented as generating targeted questions for a broader range of requirement ambiguities.Source: §2.2 · p. 5

3 APPROACH

ClarifyGPT uses generated tests and sampled code behavior to identify requirements that need clarification, then reasons about inconsistent solutions to formulate targeted questions. It incorporates question responses into a refined requirement before generating the final solution.

3 APPROACH

  • The four stages are test input generation, code consistency checking, reasoning-based question generation, and enhanced code generation.The final stage uses question feedback to refine the original requirement and generate code from the refined prompt.Source: §3 · pp. 5–6

3.1 Test Input Generation

  • LLM-generated seed inputs and heuristic mutations provide test cases intended to distinguish code solutions with different functionalities.The LLM generates semantically valid seed inputs, while heuristic mutation accelerates production of larger test sets.Source: §3.1 · p. 6

3.1.1 Seed Input Initialization.

  • Seed-input prompts combine an instruction, few-shot demonstrations, and a query; generated seeds initialize the mutation pool.The demonstrations pair requirements with ground-truth test inputs.Source: §3.1.1 · p. 6

3.1 Test Input Generation

  • Type-aware mutation selects an input from the seed pool and applies an operation matching its data type, adding new inputs back to the pool.For simple numeric types, an operation can increase or decrease the value by 1; compound types are mutated according to their internal types.Source: §3.1.2 · p. 7

3.2 Code Consistency Check

  • ClarifyGPT samples code solutions for a requirement and treats non-identical outputs on generated tests as evidence that the requirement is ambiguous.When outputs differ, solutions are clustered by test outputs, and one solution from each group is passed to question generation.Source: §3.2 · p. 7

3.3 Reasoning Based Question Generation

  • For question generation, the prompt directs the LLM to analyze differences in inconsistent solutions and then formulate targeted questions about the ambiguity.The prompt uses an instruction, few-shot requirement-solution-question demonstrations, and a query containing the requirement and code solutions.Source: §3.3 · pp. 7–8

3.4 Enhanced Code Generation

  • Each question is paired with its answer and appended to the original docstring, preserving its structure while adding clarification before final code generation.The refined requirement is then used in a prompt for generating the final solution.Source: §3.4 · p. 9

4 EXPERIMENTAL DESIGN

The experimental design evaluates ClarifyGPT with human and simulated feedback, across two chat LLMs and four code-generation benchmarks. It also examines prompt robustness and compares the framework against default generation, CoT, and GPT-Engineer2.

4 EXPERIMENTAL DESIGN

  • The design covers research questions, models, benchmarks, metrics, baselines, and implementation details.It frames evaluations of interactive feedback, simulated feedback, and the number of prompt demonstrations.Source: §4 · p. 9, §4.1 · p. 9

4.1 Research Questions

  • Human-feedback evaluation uses GPT-4 and ten participants on MBPP-sanitized and MBPP-ET, while simulated-feedback evaluation spans GPT-4 and ChatGPT across four benchmarks.A separate research question tests performance with varying numbers of demonstrations.Source: §4.1 · p. 9

4.2 Studied LLMs

  • The study selects ChatGPT and GPT-4 because its base models must understand human instructions and formulate clarifying questions.The experiments access gpt-3.5-turbo and gpt-4-turbo through OpenAI’s API.Source: §4.2 · pp. 9–10

4.3 Benchmarks

  • The evaluation uses HumanEval, MBPP-sanitized, and their extended-test versions, HumanEval-ET and MBPP-ET.The extended benchmarks add an average of 100+ test cases per problem, including edge cases.Source: §4.3 · p. 10

4.4 Evaluation Metrics

  • Code accuracy is evaluated with Pass@k, with k set to 1 and each approach run three times for an average result.A problem counts as solved when any of its k generated solutions passes all tests.Source: §4.4 · pp. 10–11

4.5 Comparison Baselines

  • Baselines include direct generation from the original requirement, CoT prompting, and GPT-Engineer2’s question-asking workflow.The CoT baseline uses the same number of demonstrations and demonstration seeds as the comparison setup.Source: §4.5 · p. 11

4.6 Implementation Details

  • Prompt demonstrations are manually created from the first three problems of each benchmark, and model settings constrain tokens and sampling.Question prompts allow 800 maximum tokens, other prompts 300, and code-solution sampling uses temperature 0.8.Source: §4.6 · p. 11

5 RESULTS AND ANALYSIS

Human feedback improves ClarifyGPT Pass@1 over baselines on two benchmarks; simulated feedback produces gains across two LLMs and four benchmarks. Across zero to three prompt demonstrations, ClarifyGPT remains ahead of Default, with performance generally rising as demonstrations increase.

5.1 RQ1: How does ClarifyGPT perform when receiving real user feedback in comparison to baseline approaches?

  • In human evaluation, GPT-4 Pass@1 rose from 70.96% to 80.8% on MBPP-sanitized and from 51.52% to 60.19% on MBPP-ET.ClarifyGPT (Human Feedback) also outperformed the baselines on both benchmarks.Source: Table 2 · p. 12, §5.1 · p. 13
  • Of 140 ambiguous MBPP-sanitized problems, each averaged 2.85 clarifying questions and was answered by three participants in a ten-person study.Participants responded using questionnaires that included the requirements, unit tests, and generated questions.Source: §5.1 · p. 12

5.2 RQ2: How does the ClarifyGPT perform when receiving simulated user feedback compared to the state-of-the-art baseline approaches?

  • The simulation method prompts LLMs with clarifying questions and ground-truth tests, whose input-output examples inform simulated answers about intended functionality.Source: §5.2 · p. 13
  • GPT-4 average Pass@1 across four benchmarks rose from 68.02% to 75.75%, and ChatGPT from 58.55% to 67.22% with simulated feedback.The reported averages cover four benchmarks for each model.Source: Table 3 · p. 14, §5.2 · p. 14

5.3 RQ3: How does the number of demonstrations in a prompt impact the performance of ClarifyGPT?

  • Across two LLMs and four benchmarks, ClarifyGPT outperformed Default with zero to three demonstrations, and average Pass@1 rose as demonstrations increased from zero to three.ChatGPT averages rose from 59.77% to 67.22%, while GPT-4 averages rose from 68.59% to 75.75%.Source: Table 4 · p. 15, §5.3 · pp. 15–16
  • One-shot performance was significantly above zero-shot and close to three-shot, which the authors identify as an effectiveness-efficiency trade-off for practical use.Source: §5.3 · p. 15

6 DISCUSSION

ClarifyGPT asks targeted questions for ambiguous requirements while avoiding unnecessary questions for clear ones; the discussion describes its benefits, limitations, and evaluation threats.

6.1 Case Study

  • In the ambiguous MBPP sorting case, ClarifyGPT asks only whether sorting should be ascending or descending and produces a correct solution, while GPT-Engineer asks five questions, some inferable from the requirement.Default ChatGPT instead sorts descending and fails the ground-truth tests.Source: §6.1 · p. 17
  • For a well-defined requirement, ClarifyGPT asks no questions and produces a correct solution, whereas GPT-Engineer asks three uninformative questions and produces an incorrect one.The extra questions make the refined prompt excessively long and may confuse the LLM.Source: §6.1 · p. 17

6.2 Benefits and Limitations

  • Clarifying requirements can make code changes interpretable to users and help them identify ambiguities without first generating and analyzing code.The authors contrast this with methods that sample candidate code and select one.Source: §6.2 · p. 17
  • ClarifyGPT requires base LLMs with enough communicative competence to understand instructions and formulate clarifying questions.The authors therefore limit the applicable models to those with that capability.Source: §6.2 · p. 17
  • Its consistency-checking approach requires generating test inputs and comparing outputs, constraining use with complex inputs or programs that return no values.The paper names image or file inputs and deep-learning programs as examples.Source: §6.2 · p. 18

6.3 Threats to Validity

  • Because LLMs train on open-source code repositories, public benchmarks may have appeared in training data and influenced evaluation results.The authors selected manually reviewed HumanEval and MBPP-sanitized datasets to mitigate this potential leakage.Source: §6.3 · p. 18
  • Simulated user feedback may be unrealistic and mislead evaluation, although the authors report simulated-feedback performance close to human-feedback performance.Their simulation prompt supplies clarifying questions and ground-truth test cases to support high-fidelity feedback.Source: §6.3 · p. 18
  • Generalizability is assessed with two chat LLMs and four datasets; the authors run experiments three times and report averages to mitigate prompt sensitivity and variance.They report consistent gains across datasets.Source: §6.3 · p. 18

7 CONCLUSION

ClarifyGPT uses test inputs and code consistency checks to identify ambiguous requirements, ask targeted questions, and generate code from refined requirements. Across four benchmarks, it improves average performance for GPT-4 and ChatGPT.

7 CONCLUSION

  • ClarifyGPT generates test inputs, checks code consistency to identify ambiguity, uses intermediate reasoning to formulate targeted questions, and generates code from the refined requirement.The framework has four stages: test-input generation, consistency checking, question generation, and enhanced code generation.Source: §7 · pp. 18–19
  • Across four benchmarks, ClarifyGPT raises GPT-4 average performance from 68.02% to 75.75% and ChatGPT average performance from 58.55% to 67.22%.The evaluation uses HumanEval, HumanEval-ET, MBPP-sanitized, and MBPP-ET.Source: §7 · p. 19
  • In a human evaluation, ClarifyGPT achieves up to 16.83% relative improvement in Pass@1 over the Default baseline.Ten participants evaluated GPT-4 on two public benchmarks.Source: §7 · p. 19

Related papers

Loading 2310.10996v1…