PaperlayerView exact revision on arXiv

Source-linked AI summary

Python Code Generation by Asking Clarification Questions

Haau-Sing Li, Mohsen Mesgar, André F. T. Martins, Iryna Gurevych

arXiv:2212.09885v2cs.CL

TL;DR

Text-to-code models struggle when natural-language descriptions lack sufficient specifications. This paper introduces interactive clarification through the CodeClarQA dataset and a CQ-driven pipeline, finding that clarification questions and answers improve code generation while deciding when and what to ask remains challenging.

  • Problem

    Pretrained text-to-code models struggle with under-specified natural-language descriptions, where missing operation, argument, or sub-task specifications make user intent incomplete.

    Source: §1 · pp. 1–2

  • Method

    The paper creates CodeClarQA with synthetic clarification questions and answers and evaluates a pipeline that predicts clarification need, selects questions, and generates code.

    Source: §1 · p. 2, §2 · p. 2, §3 · p. 4

  • Results

    Clarification questions and answers improve PLM-based code generation across evaluation metrics, although advanced PLMs still struggle with the interactive task.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    Clarification questions are a strong information-gathering method for better code generation, while when to ask and what to ask remain open challenges.

    Source: §8 · p. 9

    The method primarily targets operation-level specifications and scales its CQA construction to Python code involving heavy API usage.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Code generation from text requires understanding the user's intent from a natural language description and generating an executable code snippet that satisfies this intent. While recent pretrained language models demonstrate remarkable performance for this task, these models fail when the given natural language description is under-specified. In this work, we introduce a novel and more realistic setup for this task. We hypothesize that the under-specification of a natural language description can be resolved by asking clarification questions. Therefore, we collect and introduce a new dataset named CodeClarQA containing pairs of natural language descriptions and code with created synthetic clarification questions and answers. The empirical results of our evaluation of pretrained language model performance on code generation show that clarifications result in more precisely generated code, as shown by the substantial improvement of model performance in all evaluation metrics. Alongside this, our task and dataset introduce new challenges to the community, including when and what clarification questions should be asked. Our code and dataset are available on GitHub.

1 Introduction

Text-to-code models support digital pair-programming but struggle when natural-language descriptions omit specifications. The paper introduces interactive clarification for Python operation calls and reports improved code generation with CQAs, while advanced PLMs still face difficulty in the task.

1 Introduction

  • Under-specified natural-language descriptions can omit specifications at the operation, argument, or sub-task level, making it difficult to determine what information is missing.This problem occurs in real-world use cases and complicates text-to-code generation.Source: §1 · pp. 1–2
  • The paper introduces interactive clarification for Python operation calls through the CodeClarQA dataset of synthetic clarification questions and answers.The proposed pipeline includes clarification-need prediction, question generation or ranking, and code generation.Source: §1 · p. 2
  • Conditioning PLM-based code generation on clarification questions and answers increases performance across evaluation metrics.The authors use this result as evidence supporting their hypothesis and dataset construction.Source: §1 · p. 2
  • Advanced PLMs including RoBERTa, BART, T5, and CodeT5 still struggle under the interactive code-generation pipeline.The authors identify this difficulty as a new challenge for recent pretrained models.Source: §1 · p. 2

2 Creating the CodeClarQA Dataset

CodeClarQA is created by detecting missing operation-level specifications in natural-language/code pairs and synthesizing clarification questions and answers. The process identifies key operations, aligns documentation with the description, applies thresholded similarity, and instantiates question templates.

2 Creating the CodeClarQA Dataset

  • CodeClarQA synthetically collects clarification questions and answers to resolve under-specification before code generation.The dataset is built from NLD-code pairs rather than requiring human annotations over the whole dataset.Source: §2 · p. 2, §2.4 · p. 3

2.1 Identifying Key Operations

  • Key operations are selected from code graphs using object/function structure and the highest-level operation on each data-flow path.Examples include sklearn.GridSearchCV as the key operation for related methods and fields.Source: §2.1 · pp. 2–3

2.2 Is a Key Operation Missing in NLD?

  • A key operation is labeled missing when the highest schema-element similarity between its documentation and the NLD falls below threshold t.The threshold is selected by validation-set grid search to maximize F1.Source: §2.2 · p. 3

2.3 Generating CQAs for Missing Key Operations

  • Missing operations receive multiple-choice or yes/no clarification questions generated from API names and documentation templates.Multiple-choice questions group operations sharing the same final token, while yes/no questions ask about a named documented operation.Source: §2.3 · p. 3

2.4 Dataset

  • The dataset uses high-quality notebook NLD-code pairs, keeps examples with at most five missing key operations, and reports its statistics in Table 1.The source dataset is notebookCDG, selected for code quality and NLD quality.Source: §2.4 · pp. 3–4, Table 1 · p. 4

3 Pipeline for CQ-driven Code Generation

The interactive pipeline first predicts whether clarification is needed, then selects informative questions and generates code from the description, questions, and answers. It operationalizes clarification as a three-module process for CQ-driven code generation.

3 Pipeline for CQ-driven Code Generation

  • The pipeline comprises a clarification-need predictor, a CQ ranker, and a code generator.The predictor outputs Need or No Need; when clarification is needed, the ranker selects five questions before code generation.Source: §3 · p. 4
  • Given an NLD, selected CQs, and their answers, the code generator produces code after the system decides whether clarification is needed.The fixed selection size is five questions, chosen to encourage information-rich CQs.Source: §3 · p. 4

4 Experiments

The experiments evaluate dataset construction and an interactive code-generation pipeline across dataset labeling, clarification prediction and ranking, code generation, and end-to-end integration. They use annotated splits, pretrained transformer models, ranking metrics, code-generation metrics, and selected clarification answers to assess the approach.

4 Experiments

  • The experiments assess dataset quality and interactive code generation through separate module evaluations followed by an end-to-end pipeline evaluation.The dataset evaluation studies missing-key-operation annotations, while later experiments evaluate clarification prediction, CQ ranking, code generation, and pipeline integration.Source: §4 · p. 4

4.1 Dataset Evaluation

  • Human annotations label whether each NLD-Code pair misses any key operation, using 100 validation pairs and 100 test pairs with Fleiss Kappa of 0.74.Agreement was 0.83 on validation and 0.66 on test; one annotator’s labels were randomly selected as reference labels.Source: §4.1 · p. 4

4.2 Clarification Need Predictor

  • A binary clarification-need classifier predicts whether an NLD requires further clarification, using BERT, RoBERTa, BART, and PLBART encoders evaluated by accuracy, precision, recall, and F1.Models are fine-tuned for 10 epochs with learning rate 5×10−5, and the best validation-accuracy model is selected.Source: §4.2 · p. 4

4.3 CQ Ranker

  • The CQ ranker selects clarification questions from a universal set by scoring NLD-CQ pairs, with random and BM25 negative sampling evaluated using recall at k for k ∈{1, 3, 5, 10}.Cross-encoders classify NLD-CQ pairs, then rank all universal-set CQs by model score at inference.Source: §4.3 · pp. 4–5, Table 2 · p. 5

4.4 Code Generator

  • The code-generation experiment compares models trained with and without oracle CQAs using BLEU, CodeBLEU, and Exact Match, while excluding execution-based metrics because code context is unavailable.The evaluated models include GPT-Neo, T5, CodeT5, CodeParrot, and PLBART; missing code context can make variable naming affect execution results without necessarily indicating poor code quality.Source: §4.4 · p. 5

4.5 End-to-end Pipeline Evaluation

  • The end-to-end pipeline predicts clarification need, ranks candidate CQs, selects overlapping top-k questions for k ∈{1, 3, 5}, and feeds their answers with the NLD to the code generator.The pipeline uses the best-performing module models and compares selected clarifications with CQs created by the dataset method.Source: §4.5 · p. 5

5 Results

The results evaluate dataset construction and each interactive code-generation module, finding strong clarification-conditioned gains but persistent difficulty in deciding when and what to ask.

5.1 Dataset Evaluation

  • MPNetbaseqa-cos performs best for identifying missing operations, while remaining errors include taxonomy and element-pair mismatches.Taxonomy and Element Pair errors account for up to 7.32% and 8.57% of aligned predictions, respectively; argument errors account for 4.08%/4.35% of negative predictions.Source: §5.1 · pp. 5–6

5.2 Clarification Need Predictor Evaluation

  • BERTbase achieves the best accuracy and F1 for predicting whether an under-specified description needs clarification, while RoBERTabase overfits the imbalanced data.RoBERTabase has high recall but low precision, whereas most tested classifiers perform relatively well.Source: §5.2 · p. 6

5.3 CQ Ranker Evaluation

  • CQ ranking is more challenging than lexical matching because correct questions often have low lexical overlap with the natural-language description, and BM25 has low recall.Models trained with random negative sampling outperform those trained with BM25 negative sampling.Source: §5.3 · pp. 6–7, Table 6 · p. 6

5.4 Code Generator Evaluation

  • 14.28 BLEU, 10.5 CodeBLEU, and 6.03 EM are the largest gains from adding clarification question-answer pairs, achieved by PLBARTbase.All models fine-tuned with clarifications substantially outperform their counterparts without them, supporting the hypothesis that interactions improve code generation.Source: §5.4 · p. 7, Table 7 · p. 7

5.5 Pipeline Evaluation

  • Pipeline performance increases when more highly ranked clarifications are selected and answered, but decreases when unanswered clarifications are appended.The full pipeline nevertheless underperforms models trained directly on natural-language-description and code pairs.Source: §5.5 · p. 7, Table 8 · p. 7

6 Analysis

The analysis tests whether clarification questions improve code generation by examining missing-operation recall, correlations with generation quality, and example predictions. Clarifications recover more missing operations and correlate positively with code-generation results, but selecting useful questions remains difficult.

Recall of identified missing key operations

  • Pearson correlations between recall of missing key operations and code-generation results are high and positive, supporting the hypothesis that clarified key operations help code generation.Source: §6 · p. 8
  • Training with clarifications includes substantially more missing key operations, although it still does not outperform NLD-and-code-only training.Table 9 reports micro and macro recall across four runs and distinguishes direct top-k question appending from selected-question experiments.Source: §6 · p. 8, Table 9 · p. 8

Case study

  • Oracle CQAs produce predictions close to ground truth on operations, but the pipeline misses AdaBoostClassifier and cross_val_predict because its top five questions omit the reference CQAs.Differences remain mainly at argument-level specifications, consistent with the operation-level clarification focus.Source: §6 · p. 8

7 Related Work

Related work frames clarification questions around disambiguation and information gathering, while this paper targets missing key operations for Python code generation. Its method derives specifications from code rather than requiring predefined input facets and differs from retrieval-based code-generation approaches by modeling user interaction.

CQ generation

  • Unlike facet-based clarification methods, the proposed approach does not require predefined input facets and collects natural-language specifications from code.The authors identify this code-derived specification process as the method’s advantage over prior facet-oriented approaches.Source: §7 · p. 8
  • The paper defines clarification questions as requests for new information about missing key operations, which are difficult to identify in an original code query.This extends prior work on disambiguation, missing information, missing schema, and pseudo-questions derived by sentence deletion.Source: §7 · p. 8

Text-to-Code generation

  • The work sources natural-language/code pairs from Jupyter Notebooks but excludes dependencies between cells because they would change question necessity and reduce controllability.This design choice limits CQA creation to the local code and natural-language pair.Source: §7 · p. 9
  • Where prior systems use API documentation, archived code, or retrieved training examples, this paper retrieves clarification questions and explicitly models interaction between user and model.The comparison concerns the task formulation rather than a claim that one retrieval source universally outperforms another.Source: §7 · p. 9

8 Conclusion and Future Work

The paper introduces clarification-question asking for Python code generation, releases CodeClarQA, and evaluates a pipeline using recent text and code encoders. Results support clarification as an information-gathering approach, while question selection and broader intent completeness remain open challenges.

Conclusion and Future Work

  • CodeClarQA provides synthetic clarification questions and answers for Python API usage without requiring human annotations across the whole dataset.The dataset accompanies a new challenge focused on asking clarification questions during code generation.Source: §8 · p. 9
  • The proposed pipeline evaluates clarification-question generation and code generation with recent text and code encoders.The conclusion describes the system as an evaluation pipeline for this challenge.Source: §8 · p. 9
  • Experimental results support clarification questions and answers as information-gathering methods for better code generation, while deciding when and what to ask remains challenging.Future work targets user engagement, question diversity, argument-level intent completeness, and conversational relations among questions.Source: §8 · p. 9

Limitations

The method is mainly scoped to operation-level specifications and Python code involving heavy API usage. It also remains limited in identifying specifications that are missing or insufficiently specified in the natural-language description.

Limitations

  • The method primarily addresses operation-level specifications, leaving other specification types in real-world use cases outside its main focus.The authors identify broader specification coverage as a limitation and future-work direction.Source: Limitations · p. 9
  • CQA creation scales to Python code involving heavy API usage, with analogous scaling dependent on developing a code-knowledge-graph generator for another language.This establishes both a Python/API-usage scope and a condition for cross-language extension.Source: Limitations · p. 9
  • The method has limited ability to identify specifications that are mentioned but not specified enough in the natural-language description.The paper suggests creating clarification questions for this type of incompleteness as future work.Source: Limitations · p. 9

Ethical Concerns

The paper reports minimal copyright and privacy concerns for its data sources, and the appendix describes how key operations are extracted and filtered.

Ethical Concerns

  • The authors report no copyright issues for the 20 Kaggle competitions and check licenses for open-source API documentation.Source: Ethical Concerns · p. 10
  • The authors assess privacy-leakage risk as minimal because the data come from competitions without privacy concerns and API documentation is similarly low risk.Source: Ethical Concerns · p. 10

Appendix

B Preliminary Experiments on Identifying Missing Key Operations

The paper evaluates missing-operation identification, illustrates its similarity-based decision process and error patterns, and documents annotation and dataset examples.

B Preliminary Experiments on Identifying Missing Key Operations

C Example of Identifying if an Key Operation is Missing

E Annotation

  • The appendix algorithm extracts notebook operations, removes imports, numerical expressions, non-final line calls, print functions, and undocumented operations, then retains data-flow-terminal operations.Source: E Annotation · p. 14

F Examples of CodeClarQA Dataset

E Annotation

  • Annotation differences are attributed mainly to subjectivity, including cases where an operation is inferable from the NLD but not directly mentioned.Source: E Annotation · p. 14

Related papers

Loading 2212.09885v2…