PaperlayerView exact revision on arXiv

Source-linked AI summary

Interactive Code Generation via Test-Driven User-Intent Formalization

Shuvendu K. Lahiri, Sarah Fakhoury, Aaditya Naik, Georgios Sakkas, Saikat Chakraborty, Madanlal Musuvathi, Piali Choudhury, Curtis von Veh, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao

arXiv:2208.05950v2cs.SEcs.LGcs.PL

TL;DR

Natural-language intent is ambiguous, leaving users without guarantees that generated code matches what they meant. ITDCG uses lightweight feedback on generated tests to formalize intent and prune and rank code suggestions. On MBPP and HumanEval, the reported pass@1 gains range from 22.49% to 37.71% and from 24.79% to 53.98%, respectively, across one to five simulated queries.

  • Problem

    Natural-language intent is ambiguous and lacks formal semantics, so users cannot readily determine whether generated code satisfies their intent.

    Source: §I · p. 1

  • Method

    ITDCG uses lightweight user feedback on generated tests to formalize intent and prune and rank code suggestions.

    Source: §I · p. 2

  • Results

    The best algorithm improves pass@1 accuracy by 22.49% with one query and 37.71% with five on MBPP, and by 24.79% and 53.98%, respectively, on HumanEval.

    Source: §I · p. 2

  • Takeaways & Limitations

    The workflow studies test-driven interaction as a way to improve code and test generation accuracy using test mutation, pruning, and ranking.

    Source: §VIII · p. 11

    The evaluation is simulated; a user study and evaluation on real-world benchmarks such as CoderEval and nl2fix remain future work.

    Source: §VIII · p. 11

Abstract

from arXiv · show

Large language models (LLMs) have shown great potential in automating significant aspects of coding by producing natural code from informal natural language (NL) intent. However, when interacting with LLMs, users have no guarantees that the code suggestions produced correctly satisfy the intent they provided. In fact, it is hard to define a notion of correctness since natural language can be ambiguous and lacks a formal semantics. In this paper, we propose the workflow of {\it interactive test-driven code generation}, which leverages lightweight user feedback to (a) formalize the user intent using generated tests that can be useful for debugging, and (b) produce an improved set of code suggestions by pruning and ranking candidate code suggestions. We describe a language-agnostic abstract algorithm and a concrete implementation TiCoder. We perform an automated evaluation of TiCoder on the \emph{MBPP} and \emph{HumanEval} code generation benchmarks. Our results are promising with using the OpenAI Codex LLM: our best algorithm improves the \passk{1} code generation accuracy (in absolute percentages) between $22.49\%$ to $37.71\%$ for MBPP and between $24.79\%$ to $53.98\%$ for HumanEval using between 1 to 5 simulated user queries.

I. INTRODUCTION

Natural-language intent can admit multiple plausible programs, making correctness difficult to define; ITDCG uses lightweight feedback on generated tests to formalize intent and prune and rank code suggestions.

I. INTRODUCTION

  • Ambiguous descriptions can yield plausible programs with different behavior, while natural language's lack of precise semantics makes generated-code correctness hard to articulate.Users may struggle to understand suggestions, and scanning a long list can require rejecting incorrect candidates one by one.Source: §I · p. 1
  • ITDCG uses lightweight user feedback on generated tests as a weak formal specification to formalize intent and produce code suggestions consistent with those tests.The proposed workflow is motivated by example generation in PBE and LLMs' emerging ability to generate tests,,, [19].Source: §I · p. 2
  • Generated tests can clarify intent, prune incorrect suggestions, and serve as debugging aids and regression tests for remaining code suggestions.Source: §I · p. 2
  • The evaluation considers whether interaction costs are balanced by accepted tests and their benefits for pruning, ranking, or repairing code suggestions.The paper identifies objective cost-benefit metrics as necessary to assess the workflow's utility.Source: §I · p. 2
  • 22.49% (1 query) and 37.71% (5 queries) MBPP pass@1 gains; HumanEval gains are 24.79% and 53.98%, respectively.The abstract algorithm and TiCoder are evaluated on MBPP and HumanEval; the reported gains are absolute percentages over the LLM-based baseline.Source: §I · p. 2

II. WORKFLOW AND PROBLEM FORMULATION

This section introduces autoregressive code language models and outlines a workflow that uses test generation and user feedback to refine intent, then defines metrics and an abstract algorithm.

II. WORKFLOW AND PROBLEM FORMULATION

  • The section outlines a workflow using generated tests and user feedback to disambiguate and refine user intent.Source: §II · p. 2
  • It also defines intuitive metrics for comparing approaches that implement the workflow.Source: §II · p. 2
  • The section concludes its overview by providing an algorithm that describes the workflow.Source: §II · p. 2

A. Background: Large Language Models for Code

B. High-level Workflow

The workflow generates code and test suggestions, asks users whether tests match their intent, and uses responses to refine suggestions; the section also defines metrics for code and test quality.

B. High-level Workflow

  • The user request supplies a file prefix, natural-language description, and function signature for the agent to complete.Source: B. High-level Workflow · p. 2
  • The agent generates candidate code and test suggestions, selects tests for user judgments, then uses responses to prune, rank, and mutate suggestions.At termination, it returns user-approved tests and a ranked list of code suggestions consistent with the responses.Source: B. High-level Workflow · pp. 2–3, Figure 1 · p. 3
  • User responses may be YES, NO, or UNDEFINED; the third option covers tests with ill-defined outcomes, such as precondition violations, parse errors, or flaky behavior.Source: B. High-level Workflow · p. 3
  • The workflow seeks one of three responses rather than richer feedback such as asking users to provide desired outputs, and it also generalizes to richer tests and specifications.For Boolean-returning functions such as the running example, the distinction from richer feedback does not apply.Source: B. High-level Workflow · p. 3

C. Metrics

  • pass@k@m measures code-suggestion correctness after m user queries; pass@k@0 is the pass@k metric.A suggestion is correct if it passes all hidden tests, and pass@k measures the expected chance of selecting at least one correct suggestion among k samples.Source: C. Metrics · p. 3
  • accept@m is 1 when at least one of the m test queries matches user intent, and 0 otherwise.Source: C. Metrics · p. 3
  • pass@k@m also indirectly reflects test quality by favoring tests that better prune incorrect code suggestions.Source: C. Metrics · p. 3

D. Abstract Algorithm

The abstract algorithm generates candidate code and tests, then uses user feedback on ranked tests to refine its candidate implementations. It returns user-approved tests and a ranked list of implementations that satisfy them.

D. Abstract Algorithm

  • The formalized setting is a single function completed within a program tuple, with tests represented as input-output pairs.A function satisfies a test when execution terminates with the specified unique output.Source: D. Abstract Algorithm · p. 3
  • The workflow returns user-approved tests and a ranked list of candidate implementations consistent with those tests.It accepts a prefix, natural-language description, function header, and interaction limit.Source: D. Abstract Algorithm · p. 4
  • It generates code and test suggestions, then can modify the tests through syntactic and execution-based mutation.The initial test set depends on the LLM and prompts; seed tests use LLMs because the method body is unavailable and automated alternatives are language-specific.Source: D. Abstract Algorithm · p. 4
  • The returned code set remains a subset of the initial suggestions, and the algorithm's invariant prevents it from pruning code that passes the hidden tests.The returned approved tests record user-accepted input-output examples.Source: D. Abstract Algorithm · p. 4
  • Each round presents the highest-ranked test to the user; acceptance or rejection prunes code suggestions according to whether they agree with that test.The algorithm then reranks the remaining code suggestions, and takes no pruning action for an UNDEFINED response.Source: D. Abstract Algorithm · p. 4

III. TICODER

TICODER implements the abstract algorithm by offering alternative prompt-generation and test-mutation components. Its prompts generate code and test suggestions, while static mutations repair or simplify generated tests.

III. TICODER

  • TICODER implements the abstract algorithm's configurable components, including alternative implementations for code prompting and test ranking.The tool is called Test-driven Interactive Coder.Source: §III · p. 4

A. Code and Test Generation Prompts

  • The code prompt produces candidate implementations, while the test prompt uses a placeholder function body to elicit test suggestions.In the running example, the generated code and tests include both intent-consistent and inconsistent suggestions.Source: A. Code and Test Generation Prompts · pp. 4–5, Figure 3 · p. 5
  • Test generation can use either a pass placeholder or a sampled candidate implementation as the function body.The sampled option uses the first code suggestion returned by the LLM.Source: A. Code and Test Generation Prompts · p. 5

B. Static mutation of tests with SyntacticMutateTests

  • Static mutation keeps the longest parsable prefix after a parsing error or truncates a test at its first assertion.The single-assert choice can weaken test checks and adversely affect performance.Source: B. Static mutation of tests with… · p. 5

C. Dynamic mutation of tests with DynMutateTests

Dynamic mutation executes candidate implementations to create alternate test outputs, while the ranking section considers how to choose useful tests for user queries. An ideal ranking policy provides an upper bound but requires knowing which code suggestions are correct.

C. Dynamic mutation of tests with DynMutateTests

  • Dynamic mutation executes each candidate on a test input and uses the observed output to create an alternate test.This can produce a test consistent with a correct candidate and is more effective when outputs come from larger value spaces.Source: C. Dynamic mutation of tests with… · p. 5

D. Ranking test suggestions using RankTests

  • After ranking, the user answers a selected test, which prunes code suggestions inconsistent with the response.The test is selected from the generated and mutated set.Source: D. Ranking test suggestions using… · p. 5
  • The ranking objective is to show tests that prune the largest number of incorrect code suggestions while reducing user interactions.An ideal policy simulates each test's pruning effect, establishing an upper bound, but requires labeling candidates as correct or incorrect.Source: D. Ranking test suggestions using… · p. 5, 1) Ideal · p. 5

1) Ideal:

The section contrasts a random test-selection baseline with a discriminative ranking heuristic. The latter favors tests that split candidate code suggestions into similarly sized pass and fail groups.

1) Ideal:

2) Random:

  • Random ranking presents a randomly selected test without estimating its usefulness.Source: 2) Random · p. 5

3) Discriminative:

  • Discriminative ranking orders tests by how evenly they divide passing and failing code suggestions, assuming each suggestion is equally likely to be correct.In the example, t1 and t2 tie despite the ideal policy preferring t2, while t3 passes all suggestions and is not selected.Source: 3) Discriminative · p. 6
  • Runtime exceptions and precondition failures are excluded from the failing-candidate group because users are expected to answer UNDEFINED for those tests.Source: 3) Discriminative · p. 6

E. Ranking code suggestions using RankCodes

RankCodes orders candidate implementations by the number of generated tests they pass. The evaluation then asks how interaction affects accuracy, how performance changes with additional queries, and how design choices contribute.

E. Ranking code suggestions using RankCodes

  • RankCodes scores each code suggestion by its number of passing tests and ranks suggestions in decreasing score order.Without user feedback, the example ranks c1 first because it passes all three tests.Source: E. Ranking code suggestions using… · p. 6

IV. EXPERIMENTAL EVALUATION

  • The evaluation compares approaches for interactive test-driven code generation and describes its experimental setup before reporting results.Source: §IV · p. 6

A. Research Questions

  • The research questions ask whether interaction improves code accuracy, how query counts affect code and test correctness, and how design decisions affect metrics.Source: A. Research Questions · pp. 6–7

B. Dataset

  • Evaluation uses 427 MBPP tuples and 164 HumanEval tuples, removing HumanEval docstring examples that could make intent-consistent tests easy to construct.Source: B. Dataset · p. 7

C. Experimental setup and tools

The experiments use cached Codex-generated code and tests, compare TiCoder with several baselines, and simulate user responses using each benchmark’s reference implementation.

C. Experimental setup and tools

  • Codex code-davinci-002 generates 100 code suggestions and 50 test suggestions per problem, with temperature 0.8, top p 0.95, and a 300-token code limit.A single set of initial responses is cached and reused to account for model nondeterminism.Source: C. Experimental setup and tools · p. 7
  • TiCoder is compared with non-interactive Codex, a TiCoder baseline without test mutation, pruning, or code ranking, and the non-interactive CodeT implementation.Source: C. Experimental setup and tools · p. 7
  • IdealTests uses hidden tests ranked ideally as an upper bound, while the default TiCoder configuration combines pass prompting, test mutations, discriminative test ranking, and passing-tests code ranking.The default was selected empirically for the best MBPP pass@1@1 among configurations considered.Source: C. Experimental setup and tools · p. 7

D. Automated evaluation

  • Because offline evaluation cannot obtain real-time user responses, the benchmark reference implementation serves as an oracle for whether tests match intent.The evaluation assumes user intent is exactly the reference function’s semantics for terminating executions.Source: D. Automated evaluation · pp. 7–8
  • The oracle returns YES when the reference terminates with the test’s expected output, NO for a different output, and UNDEFINED for errors or nontermination.Source: D. Automated evaluation · p. 8

V. RESULTS

TiCoder improves single-query code accuracy over Codex and the simple baseline on both benchmarks, while additional interactions improve its performance and yield intent-consistent tests for many examples.

A. RQ1: Code suggestion accuracy

  • 70.73% pass@1@1 on MBPP exceeds Codex’s 48.24%, while 55.28% on HumanEval exceeds Codex’s 30.49% but trails CodeT’s 58.54%.An alternate TiCoder configuration reaches 59.62% on HumanEval pass@1@1.Source: A. RQ1: Code suggestion accuracy · p. 8
  • TiCoder’s pass@1@1 trails IdealTests by 16.62% on MBPP and 35.57% on HumanEval, while IdealRanking indicates up to 13.58% and 14.29% potential improvement, respectively.Source: A. RQ1: Code suggestion accuracy · p. 8

B. RQ2: Impact of user interaction

  • Increasing query limits raises performance slightly for Baseline but considerably for TiCoder, which approaches IdealRanking after five interactions.IdealTests reaches pass@100 after all hidden tests, and improved test ranking could further boost TiCoder with fewer interactions.Source: B. RQ2: Impact of user interaction · p. 9
  • After five interactions, test mutation improves accuracy over the purely LLM-generated test pool by 10.72% on MBPP and 13.74% on HumanEval.Source: B. RQ2: Impact of user interaction · p. 9
  • TiCoder outperforms CodeT with one user interaction on MBPP and two on HumanEval.Source: B. RQ2: Impact of user interaction · p. 9
  • Within 10 queries, TiCoder finds an intent-consistent test for 87.12% of MBPP and 95.73% of HumanEval examples; the first query succeeds for 61.12% and 73.17%, respectively.The reported average is 1.7 interactions for MBPP and 1.5 for HumanEval.Source: B. RQ2: Impact of user interaction · p. 9, Figure 7 · p. 9

C. RQ3: Ablation study

The ablations show that most components affect performance, but no configuration works well across every metric and both benchmarks. The effects of test and code ranking, mutation, and prompt choices vary by metric and dataset.

C. RQ3: Ablation study

  • The default was selected for MBPP pass@1@1; on HumanEval, a different code prompt scored higher on pass@1@1 and pass@2@1.The authors prioritize pass@1@1 as a natural interactive metric because it uses at most one user interaction and produces one final suggestion; finding one configuration optimal across both benchmarks is future work.Source: C. RQ3: Ablation study · p. 10
  • No configuration performed well across every metric on both benchmarks, although most components had a non-trivial effect on overall performance.Randomly selected tests performed uniformly worse than the default on both datasets; removing dynamic mutation reduced MBPP performance but raised HumanEval pass@5@1.Source: C. RQ3: Ablation study · p. 10
  • Code ranking helped MBPP pass@1@m but hurt pass@k@1 for k ≥2; on HumanEval it helped with one suggestion but could harm larger sets.The HumanEval result is reported for a nonpass prompt; the passage does not specify the larger-set metric values.Source: C. RQ3: Ablation study · p. 10
  • Disabling static test mutation improved pass@2@1, while including code suggestions in test-generation prompts improved pass@1@2; both lost on pass@1@1.These variants were not selected as defaults because each performed worse on pass@1@1.Source: C. RQ3: Ablation study · p. 10

VI. THREATS

The paper reports threats to generalization, model-output stability, and its simulated interaction evaluation. It relates its workflow to prior synthesis approaches and concludes that its dynamic components improve code and test generation accuracy, while real-world evaluation and user studies remain future work.

VI. THREATS

  • Evaluation uses MBPP and HumanEval Python benchmarks, potentially limiting generalization; Codex output changes and simulated users’ cognitive effort is unmeasured.The interaction simulation uses reference implementations as an oracle and assumes users can answer generated tests.Source: §VI · p. 10

VII. RELATED WORK

  • AlphaCode and CodeT group suggestions by tests, but do not model user interaction or guarantee suggested code; adapting CodeT ranking is future work.CodeT jointly scores tests and code suggestions.Source: §VII · p. 10
  • Unlike formal-specification synthesis, LLM code generation starts from informal specifications and hidden tests; user-provided tests for improving code generation are a future direction.Source: §VII · p. 10
  • Unlike classical synthesis that starts with examples, this workflow starts with natural-language intent and requests three-valued feedback; augmented tests can serve as regression tests.Prior approaches often use symbolic engines to generate distinguishing inputs, which the passage says is inconceivable for general-purpose imperative languages such as Python.Source: §VII · pp. 10–11

VIII. CONCLUSIONS

  • The study explores dynamic approaches and reports non-trivial improvements in code and test generation accuracy using components that mutate, prune, and rank suggestions.Source: §VIII · p. 11
  • Real-world benchmark evaluation, a user study, and extensions beyond tests to richer specifications remain future work.The proposed richer forms include parameterized tests and procedure summaries, given scalable generators and checkers.Source: §VIII · p. 11
Loading 2208.05950v2…