Source-linked AI summary
LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation
Sarah Fakhoury, Aaditya Naik, Georgios Sakkas, Saikat Chakraborty, Shuvendu K. Lahiri
TL;DR
LLM-generated code from natural-language intent is difficult to verify because natural language is ambiguous. TICODER uses generated tests and user feedback to clarify intent and rank code suggestions, with reported gains in code-generation accuracy across benchmarks and models.
Problem
Natural-language prompts may not fully capture user intent, and generated code cannot be automatically evaluated for correctness from an ambiguous prompt.
Method
TICODER interactively generates tests, asks users to validate them, and uses the responses to prune and rank LLM-generated code suggestions.
Results
TICODER improves pass@1 performance for all studied LLMs on both benchmarks, with improvements increasing across test-validation interactions.
Takeaways & Limitations
The findings provide encouraging evidence that test-guided intent clarification can support generation of more correct programs.
Takeaways & Limitations
Benchmark findings may not generalize beyond MBPP and HumanEval, other programming languages, or different problem domains.
Abstract
from arXiv · showhide
Large language models (LLMs) have shown great potential in automating significant aspects of coding by producing natural code from informal natural language (NL) intent. However, given NL is informal, it does not lend easily to checking that the generated code correctly satisfies the user intent. In this paper, we propose a novel interactive workflow TiCoder for guided intent clarification (i.e., partial formalization) through tests to support the generation of more accurate code suggestions. Through a mixed methods user study with 15 programmers, we present an empirical evaluation of the effectiveness of the workflow to improve code generation accuracy. We find that participants using the proposed workflow are significantly more likely to correctly evaluate AI generated code, and report significantly less task-induced cognitive load. Furthermore, we test the potential of the workflow at scale with four different state-of-the-art LLMs on two python datasets, using an idealized proxy for a user feedback. We observe an average absolute improvement of 45.97% in the pass@1 code generation accuracy for both datasets and across all LLMs within 5 user interactions, in addition to the automatic generation of accompanying unit tests.
I. INTRODUCTION
The paper addresses ambiguity in natural-language programming intent, which makes it difficult to determine whether LLM-generated code is correct. TICODER uses interactive, LLM-generated tests to clarify intent, filter code suggestions, and improve code-generation accuracy.
- Natural-language prompts can leave user intent ambiguous, while generated code cannot be automatically judged correct from the prompt alone.
- TICODER asks users to validate generated tests, using their feedback to partially formalize intent and rank code suggestions consistent with approved tests.The workflow can present approved tests and semantically ranked code suggestions before generating code with accompanying tests.
- Generated tests can make intent more precise, prune incorrect LLM suggestions, support debugging, and serve as regression tests for later edits.
- The workflow may not scale when users cannot validate complex tests, and its utility depends on useful test generation and favorable interaction costs.
- The evaluation combines a mixed-methods user study with 15 programmers and a large-scale benchmark study using MBPP, HumanEval, and four LLMs.
II. RELATED WORK
Prior work improves code-generation accuracy through automated test generation and test-based grouping or scoring, whereas TICODER incorporates user feedback to clarify informal natural-language intent.
- Unlike these techniques, TICODER explicitly uses user feedback and guides users in clarifying intent formally through generated tests.
- AlphaCode and CodeT generate tests and use the tests that code suggestions satisfy to group, score, or rerank candidate programs.CodeT jointly prioritizes tests satisfied by many suggestions and code suggestions satisfying many tests.
- Program-synthesis approaches typically generate code from formal logical specifications or input-output tests, while TICODER targets informal natural-language specifications.
2) Usability of AI Programming Assistants:
TICODER responds to usability problems in AI programming assistants by structuring intent clarification and code verification around interactive test feedback.
- 2) Usability of AI Programming Assistants:: AI programming assistants can leave users unable to provide feedback or control suggestions that fail to match desired functionality.
- 2) Usability of AI Programming Assistants:: Prior work identifies disambiguating candidate programs and refining initial specifications as important interaction capabilities for code assistants.
- 2) Usability of AI Programming Assistants:: Programmers often iterate on prompts and spend substantial time verifying AI-generated code, shifting effort toward reviewing rather than writing code.
- IV. PROPOSED APPROACH: TICODER: TICODER generates candidate code and tests, executes tests against candidates, and repeatedly asks users whether surfaced tests match their intent.
- IV. PROPOSED APPROACH: TICODER: The PASSFAIL variant requires lighter feedback and generalizes to richer tests, whereas OUTPUT lets users correct test outputs but may require specifying complex oracles.
B. TiCoder Implementation
TICODER generates code and test suggestions from a natural-language prompt, then ranks tests by how effectively they distinguish candidate programs before using user responses to prune and rank code.
- 1) Generating Code and Tests:: The implementation prompts an LLM separately for candidate code and tests, using a placeholder implementation and assertions to generate test suggestions.
- 2) Ranking test suggestions:: Generated tests are executed against candidate programs, and test-ranking information is used to prioritize tests that best discriminate among remaining suggestions.
- 2) Ranking test suggestions:: A discriminative test is preferred when its passing and failing candidate sets are roughly balanced, because either user response can then prune many suggestions.
- 2) Ranking test suggestions:: The same test-ranking strategy is used for PASSFAIL and OUTPUT, although OUTPUT may mutate a test's output after user feedback.
- 2) Ranking test suggestions:: In the running example, t1 and t2 each split three code suggestions 1-to-2 and receive sdiscr = 1/2, while t3 passes on all suggestions and receives 0.
3) Pruning and ranking code suggestions:
TiCoder prunes code suggestions that contradict user-validated tests and ranks the remaining suggestions by how many tests they pass. The study evaluates this workflow against alternative assistant interaction mechanisms.
- Pruning and ranking code suggestions:: TiCoder prunes suggestions whose behavior contradicts a user's PASS, FAIL, or specified-output response.PASS retains code producing the expected output, FAIL removes code producing it, and specified outputs enable further pruning.
- Pruning and ranking code suggestions:: Suggestions are ranked by descending count of tests they pass.For each generated code suggestion, the workflow executes all tests and assigns the number of passing tests as its score.
- Pruning and ranking code suggestions:: The user study compares a no-refinement assistant with TICODER-PASSFAIL and TICODER-OUTPUT across three coding tasks.Fifteen participants use each assistant through a within-subject design, with each assistant assigned to a different task.
1) Control condition: AI Programming Assistant 1:
The study's control assistant displays five pre-generated code suggestions without test-driven pruning. The treatment assistants instead use participant feedback on generated tests to filter suggestions.
- 1) Control condition: AI Programming Assistant 1:: Assistant 1 displays five unique code suggestions in random order as the experiment's control condition.This setup reflects AI coding tools that present multiple candidate completions to users.
- 1) Control condition: AI Programming Assistant 1:: Assistant 2 asks users to mark generated tests as passing or failing, then removes suggestions inconsistent with those judgments.Participants using this workflow are designed to retain three to four suggestions when they correctly evaluate the tests.
- 1) Control condition: AI Programming Assistant 1:: Assistant 3 asks users to provide expected test outputs and removes suggestions producing inconsistent outputs.Both treatment assistants use participant-specified test behavior to prune generated code.
- 1) Control condition: AI Programming Assistant 1:: Tasks were selected to fit fifteen-minute evaluations while containing diverse semantic errors and tests capable of detecting them.The study used short Python-function tasks derived from the MBPP benchmark.
1) Identifying Task Candidates:
The study constructs controlled Python code-evaluation tasks from MBPP examples, using fixed sets of four buggy and one ground-truth program per task. Participants then inspect, run, and select among assistant-provided suggestions.
- 1) Identifying Task Candidates:: Task candidates came from MBPP and were clustered by domain, cyclomatic complexity, and function size.The selection targeted varied problem domains and complexity levels for entry-level Python functions.
- 1) Identifying Task Candidates:: GPT-3.5 generated candidate code and tests, with sampling constrained so tests caught one or two suggestions.This construction produced tasks containing syntactically valid but semantically incorrect programs with varied error types.
- 1) Identifying Task Candidates:: Each task used four buggy AI-generated programs and one MBPP ground-truth implementation, fixed across treatments.Assistants either displayed these five suggestions directly or pruned them after test evaluation.
- 1) Identifying Task Candidates:: Participants could copy code and tests into an IDE, then identify whether any suggestion was correct and which one.The survey interface presented task descriptions and treatment-specific interactions before code evaluation.
- 1) Identifying Task Candidates:: A Latin Square design varied task-assistant pairings and randomized assistant order to balance order effects.Each participant completed three tasks using three different assistants.
E. Evaluation of Measured Variables
The user study found the strongest correctness result for TICODER-PASSFAIL, while test-output interactions introduced additional evaluation errors. The analysis also identifies noisy user feedback as a critical workflow risk.
- E. Evaluation of Measured Variables: Mean correctness was 0.40 with Assistant 1, 0.84 with Assistant 2, and 0.64 with Assistant 3; only Assistant 2's effect was significant (p=0.001).The comparison used mixed-effects regression, with omnibus significance testing reported in Table III.
- E. Evaluation of Measured Variables: Assistant 3 produced errors from both test-output evaluation and code-suggestion evaluation, including two participants who missed the correct suggestion in Task 3.All participants in that task evaluated the surfaced tests correctly, while all Assistant 2 participants selected the correct code.
- E. Evaluation of Measured Variables: Assistant 2 users generally evaluated incorrect edge-case tests correctly, although one of five participants misclassified a Task 1 test output.This illustrates that test validation itself can still introduce mistakes even when the workflow improves code evaluation.
- E. Evaluation of Measured Variables: Incorrect test evaluation can prune every valid program, making noisy user feedback a critical usability risk.The paper identifies skipping a test evaluation as important for reducing the consequences and uncertainty of noisy input.
- E. Evaluation of Measured Variables: Participants using Assistant 2 were significantly more likely to correctly evaluate AI-generated code.Assistant 3 was on average better than the control, but its difference was not reported as significant here.
B. Impact on Task Time
TiCoder did not significantly change task completion time, while participants using its test-supported assistants reported significantly less cognitive load.
- B. Impact on Task Time: Participants using TiCoder took less time on average, but the difference in task completion time was not significant.Mean times were 327.7, 284.15, and 253.88 seconds for Assistants 1, 2, and 3, respectively.
- B. Impact on Task Time: Validating TiCoder-generated tests did not introduce significant overhead to total task time.Test-supported assistants may offset validation overhead by pruning suggestions and supporting evaluation with test cases.
- B. Impact on Task Time: Participants who executed every suggestion took longer than those who selectively tested suggestions that appeared correct.The study found no correlation between time on task and correctness (r = 0.016, p = 0.911).
- B. Impact on Task Time: Participants using Assistants 2 and 3 reported significantly less cumulative cognitive load across the measured NASA TLX dimensions.The authors associate this reduction with fewer suggestions to evaluate and tests providing concrete reasoning mechanisms.
- B. Impact on Task Time: Test cases may reduce evaluation effort by helping participants clarify prompts and begin more extensive testing of candidate functions.The qualitative findings describe tests as concrete mechanisms for reasoning about generated code.
A. Datasets
The benchmark evaluation uses MBPP and HumanEval with multiple open- and closed-source LLMs, measuring baseline and TiCoder-augmented code-generation accuracy.
- A. Datasets: The evaluation uses sanitized MBPP with 427 examples and HumanEval with 164 examples, both containing hidden tests and reference implementations.HumanEval input-output examples in docstrings were removed to avoid making test generation trivial.
- A. Datasets: The study evaluates four state-of-the-art completion models spanning closed-source and open-source systems.The models include OpenAI completion and chat models and Salesforce CodeGen variants.
- A. Datasets: Model comparisons use default configurations while varying temperature, which is fixed at 0.8 to balance solution diversity and correctness coverage.The same cached generations are reused across experiments for each model and dataset.
- A. Datasets: Table IV reports baseline pass@1 and pass@100 alongside TiCoder results for the two datasets and up to five simulated interactions.The table highlights the highest accuracy in each column.
D. Simulating User Response
Because offline benchmarks cannot receive real-time user feedback, the evaluation simulates it with reference implementations, then measures TiCoder across models, datasets, and interaction counts.
- D. Simulating User Response: The offline evaluation uses the reference implementation as an oracle for whether generated tests match user intent.This assumes that user intent is precisely captured by the hidden reference implementation’s semantics.
- D. Simulating User Response: TiCoder significantly improves pass@1 performance across all studied models, with gains increasing from one to five test-validation interactions.Both TICODER-PASSFAIL and TICODER-OUTPUT improve over baseline, with the largest improvement generally appearing at m = 1.
- D. Simulating User Response: On MBPP, TICODER-PASSFAIL raises text-davinci-003 pass@1 from 49.16% to 68.04% after one user query.The corresponding absolute improvement is 18.88%.
- D. Simulating User Response: After one interaction, code-davinci-002 reaches 68.42% on MBPP with TICODER-PASSFAIL, exceeding the pass@1 accuracy of the three studied SOTA chat models.This illustrates that TiCoder can raise smaller-model performance to the level of larger models in this setting.
- D. Simulating User Response: TICODER-OUTPUT achieves a 45.73% average absolute accuracy improvement across both datasets and all LLMs within five interactions.It outperforms TICODER-PASSFAIL because users can correct incorrect test outputs.
- D. Simulating User Response: The lightweight TICODER-PASSFAIL setting remains within 9% of TICODER-OUTPUT’s performance even in the idealized simulated-user setting.This suggests that simple pass/fail feedback retains most of the richer feedback setting’s benefit.
B. Improving LLM code generation capabilities with verified test cases
TICODER improves code-generation accuracy by using executed, user-verified tests to prune and rank candidate programs. Its value depends on test quality, but even one highly distinguishing interaction can rival prompting with many validation tests.
- Execution-based pruning: Tests improve pass@k accuracy by constraining candidate programs and enabling TICODER to improve their ranking and pruning.The benchmark results cover both TICODER-PASSFAIL and TICODER-OUTPUT.
- Execution-based pruning: TICODER remains useful when suggestions behave consistently because it supplies meaningful tests and behavioral guarantees even without pruning.This includes cases where all suggestions are correct with respect to the user’s intent.
- Execution-based pruning: 81.56% pass@1@1 accuracy is reached by TICODER-PASSFAIL within one feedback loop, exceeding the 78.26% achieved by prompting GPT-4-32k with one test.On MBPP, TICODER is also 0.68% better than prompting with all validation tests.
- Execution-based pruning: A highly distinguishing generated test can improve accuracy with one interaction without requiring users to manually construct a complete test suite.The result matches prompting with several tests while avoiding the added burden of manually supplying them.
- AI-generated test considerations: Precondition-violating tests can produce UNDEFINED feedback and therefore provide no code pruning.In one example, 61 of 80 suggestions crashed on the test, while 8 passed and 11 failed; the reference implementation failed on an empty array.
IX. LIMITATIONS AND THREATS
The evaluation is controlled and limited to selected Python benchmarks, so its generalizability to real-world development, broader domains, and practical usage remains uncertain. TICODER also adds inference and test-execution costs, motivating evaluation in more realistic settings.
- Generalizability of user study results: The user study restricts prompt and suggestion editing and measures code-evaluation correctness rather than broader productivity or code-acceptance outcomes.Future work is proposed in realistic code settings with broader audiences and additional online metrics.
- Generalizability of user study results: Test validation may become difficult for more complex tasks, especially when users must specify generated-test outputs or work with intricate testing frameworks.The study used controlled conditions and two test-validation mechanisms, TICODER-PASSFAIL and TICODER-OUTPUT.
- Generalizability of benchmark evaluation results: The benchmark results may not generalize beyond the MBPP and HumanEval Python datasets to other languages, programs, or problem domains.Both benchmarks exercise common programming patterns but may not represent real-world software development.
- Test execution overhead: TICODER incurs additional LLM inference and test-execution costs that may not scale for complex code generation scenarios.The paper frames reliability guarantees and reduced verification effort as a trade-off against these costs.
- Future directions: The paper plans evaluation in realistic benchmarks and richer specifications, including property-based tests and pre- and post-conditions.These directions target more complex programs and in-situ software-development settings.