Source-linked AI summary
Large Language Models Should Ask Clarifying Questions to Increase Confidence in Generated Code
Jie JW Wu
TL;DR
LLMs can generate code, but a gap remains between coding capability and software-engineering work that includes communication and requirements. This paper explores a coder-communicator process that asks users clarifying questions and uses their responses to refine code. Its initial example suggests potential improvements, while the author identifies communication techniques and prompting as areas for further work.
Problem
Code-generation research rarely evaluates communication, and one-off prompts leave models unable to clarify incomplete or error-prone requirements.
Source: §1 · p. 1Method
A communicator reviews the problem description and prior code for uncertain parts, asks the user clarifying questions, and sends responses to a coder for refinement.
Source: §2 · pp. 1–2Results
The initial Fibonacci example suggests that clarifying questions may improve final code output on topics such as negative input and documentation.
Source: §3 · pp. 2–3Takeaways & Limitations
The author argues that communication skills are an important factor toward bridging the gap between LLMs and software developers, despite the additional conversational effort required from users.
Source: §5 · p. 3The work is an initial step, with further work needed on effective communication instructions and tuning models to vary their communication level.
Source: §5 · p. 5
Abstract
from arXiv · showhide
Large language models (LLMs) have significantly improved the ability to perform tasks in the field of code generation. However, there is still a gap between LLMs being capable coders and being top-tier software engineers. Based on the observation that toplevel software engineers often ask clarifying questions to reduce ambiguity in both requirements and coding solutions, I argue that the same should be applied to LLMs for code generation tasks. By asking probing questions in various topics before generating the final code, the challenges of programming with LLMs, such as unclear intent specification, lack of computational thinking, and undesired code quality, may be alleviated. This, in turn, increases confidence in the generated code. In this work, I explore how to leverage better communication skills to achieve greater confidence in generated code. I propose a communication-centered process that uses an LLM-generated communicator to identify issues with high ambiguity or low confidence in problem descriptions and generated code. I then ask clarifying questions to obtain responses from users for refining the code.
1 INTRODUCTION
LLMs can generate code from natural-language requirements, but reported problems with intent specification, decomposition, code quality, overconfidence, and usability leave them short of software engineers. This paper examines whether clarifying questions can increase confidence in ChatGPT-generated code.
1 INTRODUCTION
- LLMs generate code from natural-language requirements, but reported issues include intent specification, problem decomposition, code quality, overconfidence, and usability.The paper frames these gaps as evidence that current LLMs cannot fully replace professional software developers.Source: §1 · p. 1
- The paper asks whether clarifying questions increase confidence in ChatGPT-generated code, addressing communication as a link between coding and broader software-development concerns.Those concerns include requirements, design, domain knowledge, and project context.Source: §1 · p. 1
- Code-generation evaluations commonly use one-off problem descriptions, while professional developers communicate to clarify requirements, project context, and design alternatives.Incomplete or error-prone descriptions therefore leave models without an opportunity to ask necessary questions.Source: §1 · p. 1
- The paper proposes an initial communication-centered exploration, whose empirical example is described as promising for boosting LLM communication skills in code generation.The stated aim is to refine generated code through clarifying questions.Source: §1 · p. 1
2 COMMUNICATION-CENTERED PROCESS
The proposed process pairs a coder with a communicator that identifies uncertain parts of the description or prior code and asks the user questions. User responses return to the coder to refine the code, and the loop can continue until a condition is met.
2 COMMUNICATION-CENTERED PROCESS
- A coder generates code from the problem description and communicator information, while a communicator reviews the description and prior code for low-confidence parts.The communicator asks clarifying questions, returns the user's responses to the coder, and the process can repeat until a condition is met.Source: §2 · pp. 1–2, Figure 1 · p. 2
- The communicator targets uncertain or low-confidence parts with questions about intent, requirements, and coding style to obtain useful information from users.Either component can be implemented using an LLM.Source: §2 · p. 2
3 EXAMPLE USAGE
A ChatGPT 3.5 coder and communicator demonstrate the process on a request for an nth Fibonacci number. The communicator surfaces unspecified negative-input behavior, and the example illustrates possible refinements through user responses and tailored question topics.
3 EXAMPLE USAGE
- The communicator's top question in the Fibonacci example concerns negative input, which the problem description had not specified.The user sets an expectation, after which the coder refines the code; the example uses one iteration, though the loop could continue until criteria are met.Source: §3 · p. 2
- ChatGPT 3.5 serves as both coder and communicator, with the communicator using zero-shot prompting on the task description and previous code to generate clarifying questions.The prompt asks for questions that may lead to code refinement.Source: §3 · p. 2
- A simple zero-shot prompt already produces questions on multiple topics in the example, and the author suggests prioritizing topics for customized use or code quality.The passage points to code context, coding style, non-functional requirements, code analysis, and explanation as possible priorities.Source: §3 · pp. 2–3
- The author sees potential for communication-focused prompting to improve final code output on topics including negative input and documentation.This is presented as potential inferred from the example, not a broader benchmark result.Source: §3 · p. 3
4 BACKGROUND AND RELATED WORK
Related work describes code-generation models and evaluations centered on one-off descriptions, alongside methods for LLM self-correction. The paper positions its exploration as addressing communication and usability through clarifying questions and user feedback.
4 BACKGROUND AND RELATED WORK
- Code-generation models such as Codex [7], AlphaCode [17], and CodeGen [24] demonstrate capability, but their communication skills are not emphasized or evaluated.The discussed evaluations generate code from one-off problem descriptions without further conversational information.Source: §4 · p. 3
- When descriptions are incomplete or error-prone, one-off generation leaves models without a chance to clarify critical questions; the paper frames its work as exploring this usability problem.The passage describes this as a limitation of the evaluation setup discussed.Source: §4 · p. 3
- The paper places its approach among self-correction methods using human and automated feedback, while distinguishing its focus on communication skills for code generation.Related approaches include human-feedback refinement, self-training, generate-then-rank, feedback-guided decoding, and iterative post-hoc revision.Source: §4 · p. 3
5 CONCLUSION AND FUTURE WORK
The paper presents a coder–communicator process as an initial step toward using clarifying questions to refine generated code and increase confidence. It argues this communication work may support user trust, while identifying evaluation and communication-improvement methods as future work.
5 CONCLUSION AND FUTURE WORK
- The exploratory coder–communicator process asks clarifying questions and refines generated code, aiming to increase confidence and user trust.The author describes it as an initial step and says user input adds effort.Source: §5 · pp. 3, 5
- An objective benchmark is needed to quantify communication skills in LLM-based code generation; blurred or noisy problem descriptions are one proposed test direction.In that setting, models should ask appropriate questions rather than directly generate low-quality code.Source: §5 · p. 5
- Future work includes designing communication-focused prompts and tuning models to avoid under- or over-communicating.The author envisions future programming agents with different communication levels and styles.Source: §5 · p. 5