PaperlayerView exact revision on arXiv

Source-linked AI summary

ClarifyCodeBench: Evaluating LLMs on Clarifying Ambiguous Requirements for Code Generation

Zheng Fang, Dongming Jin, Yihong dong, Yongmin Li, Kechi Zhang, Zhi Jin, Ge Li

arXiv:2607.00711v2cs.SEcs.AI

TL;DR

Ambiguous requirements can lead to code that misses users’ intent, but existing evaluation has given limited attention to models clarifying that ambiguity. ClarifyCodeBench evaluates interactive clarification, and its six-model study finds that strong coding performance does not ensure effective clarification, especially as ambiguity increases.

  • Problem

    Requirements are often ambiguous or incomplete, making it important to know whether LLMs can recognize ambiguity and clarify user intent through interaction.

    Source: §1 · pp. 1–2

  • Method

    ClarifyCodeBench evaluates interactive clarification on underspecified coding tasks with annotated questions, answers, ambiguity types, executable tests, and the TKQR and ORA metrics.

    Source: §1 · p. 2

  • Results

    Across six LLMs, strong code generation does not ensure effective clarification; reasoning brings marginal gains in ambiguity identification, and clarification performance drops sharply as ambiguity increases.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    Reliable LLM-based code generation requires not only synthesis ability but also better support for ambiguity-aware interaction.

    Source: §7 · p. 10

    Because tasks derive from public LiveCodeBench problems, closed-source models may have seen the original statements and infer intended meanings rather than demonstrate clarification ability.

    Source: §6.2 · p. 10

Abstract

from arXiv · show

Large Language Models have emerged as programming assistants. However, the efficacy of code generation is constrained by the quality of input requirements, which are frequently ambiguous, incomplete, or underspecified. While LLMs excel at one-shot code synthesis, their ability to proactively clarify intent remains underexplored, as a critical trait for robust software engineering. Existing benchmarks largely overlook this interactive bottleneck, assuming perfectly specified prompts that do not reflect the iterative nature of requirement elicitation. To bridge this gap, we introduce ClarifyCodeBench, a novel interactive benchmark for evaluating LLMs' capability in resolving requirement ambiguity. Constructed from real-world programming tasks, ClarifyCodeBench features high-quality manual annotations, including N unique ambiguity types, associated clarification questions, and corresponding ground-truth answers. Furthermore, we formalize two rigorous metrics to assess the interaction quality: Turn-discounted Key Question Rate, which penalizes inefficient questioning, and Optimal Round Adherence, which measures the precision of the elicitation process. We conduct a systematic evaluation of six state-of-the-art LLMs using ClarifyCodeBench. Our empirical results yield three critical insights: 1) Capability Decoupling: Strong code generation performance does not inherently translate to effective requirement clarification; 2) The Reasoning Paradox: While increased computational thinking enhances code correctness, it yields marginal gains in identifying ambiguities; 3) The Multi-ambiguity Ceiling: LLMs' clarification performance degrades sharply as the density of ambiguities increases, revealing a significant bottleneck in handling complex, real-world specifications. Our work underscores the necessity for future AI4SE research to transition from static synthesis to interactive elicitation.

1 Introduction

Ambiguous and incomplete requirements can cause code that misses users’ intent, yet existing benchmarks largely test generation from specified requirements. ClarifyCodeBench evaluates interactive clarification, and results across six LLMs show that coding strength does not ensure effective clarification.

1 Introduction

  • Requirements can admit multiple interpretations, so code generated from an implicit assumption may fail to match the user’s intent.The introduction illustrates this with two possible interpretations of an expression and notes that practical requirements are rarely fully specified at the outset.Source: §1 · pp. 1–2
  • Existing benchmarks mainly test code correctness from fully specified requirements, while interactive benchmarks often begin after requirements have already been refined.They do not test the full sequence of detecting initial ambiguity, asking appropriate questions, and using the answers to generate code.Source: §1 · p. 2
  • ClarifyCodeBench evaluates clarification through underspecified tasks, annotated questions and answers, ambiguity types, and executable tests, using TKQR and ORA alongside functional correctness.TKQR rewards early key questions while penalizing delayed or redundant interaction; ORA measures whether models use the appropriate number of rounds.Source: §1 · p. 2
  • Across six LLMs, strong code generation does not ensure effective clarification; reasoning yields marginal gains in ambiguity identification, and clarification degrades sharply as ambiguity increases.The benchmark study identifies these as three central findings about capability, reasoning, and ambiguity density.Source: §1 · p. 2

2 Background

Code-generation benchmarks have expanded from standalone tasks to richer and interactive settings, but generally do not test whether models resolve ambiguity in initial requirements. Clarification systems address this area, while fine-grained ambiguity characterization remains limited.

2.1 Code Generation Benchmarks

  • HumanEval, MBPP, APPS, and LiveCodeBench broaden code-generation evaluation across standalone tasks, difficulty levels, and more recent programming problems.ConvCodeWorld and CodeFlowBench add multi-turn coding workflows involving feedback or iterative development, but these settings still assume sufficiently specified requirements.Source: §2.1 · p. 2
  • Existing benchmarks do not directly assess whether models identify ambiguity in initial requirements and clarify it before generating code.Table 1 contrasts their focus on functional correctness or interactive coding with this dataset’s ambiguity-aware clarification and explicit ambiguity annotations.Source: §2.1 · pp. 2–3
  • HumanEvalComm examines clarification for problematic requirements, but lacks fine-grained instance-level ambiguity and ambiguity-type annotations for systematic analysis.This limits comparisons of which ambiguity types are easier or harder to recognize and clarify.Source: §2.1 · p. 3

2.2 Clarification under Ambiguous Requirements

  • TICODER, ClarifyGPT, and ClarifyCoder explore requirement refinement or model-level clarification, while this work focuses on ambiguity-centered data construction and evaluation.Prior work provides limited systematic characterization of ambiguity types and detailed diagnosis of recognition and clarification difficulty.Source: §2.2 · p. 3

3 Benchmark Construction

Section 3 motivates ClarifyCodeBench and describes its construction, clarification metrics, and interactive evaluation protocol. The benchmark removes information from complete coding tasks and evaluates whether models recover it through clarification before generating code.

3.1 Motivation Experiment

  • Ambiguous requirements sharply reduce pass@1, while interactive and oracle clarification improve over ambiguous-only inputs, motivating direct evaluation of clarification ability.Oracle clarification remains slightly below complete requirements.Source: §3.1 · p. 3

3.2 Dataset Construction

  • ClarifyCodeBench starts from LiveCodeBench v6 and creates ambiguity by deleting needed information while preserving each task’s original objective.Each task receives one to three ambiguity points, with a key question and answer drawn from the deleted information.Source: §3.2 · pp. 3–4
  • Two PhD annotators independently annotate and cross-check tasks, resolving disagreements by consensus and verifying plausibility, genuine ambiguity, and answer sufficiency.These checks are intended to improve annotation quality and consistency while reducing subjective drift.Source: §3.2 · p. 4
  • The task-specific ambiguity taxonomy is grounded in requirement-quality principles; Terminology and Output Format are the most common categories.Table 2 pairs each type with a definition and example clarification question.Source: §3.2 · p. 4, Figure 3 · p. 4, Table 2 · p. 5
  • The resulting benchmark contains 419 ambiguous tasks: 199 with one ambiguity, 169 with two, and 51 with three.It supports evaluating both clarification and code generation after answers are provided.Source: §3.2 · p. 4

3.3.1 Turn-discounted Key Question Rate.

  • Turn-discounted Key Question Rate (TKQR) adapts normalized DCG to reward key questions asked early and penalize delayed or non-key turns.Its ideal case asks key questions as early as possible, up to min(n, K) turns, and the score is normalized by that case.Source: §3.3.1 · p. 4

3.3.2 Optimal Round Adherence.

  • Optimal Round Adherence (ORA) rewards using the optimal number of question rounds and decreases as the actual count deviates from that target.The metric complements TKQR, which measures whether key questions appear early; ORA measures whether questioning stops after a reasonable number of rounds.Source: §3.3.2 · p. 5

3.4 Evaluation Protocol

  • The protocol standardizes a choice between asking exactly one clarification question and returning final code, then continues question-and-answer rounds until code or a preset limit.An LLM judge matches questions to annotated key questions using three trials and majority vote; unmatched questions receive a default reply.Source: §3.4 · p. 5

4 EXPERIMENTAL SETUP

The experiments evaluate six advanced LLMs on clarification and code-generation behavior under an interactive protocol. The setup specifies research questions, complementary metrics, and run limits.

4 EXPERIMENTAL SETUP

  • The study asks how well LLMs ask clarification questions, how ambiguity type and difficulty affect performance, how many turns they use, and what errors they make.These questions are framed as RQ1–RQ5.Source: §4 · pp. 5–6

4.1 Studied LLMs

  • Six models span non-reasoning and reasoning systems: GPT-4o, GPT-5, Gemini-2.5-Flash, Claude-Sonnet-4.5, DeepSeek-V3.2, and Qwen3-235B-A22B.The paper selects models reported to have strong results on code-generation benchmarks.Source: §4.1 · p. 6

4.2 Evaluation Metrics

  • The evaluation uses pass@1 for functional correctness and TKQR and ORA for complementary measures of clarification behavior.Together, these metrics capture downstream code correctness and clarification quality.Source: §4.2 · p. 6

4.3 Implementation Details

  • Runs allow at most six interaction rounds; output limits are 1,024 tokens for non-reasoning runs and 8,192 for reasoning runs, with evaluated-model temperature set to zero.The GPT-4o judge uses temperature 1.0 and majority voting across three samples.Source: §4.3 · p. 6

5 RESULTS

Across five research questions, the results show that LLMs struggle to clarify ambiguous requirements, with performance varying by ambiguity type and count. Models often ask too few questions, and unresolved ambiguity can produce runtime errors, wrong answers, or inefficient implementations.

5.1 RQ1: Effectiveness of clarification question asking by LLMs.

  • DeepSeek-V3.2 achieves the highest TKQR (0.30) and ORA (0.50), while pass@1 (Ambig.) trails complete-requirement pass@1 by 7.8 to 19.8 points across models.Clarification quality is low across models on both early key-question coverage and near-optimal interaction rounds.Source: §5.1 · pp. 6–7, Table 3 · p. 7

5.2 RQ2: Performance across ambiguity types

  • Clarification varies by ambiguity type: models do best on explicit local constraints such as Units and Numerical Precision, but struggle with semantics-heavy categories.DeepSeek-V3.2 leads on most types, while Qwen3-235B-A22B performs worst overall and records 0.0% on Ordering & Atomicity, Units, and Numerical Precision.Source: §5.2 · p. 7

5.3 RQ3: Performance across difficulty levels

  • As ambiguity points increase, models rarely resolve every issue: the best D2-H2 rate is 0.08, and D3-H3 is almost zero across models.On single-question tasks, the best D1-H1 rate is 0.30 for Deepseek-V3.2.Source: §5.3 · pp. 7–8, Figure 7 · p. 8

5.4 RQ4: Clarification depth on ClarifyCodeBench

  • Models generally ask fewer clarification questions than required, while Gemini-2.5-Flash asks more questions but often misses key ambiguities.Thinking does not increase clarification depth; Claude-Sonnet-4.5’s thinking version has slightly fewer turns and a more negative Diff than its no-thinking version.Source: Table 4 · p. 8, §5.4 · p. 8

5.5 RQ5: Error Analysis

  • Unresolved ambiguity can cause avoidable runtime errors or wrong answers when models proceed on incorrect interpretations.A removed stack constraint led to an incorrect implementation; a missing palindrome example led to a valid but non-shortest palindrome.Source: §5.5 · pp. 8–9
  • In a time-limit case, underspecified boundary-kick rules made the model more likely to use a conservative mutable-grid simulation instead of 0-1 BFS.The timeout was not caused by ambiguity alone; the overly conservative state formulation was the direct source of inefficiency.Source: §5.5 · p. 9

6 Discussion

The discussion limits the evaluation to model-level clarification under unified prompting and identifies data leakage and judge reliability as threats to validity. It also reports that clarification quality remains limited despite a favorable leakage condition.

6.1 Agentic Workflows

  • The evaluation excludes agent-based methods and therefore reflects base-model clarification, not the end-to-end performance of agents that can use tools, inspect repositories, run tests, and plan.The authors exclude agents to isolate model-level behavior under controlled prompting and preserve comparability across models.Source: §6.1 · pp. 9–10

6.2 Threats to Validity

  • Public LiveCodeBench source problems may have appeared in closed models’ training data, allowing prior exposure to inflate some models’ apparent clarification ability.The authors filtered source problems and prioritized recent ones, but note that the original statements may still have been seen during training.Source: §6.2 · p. 10
  • Even under this potentially favorable leakage condition, current models show limited clarification quality, supporting the conclusion that ambiguity remains a challenge for reliable code generation.The authors state that leakage could overestimate some models’ abilities, yet their main finding still holds under that condition.Source: §6.2 · p. 10
  • Human labels agreed strongly with the LLM judge (Cohen’s kappa = 0.89), though some noise may remain in borderline cases.The human comparison used a randomly sampled subset of instances.Source: §6.2 · p. 10

7 Conclusion

The conclusion reports persistent difficulty in LLM clarification, including inconsistent gains from stronger reasoning and sharp performance drops as ambiguity increases. It points to ambiguity-aware interaction and future elicitation research as important alongside code synthesis.

7 Conclusion

  • Clarification remains difficult: stronger reasoning does not consistently improve it, and performance drops sharply as ambiguity increases.The authors argue that reliable LLM-based code generation requires ambiguity-aware interaction as well as synthesis ability.Source: §7 · p. 10
  • ClarifyCodeBench evaluates how well LLMs clarify ambiguous requirements before generating code.The benchmark uses an interactive evaluation protocol and the clarification-oriented metrics TKQR and ORA.Source: §7 · p. 10
  • The authors hope ClarifyCodeBench will support future research on interactive requirement elicitation for AI-assisted software engineering.Source: §7 · p. 11
Loading 2607.00711v2…