PaperlayerView exact revision on arXiv

Source-linked AI summary

Don't Guess, Just Ask: Evaluating Interactive Requirement Clarification for Repository-Level Software Development

Mingyu Chen, Zhiqing Zhong, Manyi Wang, Linxi Liang, Rouyu Chen, Pinjia He

arXiv:2610.13022v1cs.SEcs.AI

TL;DR

Repository-level development benchmarks often omit interactive clarification, leaving unclear how LLMs transform vague intent into requirements and how that affects code generation. ReqClarifyBench evaluates this process across repository-level tasks using LLM-based user simulators with diverse personas. The study finds that clarification effectiveness does not necessarily translate into document quality, and performance varies across personas.

  • Problem

    Existing repository-level benchmarks omit multi-turn clarification or require inference from vague requests, while prior clarification studies mainly address function-level tasks.

    Source: §1 · p. 2, §2.2 · p. 5

  • Method

    ReqClarifyBench pairs vague intents with human-validated requirements and code-evaluation artifacts, using LLM-based user simulators guided by diverse personas.

    Source: §1 · p. 2, §4.2 · p. 8

  • Results

    Clarification effectiveness does not necessarily lead to higher-quality requirement documents, and model performance varies substantially across user interaction styles.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Reliable requirement clarification depends on eliciting key information, integrating multi-turn requirements, and adapting strategies to different user personas.

    Source: §1 · p. 3

    LLM-based user simulators may not fully capture the diversity of how real developers express requirements, provide information, and revise needs.

    Source: §6.2 · p. 18

Abstract

from arXiv · show

As LLMs become increasingly capable of implementing software, a critical bottleneck emerges: whether they truly understand what users want. Users often begin with vague intent, leaving important requirements to be clarified through interaction. In repository-level development, misunderstandings can propagate and cause costly rework. However, existing benchmarks typically evaluate implementation from complete specifications or underspecified requests, overlooking pre-coding requirement clarification. Prior clarification studies mainly focus on function-level tasks. It therefore remains unclear how well LLMs can transform vague intent into clear requirements through interaction and how this capability affects subsequent code generation. We introduce ReqClarifyBench, an interactive requirement clarification benchmark for repository-level software development. It contains 341 tasks spanning game, miniapp, and web development, each with a vague user intent, a human-validated requirement document, and code evaluation artifacts. We employ LLM-based user simulators with diverse personas derived from real-world conversations. We evaluate six LLMs in terms of clarification effectiveness and generated requirement document quality. Results show that stronger clarification effectiveness does not necessarily translate into higher document quality, and performance varies substantially across user personas. We also find that coding-agent harnesses do not further improve requirement document quality compared with their base LLMs. These findings offer insights for building LLM-based end-to-end software development systems.

1 Introduction

The paper introduces ReqClarifyBench to evaluate how LLMs clarify vague repository-level software requests before implementation. Its study finds that clarification effectiveness does not necessarily yield higher-quality requirement documents, and that user personas substantially affect performance.

1 Introduction

  • Vague requests can prompt incorrect assumptions that propagate through design, implementation, and testing, increasing human oversight and rework.Models may spend time and tokens acting on misunderstandings even as implementation benchmark scores rise.Source: §1 · p. 2
  • Existing repository-level benchmarks generally use complete specifications or vague requests without multi-turn clarification, while prior clarification studies focus mainly on function-level tasks.Existing simulated users may also follow fixed interaction patterns rather than capture diverse real-user behavior.Source: §1 · p. 2
  • ReqClarifyBench contains 341 repository-level tasks, human-validated requirement documents, code-evaluation artifacts, and three metrics for clarification and document quality.Its tasks span game, miniapp, and web development, and its user simulators follow three personas derived from real-world developer-agent conversations.Source: §1 · pp. 2–3
  • Clarification effectiveness does not necessarily produce higher-quality requirement documents, and user interaction styles substantially affect model performance.The study also reports distinct clarification dynamics, consistent quality gains from clarification, benefits to subsequent implementation, and no additional document-quality gains from coding-agent harnesses.Source: §1 · p. 3

2 Background & Related Work

Prior repository-level development approaches generally infer missing requirements rather than clarify them with users. Existing benchmarks and clarification studies leave gaps in evaluating interactive clarification for complex software tasks with diverse user behaviors.

2.1 LLM-based Repository-level Code Generation

  • Repository-level development approaches such as MetaGPT, ChatDev, and RPG generally infer missing information instead of clarifying it interactively with users.This can result in software that deviates from actual user intent.Source: §2.1 · p. 4

2.2 Code Generation Benchmarks

  • Code-generation benchmarks span function-level tasks, repository fixes, and end-to-end applications, but typically evaluate implementation from fixed requirements.Complete specifications omit clarification, while vague requests require inference without user feedback, obscuring whether failures reflect coding limitations or misunderstandings.Source: §2.2 · pp. 4–5

2.3 Interactive Requirement Clarification

  • Prior interactive clarification work mainly targets function-level problems, and ReqElicitGym uses a single oracle-style simulated user and does not study subsequent code generation.ReqClarifyBench instead targets repository-level development with LLM-based user simulators using diverse personas.Source: §2.3 · p. 5

3 ReqClarifyBench

ReqClarifyBench adapts repository-level source benchmarks into tasks pairing vague user intents with detailed requirements and code-evaluation artifacts. Its construction combines requirement reconstruction, intent abstraction, and human validation; the resulting 341 tasks have a substantial gap between intent and requirement-document length.

3.1 Benchmark Construction

  • ReqClarifyBench pairs a high-level vague intent with a complete requirement document and code-evaluation artifacts through a semi-automated, human-in-the-loop pipeline.The pipeline is designed to adapt to new datasets.Source: §3.1 · p. 5
  • Source benchmarks are curated for real-world requirements, repository-level complexity, and behavioral evaluation rather than rigid unit-test interfaces.Reusing existing benchmarks also enables code-evaluation protocols to measure clarification's impact on implementation.Source: §3.1 · pp. 5–6
  • The benchmark selects GameCraft-Bench, MiniAppBench, and WebGenBench for real-world repository tasks with behavioral evaluation protocols.Their protocols assess gameplay through video and frame judgments, miniapps through static analysis and browser evidence, and websites through UI-agent tests and visual appearance.Source: §3.1 · p. 6
  • LLMs reconstruct structured requirements from benchmark artifacts, then abstract them into high-level intents that preserve core software objectives while removing implementation details.Requirement–artifact traceability records evidence for reconstructed requirements and helps prevent unsupported details.Source: §3.1 · pp. 6–7
  • Two authors validate requirements for correctness, unambiguity, completeness, and consistency, then check that abstracted intents preserve core objectives.Requirements failing validation are regenerated and rechecked; only samples passing both validation stages are included.Source: §3.1 · p. 7

3.2 Benchmark Statistics.

  • 341 tasks comprise 100 MiniAppBench tasks, 140 GameCraft-Bench tasks, and 101 WebGen-Bench tasks, with requirement documents averaging 2,380 tokens and intents averaging 13 tokens.The length gap holds across all three sources, leaving substantial room for clarification.Source: §3.2 · p. 7

4 Experiment Design

The experiment evaluates repository-level clarification through simulated multi-turn interactions, measuring both question effectiveness and final requirement-document quality. Its design varies the evaluated LLM while controlling task inputs, personas, and interaction protocol.

4.1 Interactive Requirement Clarification

  • Each task starts with a high-level intent; the LLM asks questions, decides when to stop, and produces a detailed requirement document, with the interaction recorded for evaluation.The benchmark pairs each intent with a human-validated requirement document and code evaluation artifacts.Source: §4.1 · pp. 7–8

4.2 LLM-based User Simulator

  • The simulator answers from human-validated requirements and adopts Expert Nitpicker, Vague Requester, or Mind Changer behaviors derived from real interaction data.In 180 sampled trajectories, blind persona classification reached 95.8% accuracy, with Cohen’s κ of 0.892.Source: §4.2 · p. 8

4.3 Evaluation Metrics

  • Evaluation separates the effectiveness of the clarification process from the quality of its final requirement document.The first concerns how effectively questions obtain information; the second concerns the resulting document.Source: §4.3 · p. 9

4.3.1 Clarification Effectiveness.

  • EQR measures the share of questions clarifying at least one ground-truth atomic requirement, while AvgClarified measures the average number clarified per question.Both metrics use atomic requirements clarified in question-response pairs.Source: §4.3.1 · p. 9

4.3.2 Requirement Document Quality.

  • Document quality uses BERTScore F1 for semantic similarity and ReqAlign, an LLM-judge average of atomic-requirement coverage, omission, or violation scores.ReqAlign ranges from −1 to 1, with higher scores indicating better alignment.Source: §4.3.2 · pp. 9–10

4.4 Studied LLMs

  • The study compares six LLMs: three proprietary models and three open-weight models, while holding the other experimental settings fixed.The proprietary models are GPT-5.6-Sol, Claude Opus 5, and Gemini-3.1-Pro; the open-weight models are GLM-5.3, DeepSeek-V4-Pro, and Qwen3-235B-A22B.Source: §4.4 · p. 10

4.5 Implementation Details

  • Evaluated models use temperature 0 and sessions cap at 20 rounds; DeepSeek-V4.1-Flash is selected as judge for competitive accuracy and lower inference cost.The simulator uses GPT-5.6-Sol at temperature 0.7, and configurations are to be released with the source code.Source: §4.5 · pp. 10–11

5 Evaluation

Across six LLMs and three personas, the evaluation finds distinct clarification and document-generation capabilities, substantial persona effects, and different interaction dynamics. Clarification improves requirement quality and code generation, while agent harnesses improve clarification but not final document quality.

5.1 RQ1: Overall Performance

  • 0.727 ReqAlign for GPT-5.6-Sol exceeds Claude Opus 5 by 26.0%, despite GPT’s lowest EQR (0.196); Gemini-3.1-Pro leads EQR (0.334) but scores 0.357 ReqAlign.The findings characterize effective clarification and high-quality requirement-document generation as related but distinct capabilities.Source: §5.1 · pp. 11–12
  • 0.609 average ReqAlign under Expert Nitpicker falls to 0.433 with Vague Requester and 0.439 with Mind Changer, while persona robustness differs by model.GPT-5.6-Sol retains about 81% and 78% of its Expert Nitpicker ReqAlign under Vague Requester and Mind Changer; Qwen3-235B-A22B retains about 46% and 53%.Source: §5.1 · p. 12

5.2.1 Stopping Behavior.

  • Expert Nitpicker interactions stop earliest and Vague Requester interactions latest, with the size of this difference varying substantially by model.GPT-5.6-Sol typically stops after 2–3 rounds with Expert Nitpickers, around 6 with Mind Changers, and 14–15 with Vague Requesters; Gemini-3.1-Pro stays near 2–3 rounds across personas.Source: Figure 5 · p. 13, §5.2.1 · pp. 13–14

5.2.2 Clarification Dynamics.

  • Expert Nitpickers yield early, information-rich rounds; Mind Changers reveal requirements later; Vague Requesters sustain low-yield exchanges, with models differing in per-round gains and stopping.For Vague Requesters, Gemini-3.1-Pro and Qwen3-235B-A22B have relatively high per-round gains, while GPT-5.6-Sol continues low-gain clarification longer.Source: Figure 6 · p. 14, §5.2.2 · pp. 14–15

5.3 RQ3: Impact of Clarification on Requirement Document Quality

  • 133.1% relative improvement in average ReqAlign (0.212 to 0.494) exceeds the 7.2% BERTScore F1 gain (0.664 to 0.712) after clarification.All models improve document quality, and GPT-5.6-Sol has the highest ReqAlign both with and without clarification.Source: Figure 7 · p. 15, §5.3 · p. 15

5.4 RQ4: Effect of Coding-Agent Harness

  • Higher EQR and AvgClarified for Claude Code and Codex accompany lower final-document quality than base models; Expert Nitpicker ReqAlign falls 15.2% for Claude and 24.0% for GPT.The same quality trend occurs with Vague Requesters and Mind Changers.Source: Figure 8 · p. 16, §5.4 · p. 16

5.5 RQ5: Impact on Code Generation

  • Across GameCraft-Bench, MiniAppBench, and WebGen-Bench, scores rise from intent to direct documents (0.012 to 0.273, 0.451 to 0.543, 0.271 to 0.384), then improve further after clarification.The results show gains from requirement documentation even without added user feedback, with further gains from interactive clarification.Source: Table 5 · p. 17, §5.5 · p. 17

6 Discussion

The discussion presents clarification and requirement documentation as explicit capabilities and suggests separating requirement analysis from coding. It also identifies threats involving user simulation, experimental procedure, and benchmark ground truth.

6.1 Practical Implications

  • Interactive clarification can elicit additional requirement information that improves subsequent code-generation performance.The authors argue that clarification should be treated as an explicit capability rather than leaving developers to fully specify requirements upfront.Source: §6.1 · p. 17
  • Explicit requirement documents consistently outperform direct code generation from initial intent, even without additional user feedback.The document serves as an intermediate representation between user intent and code implementation.Source: §6.1 · p. 18
  • Coding-agent harnesses ask more effective clarification questions but produce lower-quality requirement documents than their corresponding base models.The authors suggest using base models for requirement analysis and document generation, then coding-agent harnesses for implementation.Source: §6.1 · p. 18

6.2 Threats to Validity

  • LLM-based user simulators may not fully capture real developers’ diversity in expressing requirements, providing information, and revising needs.Three personas derived from SWE-chat and human validation are used to mitigate this threat.Source: §6.2 · p. 18
  • Errors in interaction protocols, prompts, configurations, or execution could affect observed clarification behavior and code-generation evaluation.The authors use consistent protocols, standardized prompts, fixed configurations where possible, and open-source materials to reduce this risk and enable external validation.Source: §6.2 · p. 18
  • Ground-truth requirement quality remains a construct-validity threat, despite manual checks of correctness, unambiguity, completeness, and consistency.Two authors also check requirement–artifact traceability and intent abstraction validity; only samples passing both human-validation stages are included.Source: §6.2 · p. 18

7 Conclusion

The conclusion presents ReqClarifyBench as a repository-level benchmark for interactive clarification and reports that clarification effectiveness may not yield high-quality documents. It also emphasizes persona-dependent performance and the lack of document-quality gains from coding-agent harnesses.

7 Conclusion

  • ReqClarifyBench evaluates how LLMs transform vague intents into clear, complete requirement documents through multi-turn interactions.The paper describes it as the first interactive requirement-clarification benchmark for repository-level software development and uses three representative user personas.Source: §7 · p. 19
  • Clarification effectiveness does not necessarily yield high-quality requirement documents; performance varies across interaction styles, and coding-agent harnesses do not improve document quality.Source: §7 · p. 19
  • The authors argue that development-agent autonomy should include proactively understanding user intent and aligning development goals.Source: §7 · p. 19

Related papers

Loading 2610.13022v1…