CONTRA: Discovering and Qualifying Behavior-Changing Questions for Selective Clarification in LLM Code Generation

1 INTRODUCTION

• Underspecified requirements let agents silently choose plausible behavior that may mismatch user intent, making later correction increasingly costly.Clarification can expose those assumptions before subsequent development builds on them.Source: §1 · p. 1
• Unnecessary questions can interrupt developers, including questions already answered by requirements or prior interactions and questions unrelated to the task.Existing approaches illustrate the trade-off: some miss necessary questions, while behavioral disagreement can prompt unnecessary ones.Source: §1 · pp. 1–2
• CONTRA discovers candidate questions, filters for relevance and unresolved status, verifies behavioral differences under plausible answers, then selects a question or stops using interaction history.Its qualification is designed to identify key questions while filtering out unnecessary ones.Source: §1 · p. 2
• CONTRA achieves the highest F1 and TKQR across four coding agents, with macro-average F1 of 41.20%, 13.88 percentage points above the best baseline.Under the same LLM and evaluation protocol, it also achieves higher recall and F1 than Claude Code and OpenHands.Source: §1 · p. 2
• The authors implement CONTRA as a Claude Code plugin that reports implementation decisions requiring clarification.The plugin is intended to help developers confirm intended program behavior during everyday development.Source: §1 · p. 2

2 RELATED WORK

• Requirement-refinement methods enrich requirements with answers or feedback, train models to recognize underspecification, or revise specifications without user interaction.Examples include CodeClarQA, TiCoder, ClarifyCoder (Wu et al., 2025), SpecFix (Jia et al., 2025), and Specine.Source: §2 · p. 2
• Selective-clarification research studies whether to ask, estimates utility from simulated intents, or learns policies from simulated turns, task relevance, and answerability.Examples include CLAM (Kuhn et al., 2022), IntentSim, and learning-based approaches (Zhang et al., 2025; Vijayvargiya et al., 2026a).Source: §2 · pp. 2–3
• In code generation, ClarifyGPT uses sampled-program disagreement and Ask-or-Assume uses an agent’s judgment; CONTRA checks each question’s relevance, resolution, and behavioral consequences.It then uses interaction history to select qualified questions that still require user input.Source: §2 · pp. 2–3

3.1 PROBLEM FORMULATION

• Given a requirement and question budget, the agent uses prior question–answer history to ask key questions or stop while avoiding irrelevant or already resolved questions.A clarification question requests additional user input needed to implement intended behavior when the requirement is underspecified.Source: §3.1 · p. 3

3.2 OVERVIEW

• The overview presents three stages: Question Discovery builds candidates, Question Qualification filters them, and Adaptive Question Selection uses history to choose a question or stop.Figure 1 is identified as the overview of CONTRA for selective clarification.Source: §3.2 · p. 3, Figure 1 · p. 3

3.3 QUESTION DISCOVERY

• Question Discovery generates a broad candidate set with two plausible answers per question, combining context-aware and independent generation and merging duplicates or overlaps.It stops after two consecutive rounds add no distinct questions or a predefined round limit, then creates completion pairs differing only in the answer to the question.Source: §3.3 · pp. 3–4

3.4 QUESTION QUALIFICATION

• Question Qualification retains relevant, unresolved questions only when programs conditioned on alternative completions produce different stable outputs on a shared input.It generates three programs per completion and uses at least seven shared valid inputs; a failed check does not establish behavioral equivalence.Source: §3.4 · p. 4

3.5 ADAPTIVE QUESTION SELECTION

• Adaptive Question Selection reassesses which qualified questions still need user input using the interaction history, updating that history after each answer.Clarification ends when the selector stops, no questions remain, or the question budget is reached.Source: §3.5 · p. 4

4.1 EXPERIMENTAL SETUP

• ClarifyCodeBench evaluation used 419 underspecified and 80 fully specified tasks, a five-question cap, and majority matching by three GPT-4o judgments.The compared agents used GPT-5-mini, GPT-5.5, Grok 4.5, and Gemini 3.1 Pro, with the same LLM used across methods in each comparison.Source: §4.1 · p. 5

4.2 MAIN RESULTS

• 41.20% macro-average F1 and 38.32% TKQR exceed the best baselines by 13.88 and 12.33 percentage points; CONTRA leads all agents on F1/TKQR and macro-average precision/recall.Table 2 reports macro-average results across the four agents.Source: §4.2 · pp. 5–6, Table 2 · p. 6

4.3 EFFECT OF DISCOVERY ROUNDS

• Recall rises rapidly and then saturates, recovering most gold questions within about four rounds, while precision falls as later rounds add candidates that identify no new gold questions.Figure 2 evaluates all distinct candidates found up to each discovery round.Source: §4.3 · p. 6, Figure 2 · p. 6

4.4 EFFECT OF QUESTION QUALIFICATION

• F1 rises at every qualification stage across four agents; filtering lifts macro-average TKQR from 19.03% after Question Discovery to 32.52% after Behavioral Verification.Successive filtering improves candidate quality while preserving the relative order of retained questions.Source: §4.4 · pp. 6–7, Figure 3 · p. 7

4.5 EFFECT OF ADAPTIVE QUESTION SELECTION

• Under the five-question interaction, Adaptive Question Selection raises macro-average F1 from 28.86% to 41.20% and TKQR from 32.52% to 38.32%, while recall falls from 50.16% to 38.01%.Macro-average precision rises from 21.57% to 45.42% as the comparison shifts from the full qualified set to the questions actually asked.Source: §4.5 · p. 7, Figure 3 · p. 7

4.6 ABLATION ANALYSIS

• Removing Relevance or Resolution reduced GPT-5-mini precision from 36.20% to 31.27% or 26.46%, respectively; Resolution checks whether requirements already answer questions.Table 3 reports component ablations with GPT-5-mini.Source: §4.6 · p. 7, Table 3 · p. 7
• Removing Behavioral Verification lowers precision from 36.20% to 28.04% and F1 from 35.36% to 32.58%; removing Adaptive Question Selection drops F1 by 8.67 percentage points.Removing Adaptive Question Selection also lowers TKQR from 34.32% to 30.70% despite higher recall; removing Behavioral Verification and Adaptive Question Selection both increase recall, but only the former improves TKQR.Source: §4.6 · pp. 7–8, Table 3 · p. 7

5.1 EFFECT ON DOWNSTREAM CODE GENERATION

• 57.32% pass@1 with Gemini 3.1 Pro and 64.63% with GPT-5.5 are CONTRA’s highest results on underspecified tasks.These exceed Direct Prompting by 1.22 percentage points and ClarifyGPT by 0.61 percentage points, respectively.Source: Table 4 · p. 8, §5.1 · p. 8

5.2 COMPARISON WITH CODING HARNESSES

• CONTRA achieves 19.86% F1 and 18.79% TKQR, exceeding the best harness results by 9.25 and 9.80 percentage points, respectively.Its recall is 20.73%, versus 6.26% for Claude Code and 8.42% for OpenHands, under the same question budget.Source: Table 5 · p. 8, §5.2 · p. 8

5.3 EFFICIENT CLARIFICATION AND PLUGIN DEPLOYMENT

• CONTRA-Flash reduces qualification evaluations by 65.7–96.0%, with F1 changing by at most 1.8 percentage points for three agents.For Gemini 3.1 Pro, it achieves 35.7% F1 and 28.3% TKQR with 65.7% fewer evaluations; its F1 is within 1.9 percentage points of the best baseline.Source: §5.3 · pp. 8–9, Figure 4 · p. 9
• The Claude Code plugin combines the agent’s report with an independent review, prioritizes unresolved choices, and uses distinguishing tests for behavioral verification.Verification runs in the background, and the report explains what is unspecified, what was implemented, and a plausible alternative.Source: §5.3 · p. 9
• On 40 repository tasks, independent review identified 116 of 262 reported decisions (44.3%) that were absent from the agent’s report.With the plugin, reports rose from 4.3% to 59.7% for unspecified information, from 7.1% to 47.8% for alternative interpretations, and from 2.9% to 46.3% for covering all three elements.Source: Table 6 · p. 9, §5.3 · p. 9

6 CONCLUSION

• CONTRA combines question discovery, qualification, and adaptive selection; experiments show consistent F1 and TKQR improvements across four agents and gains over existing coding harnesses.The authors plan to evaluate effects on developer effort and code quality over longer development sessions.Source: §6 · p. 10

AI USE STATEMENT

• The authors used generative AI tools for manuscript work, reference checking, figure design, and an illustrative cartoon, and manually reviewed all 40 repository task labels.They state that they take full responsibility for the final content.Source: AI USE STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

• The paper describes its benchmark, baselines, models, question budget, evaluation protocol, and downstream code evaluation in Sections 3–5.Appendix B covers plugin implementation and repository evaluation, including versions, sampling, manual review, and report-quality assessment.Source: REPRODUCIBILITY STATEMENT · p. 11
• The qualification and adaptive-selection code and developer plugin are released, but benchmark data and experiment records are not included.Source: REPRODUCIBILITY STATEMENT · p. 11

• The figure frames the report as a way to review implementation choices while the requirement and change remain in context.Source: Figure 5 · p. 13
• The demonstration starts with a request to add Brotli decoding, but the completion report does not explain dependency-version interactions, advertised encodings, or whether a new constant is public API.The report says Brotli is included when available and the Quickstart documentation was updated.Source: A ONE REQUEST, FIVE DECISIONS: AN… · p. 13
• Brotli requires urllib3 1.25 or later, while the package minimum is 1.21.1; a version 1.24 user would get older encodings despite the documentation.The request leaves open whether to raise the dependency bound, warn, or accept fallback behavior.Source: A ONE REQUEST, FIVE DECISIONS: AN… · p. 13
• The plugin report presents five decisions, including scope additions, deferred work, and forwarding all urllib3-detected encodings, including zstd, rather than only br.The entries also flag a public name, documentation update, dependency mismatch, and missing regression coverage.Source: A ONE REQUEST, FIVE DECISIONS: AN… · p. 13
• A distinguishing test shows a behavioral difference between the implemented encoding policy and an alternative, giving the developer executable evidence to review.The choice is whether to enable Brotli alone or delegate the advertised encoding set to the dependency.Source: A ONE REQUEST, FIVE DECISIONS: AN… · p. 13

• A hook snapshots the working tree when a request arrives and compares it with the final tree to isolate changes made during that interaction.This avoids including unrelated changes already present before the request.Source: §B.1 · p. 14
• Two parallel processes extract decisions from the agent’s report and independently review the requirement, code changes, and repository context; their results are merged by text overlap.Rule-based prioritization uses decision category, affected scope, trigger conditions, reversibility, and verification outcomes.Source: §B.1 · p. 14
• For an unresolved choice, the plugin tests the current implementation and a patched alternative in isolated repository copies to check for behavioral differences.A test passing on the current version and failing on the executable alternative is evidence of a difference; errors are distinguished from such evidence.Source: §B.1 · p. 14

B.2 REPOSITORY EVALUATION

• The evaluation samples 40 single-function commits from six Python repositories, with 20 labeled underspecified and 20 fully specified, and runs each task once with and without the plugin.Each task restores the repository to its pre-commit state; the commit message is the requirement, and Claude Code modifies the target function.Source: §B.2 · p. 14
• Task labels are manually reviewed and reflect whether the commit message and pre-change code determine observable behavior without relying on implementation conventions.Source: §B.2 · p. 14
• Final-reply report quality is measured by whether entries state the unspecified information, implemented choice, and plausible alternative, individually and together.The comparison covers 70 entries without the plugin and 67 with it.Source: §B.2 · p. 14
• Of 262 decisions, 116 appeared only in independent review, 89 only in the agent-derived list, and 57 in both.The repository evaluation measures both decision discovery and report completeness.Source: §B.2 · p. 14

C PLUGIN INTERFACE AND OPERATIONAL SCOPE

• The demonstration keeps the request, code change, and completion report visible above five decisions, with three categories and a test-artifact path.The verification indicator applies to the fifth entry.Source: Figure 6 · p. 15, C PLUGIN INTERFACE AND OPERATIONAL SCOPE · p. 15
• Entries expose implemented choices or deferred work and available alternatives; developers can accept, request a revision, or defer a choice.A separate indicator marks entries supported by an observed behavioral difference.Source: C PLUGIN INTERFACE AND OPERATIONAL SCOPE · p. 15
• The single-run demonstration reports 40 s ($0.47) for coding, 90 s ($0.17) for reporting, and 15 s ($0.06) for background verification; total cost is $0.70.The report appears about 130 s after the request.Source: C PLUGIN INTERFACE AND OPERATIONAL SCOPE · p. 15
• Executable evidence in this build uses Python and pytest, while other supported source languages receive decision reports.Verification supports behavior exercised by each distinguishing test.Source: C PLUGIN INTERFACE AND OPERATIONAL SCOPE · p. 15