PaperlayerView exact revision on arXiv

Source-linked AI summary

Asking What Matters: Reward-Driven Clarification for Software Engineering Tasks

Sanidhya Vijayvargiya, Vijay Viswanathan, Graham Neubig

arXiv:2604.14624v1cs.SEcs.AI

TL;DR

Underspecified software-engineering tasks leave unclear which information agents should request and whether users can answer. The paper measures information impact and answerability, then uses both to train CLARITI with multi-stage rewards. CLARITI matches GPT-5’s resolution rate while generating 41% fewer questions.

  • Problem

    Prior work provides limited evidence about which missing information matters most, which clarification questions users can answer, and how these factors affect task success.

  • Method

    The study combines Shapley attribution, distributional analysis, and multi-stage reinforcement-learning rewards for an 8B-parameter clarification module.

  • Results

    CLARITI matches GPT-5’s performance on underspecified issues while generating 41% fewer questions.

  • Takeaways & Limitations

    Empirically grounding rewards in task relevance and user answerability improves clarification efficiency within software-engineering tasks.

  • Takeaways & Limitations

    The study uses a single-turn design, focuses empirical grounding on software engineering, and evaluates with an LLM judge that may diverge from human judgment.

Abstract

from arXiv · show

Humans often specify tasks incompletely, so assistants must know when and how to ask clarifying questions. However, effective clarification remains challenging in software engineering tasks as not all missing information is equally valuable, and questions must target information users can realistically provide. We study clarification in real software engineering tasks by quantifying which types of information most affect task success and which questions elicit useful responses from simulated users. Using Shapley attribution and distributional comparisons, we identify two key properties of effective clarification: task relevance (which information predicts success) and user answerability (what users can realistically provide). We operationalize these properties as multi-stage reinforcement learning rewards to train CLARITI, an 8B-parameter clarification module, that matches GPT-5's resolution rate on underspecified issues while generating 41% fewer questions. Our results suggest that grounding reward design in empirical analysis of information impact and user answerability improves clarification efficiency.

1. Introduction

Underspecified software-engineering tasks can cause agent failure, yet prior work offers limited guidance on which missing information to request and whether users can answer. This study evaluates task relevance and user answerability, then trains CLARITI with empirically grounded rewards.

  • 41% fewer questions (3.0 vs 5.1) let CLARITI match GPT-5-level performance at 36.80%.The model prioritizes task relevance and user answerability through multi-stage rewards.
  • Prior clarification research largely addresses isolated linguistic ambiguity, leaving broader underspecification and question prioritization insufficiently understood.The gap includes both identifying valuable missing information and determining what users can realistically provide.
  • The study examines software-engineering issues in a controlled single-turn setting, separating clarification generation from task execution.This isolates clarification quality while testing whether requested information improves downstream task success.
  • RQ1 measures which information categories predict success, while RQ2 analyzes which clarification questions are answerable and how question quantity affects burden.Error information has the strongest association with success; answerable questions ground in observable behavior and avoid unknowable internal state.
  • CLARITI operationalizes task relevance and answerability alongside non-redundancy and diversity in an 8B-parameter clarification module.The framework connects empirical impact analysis, answerability characterization, and reward-driven training.

2. Information Categorization

The paper defines information needs as missing issue information required for a correct patch without unverifiable assumptions. It builds a six-category taxonomy from highly underspecified SWE-Bench issues and makes later annotation scalable through LLM judges.

  • 2. Information Categorization: An information need is absent issue information required to produce a correct patch without relying on unverifiable assumptions.
  • 2.1. Categories of Information Needs: The taxonomy derives from 112 highly underspecified SWE-Bench Verified issues with expert justifications describing concrete gaps such as errors, expected outputs, and reproduction steps.The source issues had underspecification scores of at least 2 on a 0–3 scale.
  • 2.1. Categories of Information Needs: Two authors iteratively developed the codebook, independently coding an initial sample and confirming category completeness on 50 held-out issues.Subsequent per-instance annotation can be automated with LLM judges, while expert involvement remains limited to codebook development.
  • 2.1. Categories of Information Needs: The resulting codebook contains six distinct information-need categories, whose frequencies range from 65% to 3% of issues.These categories represent different missing-information types agents may need to obtain through clarification.

3. Experimental Setup

The experiments use controlled underspecified variants of SWE-Bench Verified issues and a fixed coding-agent setup to measure clarification effects objectively. The protocol separates information-impact analysis from clarification experiments while evaluating success by repository tests.

  • 3. Experimental Setup: The shared infrastructure uses controlled datasets, a fixed agent framework, and a binary test-passing measure for downstream task success.
  • 3.1. Categorization-Grounded Evaluation Dataset: Fifty sampled rewrites were manually checked for intended omissions, preserved non-target information, and plausible GitHub issue descriptions.
  • 3.1. Categorization-Grounded Evaluation Dataset: 1,500 underspecified variants come from 500 issues with three rewrites each, while RQ1 samples 700 instances and RQ2–RQ3 use 500 issues.Variants omit selected annotated categories while preserving the remaining issue information.
  • 3.2. Agent Framework and Evaluation Protocol: Agents use OpenHands with Seed OSS 36B under fixed configuration, including a 30-iteration maximum and temperature 0.3.Keeping the backbone and configuration fixed isolates clarification-strategy effects.
  • 3.2. Agent Framework and Evaluation Protocol: For clarification experiments, a GPT-5 simulated user answers questions using the fully specified issue, and the answers are supplied before code generation.

4. RQ1: Impact of Information Categories on Task Success

RQ1 quantifies how missing information relates to software-engineering task success using SHAP analysis over controlled issue variants. Concrete diagnostic and implementation information shows stronger associations than some more frequent or abstract categories, although category ordering is uncertain.

  • 43.8% to 23.7% success: converting fully specified issues to underspecified variants substantially reduces agent performance.This comparison covers the 700 evaluated instances.
  • Error Information has the highest mean SHAP value at 0.183, followed by Implementation Details at 0.0982.Expected Behavior is 0.0572 and External References is 0.0355; overlapping bootstrap intervals make the ordering indicative rather than statistically significant.
  • The impact hierarchy differs from missingness frequency: Expected Behavior occurs in 65% of issues, whereas Error Information is absent in 33% but has the highest mean SHAP value.Concrete failure signals may help agents localize debugging search through information such as stack traces or failure points.
  • Version/Environment is missing in only 3% of issues yet has comparatively high predictive contribution when absent.Implementation Details shows a similar pattern, with relatively high contribution despite being missing in 37% of issues.
  • Categories providing diagnostic signals or implementation guidance tend to contribute more than categories describing expected outcomes or external context.This impact-weighted hierarchy motivates the reward signals used for clarification training.

5. RQ2: User-Answerable Clarification Questions

RQ2 examines how clarification questions can target missing information that users can realistically provide, and how answerable-question composition affects task success. Across software-engineering issues, the analysis distinguishes answerable from unanswerable questions and shows that asking more questions does not reliably improve outcomes.

  • 5.1. Experimental Setup: The study evaluates 500 underspecified issues per model by judging whether each question requests information missing from the issue but present in its fully specified version.Questions are classified as answerable, unanswerable, or redundant; the main analyses focus on answerable versus unanswerable questions.
  • 5.2. Distributional Analysis: What Makes Questions Answerable?: Distributional analysis compares answerable and unanswerable questions across 60 significantly different characteristics and groups them into four strategic themes.The analysis uses Vargha–Delaney effect sizes across GPT-5 and GPT-5 nano outputs, with all reported discoveries satisfying p < 0.05.
  • 5.2. Distributional Analysis: What Makes Questions Answerable?: Effective clarification requires both targeting high-value information and phrasing questions so users can supply the answer.An important target can remain unanswerable when it requests internal information, whereas a lower-impact target may still be answerable if grounded in observable details.
  • 5.3. Impact of Answerability on Performance: Question success plateaus as clarification sets grow, while the proportion of answerable questions declines.Figure 3 summarizes the relationship between question count, answerability, and downstream task success.
  • 5.3. Impact of Answerability on Performance: Higher answerability proportions correlate with stronger task performance and lower user burden, motivating rewards that jointly target information impact and user accessibility.The proposed training strategy combines RQ1’s information-impact hierarchy with answerability signals rather than rewarding question quantity alone.

6. RQ3: Do Empirically-Grounded Objectives Improve Clarification?

RQ3 trains an 8B clarification module with empirically grounded rewards for task relevance and answerability, alongside non-redundancy and diversity constraints. The resulting model matches GPT-5-level task success with fewer questions, while ablations and qualitative analyses show why staged reward design matters.

  • 6.1. Training Data and Model: The study trains Qwen3 8B using 3,000 supervised pairs and 1,000 reinforcement-learning instances derived from underspecified SWE-Gym issues.Variants are created by removing information, identifying gaps, converting gaps into questions, and filtering ungrounded questions.
  • 6.2. Four-Stage Reward Pipeline: The reward pipeline progressively filters non-redundancy and diversity before scoring answerability and task relevance, preventing simpler objectives from being gamed in isolation.Candidates failing a stage receive zero reward and are excluded from later stages; the four stages are equally weighted in the final reward.
  • 6.2. Four-Stage Reward Pipeline: Task relevance weighting corrects frequency-importance misalignment by incentivizing high-impact information, such as Error Information, rather than merely common missing categories.Expected Behavior appears in 65% of underspecified issues but has lower predictive contribution than less frequent categories such as Error Information.
  • 6.3. Downstream Evaluation: The evaluation separates training and test distributions while comparing no clarification, GPT-5 Nano, GPT-5, the trained model, and a fully specified upper bound.Training uses SWE-Gym Raw repositories and DeepSeek-V3, whereas evaluation uses separate datasets and GPT-5 judges.
  • 6.4. Results: 36.8% task success with 3.0 questions on average—41% fewer than GPT-5’s 5.1—recovers 88% of fully specified performance.The evaluation uses 250 underspecified issues, simulated user responses, and fixed downstream components.
  • 6.4. Results: The trained model allocates 26.4% of questions to Error Information versus 10.2% for GPT-5, while reducing Implementation Details and Reproduction Steps.Its category distribution more closely follows the empirical impact hierarchy, whereas GPT models often cluster questions around one or two categories.
  • 6.4. Results: Training progresses from non-redundancy to diversity, answerability, and task relevance, suggesting staged rejection filtering acts as an implicit curriculum for harder properties.Answerability improves later because it requires modeling information outside the issue description, unlike the context-verifiable structural constraints.
  • 6.5. Qualitative Analysis: Qualitative examples show that the model prioritizes observable, issue-specific evidence, avoids burdensome or inaccessible requests, and sometimes abstains when all candidate questions are low quality.Its questions target visible symptoms, concrete artifacts, and relevant components rather than reusable templates or internal state.

7. Related Work

Prior clarification research emphasizes ambiguity detection and linguistic disambiguation, while this work studies which missing information most improves task completion in complex underspecified settings. It isolates single-turn clarification and grounds training objectives in information impact and user answerability.

  • 7.1. Clarification and Ambiguity Resolution: Existing systems address ambiguity detection, clarification generation, and linguistic disambiguation across question answering, search, and dialogue tasks.
  • 7.1. Clarification and Ambiguity Resolution: Most prior approaches optimize conversational clarity around limited linguistic ambiguities, whereas this work selects multiple missing information types by downstream utility and user answerability.
  • 7.2. Clarification in Agentic Settings: Agentic clarification must decide both when to query and what information blocks progress, with prior work using uncertainty estimation, supervised training, user modeling, and burden-aware policies.
  • 7.2. Clarification in Agentic Settings: This study isolates single-turn clarification, quantifies how information types affect task success, and uses those findings to design training objectives rather than relying on generic quality heuristics.

8. Conclusions

The paper argues that effective clarification balances task relevance with user answerability, then operationalizes both properties in a staged reward design. CLARITI matches GPT-5 while asking fewer questions, but the study remains bounded by single-turn, domain-specific, judge-based evaluation.

  • CLARITI, an 8B-parameter clarification module, matches GPT-5 while generating 41% fewer questions.
  • Effective clarification depends on task relevance and user answerability, with Shapley analysis identifying information-impact hierarchies and distributional analysis identifying answerable-question characteristics.
  • The study’s single-turn design omits multi-turn interaction, its empirical grounding is specific to software engineering, and LLM-judge evaluation may diverge from human judgment.
  • The proposed methodology quantifies information impact, identifies answerable-question characteristics, and operationalizes them through multi-stage rewards for broader agent settings.

Impact Statement

The work aims to make clarification more efficient by reducing user burden while improving performance on underspecified software-engineering tasks and similar technical domains. It presents general-purpose methods without introducing risks beyond those associated with large language model deployment.

  • More efficient clarification could reduce user burden and help general models improve performance in software engineering and similar technical domains.
  • The techniques are presented as general-purpose methods that introduce no novel risks beyond those already documented for large language model deployment.
  • The paper frames improved question generation as a way to enable more efficient human-AI interaction.

A.1. Information Needs Annotation

The paper uses annotated SWE-bench examples to document underspecified instances and the categories of information they are missing.

  • Table 7 presents underspecified SWE-bench instances with annotator notes and identified missing-information categories.
  • The examples provide annotated cases for examining what information underspecified software-engineering tasks omit.
  • The annotation focuses on missing-information categories within underspecified SWE-bench instances.

A.2. Prompts for Controlled Underspecification Generation

This appendix describes the prompts, annotation procedures, and evaluation methods used to generate controlled underspecification, analyze clarification-question properties, and validate the Shapley analysis. It also presents a four-stage reward pipeline and GRPO training for producing non-redundant, diverse, answerable, and task-relevant clarification questions.

  • A.2. Prompts for Controlled Underspecification Generation: Controlled underspecification first identifies present information categories and examples, then instructs a rewriting model to remove selected categories while preserving a realistic issue.The taxonomy covers error information, reproduction steps, implementation details, version/environment information, external references, and expected behavior.
  • A.2. Prompts for Controlled Underspecification Generation: The rewriting prompt requires complete removal of targeted details, natural developer-style language, no disclosure of omissions, and output only within <rewrite></rewrite> tags.It specifies category-specific deletion rules for reproduction, error, implementation, environment, external-reference, and expected-behavior information.
  • A.3. Annotation Process Example: The annotation example transforms an Astropy issue by identifying present categories, randomly hiding version/environment information, external references, and expected behavior, and producing a revised underspecified issue.The rewritten issue retains the general problem while replacing concrete implementation context with questions about use cases, regressions, and transition planning.
  • A.4. Shapley analysis: >0.80 accuracy was achieved by all three Shapley-analysis models, exceeding the 0.52 majority-class baseline.The ensemble used L2-regularized logistic regression, a 100-tree random forest, and gradient boosting with 100 estimators, averaging mean absolute SHAP values across models.
  • A.5. Clarification Questions: Clarification questions are generated as a limited numbered list targeting information needed to solve an underspecified software-engineering issue.The prompt frames the model as an expert developer reviewing the issue and requires questions based on the underspecified problem statement.
  • A.6. Distributional Data Analysis Findings: The distributional analyses compare answerable questions targeting information present in the original issue with non-answerable questions requesting unavailable information across pooled, GPT-5, and GPT-Nano settings.Results are organized into evidence grounding, precision targeting, scope minimization, and user actionability strategies, with significance annotations in Tables 8–10.
  • A.6. Distributional Data Analysis Findings: 60 top discoveries—20 each from cross-model, GPT-5, and GPT-Nano analyses—characterized answerable versus non-answerable question formulation strategies.The discoveries concern how to ask, rather than which information categories to request.
  • A.7. Intrinsic Evaluations: Answerability evaluation separately asks whether each clarification question can be answered from the original issue and from the underspecified issue.The evaluator returns a true/false judgment for each source and question in valid JSON.
  • A.7. Intrinsic Evaluations: The evaluation prompts classify questions as answerable from the original only, non-answerable, or answerable from both, discarding questions answerable from both sources.Generic requests such as “provide more details” are treated as non-answerable rather than useful clarification.
  • A.7. Intrinsic Evaluations: Task-relevance prompts judge whether a question seeks information needed to understand, reproduce, or fix the issue and reject generic, unrelated, or tangential questions.The relevance criteria require that an answer help a developer make progress on the task.
  • A.8.1. FOUR-STAGE REWARD PIPELINE: The four-stage reward pipeline sequentially filters redundancy and diversity before scoring answerability and task relevance, with failed candidates receiving zero reward in later stages.The final reward equally weights the four stage rewards, preventing policy gaming of individual metrics.
  • A.8.1. FOUR-STAGE REWARD PIPELINE: Redundancy is assessed by answering questions from the underspecified issue and filtering generations where r_redundancy < 0.5, while diversity penalizes generic or similar questions.The pipeline uses judge-based evaluations to favor issue-specific questions referencing concrete entities such as files, functions, and error types.
  • A.8.1. FOUR-STAGE REWARD PIPELINE: Answerability is estimated from the original fully specified issue, whereas task relevance weights information categories according to their relative mean SHAP values, with error information weighted highest.Redundancy filtering prevents questions already answered by the full issue from receiving artificially high answerability scores.
  • A.8.2. MODEL ARCHITECTURE AND INFRASTRUCTURE: The model uses GRPO starting from Qwen3-8B with an actor, frozen reference, and Qwen3-32B reward model distributed across three training components.Training uses AdamW with learning rate 5 × 10^-6, batch size 4, gradient accumulation 4, N = 8 samples per prompt, and KL coefficient β = 0.05.
  • A.8.3. TRAINING REWARD CURVES: Mean reward rises from approximately 0.2 to 0.6 over 200 training steps, showing that GRPO learns higher-quality clarification questions.Figure 5 reports steadily increasing rewards for non-redundant, novel, answerable, and useful questions.
  • A.10. Qualitative Examples: The appendix includes qualitative comparisons of clarification questions across models and distributions across information categories, using labels for answerability, unanswerability, and redundancy.The reported categories include error information, reproduction, and expected behavior.
Loading 2604.14624v1…