PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-Clar: Can Coding Agents Clarify Incomplete Client Requirements and Deliver Greenfield Software?

Jialei Li, Dingxian Wang, Xiaoyan Zhao, Yimeng Bai, Xuanyi Zhu, Yang Zhang, Chao Yin, Wentao Shi, Ganesh Prabhu, Karthik Gomadam Rajagopal, Brett Levert, Darvin Yi, Andrew Rabinovich, Zhao Chen

arXiv:2610.31854v1cs.SEcs.AIcs.CL

TL;DR

Greenfield freelance projects start from incomplete requests, and ongoing feedback may be hard to sustain while complex requirements can remain misunderstood. SWE-Clar evaluates clarification before coding through separate requirement-question and implementation runs across real Upwork commissions. GPT-6 Astra, the highest-scoring evaluated model on implementation, passes 67% of tasks.

  • Problem

    Greenfield freelance commissions often omit critical requirements, while prolonged development makes continuous client feedback difficult and answers do not ensure correct understanding.

    Source: §1 · p. 2

  • Method

    SWE-Clar evaluates 100 real Upwork greenfield tasks by pairing pre-implementation client interaction with isolated acceptance-test and reference-answer assessments.

    Source: §1 · p. 2

  • Results

    GPT-6 Astra’s task pass rate is 67%, the highest observed implementation score among the nine evaluated frontier models.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Clarification improves performance but recovers only part of the gap to complete requirements.

    Source: §5 · p. 10

    Future work can extend SWE-Clar to changing requirements and different patterns of client availability.

    Source: §5 · p. 10

Abstract

from arXiv · show

Greenfield software development in freelance settings requires coding agents to turn incomplete client requests into working software. This makes requirement clarification essential, yet clients may be unable to provide continuous feedback over long development cycles, and receiving their answers does not guarantee correct understanding of complex requirements. These challenges motivate two benchmark design choices: clarification precedes code implementation, and its effectiveness is assessed through explicit tests of requirement understanding. Guided by these choices, we introduce SWE-Clar, a benchmark grounded in real, high-value client commissions to evaluate requirement clarification and code implementation in this setting. It comprises 100 tasks based on real software development jobs on Upwork, each with at least US\$4,000 in total payments. Each task begins with multi-round interaction between a coding agent and a client agent, followed by separate runs for code implementation with acceptance testing and clarification assessment. For the latter, we construct 512 questions paired with reference answers from real client-freelancer interactions, covering both explicitly stated requirements and those inferred from the conversational context. Both runs begin from the same clarification context but remain isolated, preventing assessment questions from guiding code implementation. Experiments with nine frontier models show that even GPT-6 Astra, which has the highest observed implementation score, achieves a task pass rate of only 67%, underscoring the challenge of independent software delivery in this setting. The dataset and evaluation code are available at https://github.com/upwork/SWE-Clar.

1 INTRODUCTION

Greenfield freelance commissions begin with incomplete requests, while prolonged development limits ongoing feedback and complex requirements may remain misunderstood. SWE-Clar evaluates clarification before implementation and independently tests requirement understanding and software delivery; even its strongest evaluated model passes only 67% of tasks.

1 INTRODUCTION

  • Real freelance commissions often omit critical functional and interface requirements, while long development cycles limit sustained feedback and complex requirements can remain misunderstood after replies.These challenges make evaluation of both implementation and clarification relevant.Source: §1 · pp. 1–2
  • The paper proposes completing clarification before coding and assessing requirement understanding with reference-answer questions in a run isolated from implementation.These choices target understanding rather than information acquisition alone.Source: §1 · pp. 2–3
  • SWE-Clar comprises 100 real Upwork greenfield tasks, each with at least US$4,000 in payments, and evaluates clarification and code implementation.The benchmark and evaluation code are released.Source: §1 · pp. 2–3
  • 67% of tasks pass for GPT-6 Astra, the model with the highest observed implementation score, underscoring the difficulty of independent software delivery.The study evaluates nine frontier models.Source: §1 · p. 3

2 SWE-CLAR: BENCHMARK DESIGN

SWE-Clar evaluates greenfield freelance work by separating a completed clarification phase from independent implementation and requirement-understanding assessments. Its 100 real Upwork tasks span varied development types, and source conversations show that many later-disclosed requirements emerge early.

2.1 OVERVIEW

  • Each task starts with an incomplete request and workspace, then a coding agent clarifies with a client agent before separate acceptance-tested implementation and requirement-question runs.Both evaluation runs begin from the same clarification context and proceed without further client guidance.Source: §2.1 · p. 3, Figure 2 · p. 4

2.2 REQUIREMENT CLARIFICATION

  • A supervisor reviews client-agent replies for accuracy, completeness, and scope; the coding agent may inspect materials and ask multiple questions but must finish clarification before coding.Only approved replies reach the coding agent.Source: §2.2 · p. 3

2.3 INDEPENDENT EVALUATION

  • Implementation score is the percentage of tasks passing all acceptance tests; clarification score averages question-level ratings against reference answers, separately across tasks.Clarification answers receive full, partial, or zero credit, and are graded without implementation test results.Source: §2.3 · p. 4

2.4 BENCHMARK CONSTRUCTION

  • The benchmark adapts 100 real Upwork commissions worth at least US$4,000 each for local development while preserving their business goals and fixing delivery interfaces.The coding agent receives partial materials, while complete requirements guide client replies and both evaluations.Source: §2.4 · pp. 4–5
  • The authors construct 512 requirement questions with reference answers and separate acceptance tests, and two industry-background reviewers cross-check consistency among requirements, questions, and tests.Questions cover required behavior, relevant conditions, and exceptions.Source: §2.4 · p. 5

2.5 DATASET ANALYSIS AND DISCUSSION

  • The 100 tasks span eight development-content categories and multiple runtime environments, enabling evaluation across different kinds of greenfield projects.Figure 3 groups tasks by development type and orders bars by task count.Source: §2.5 · p. 5, Figure 3 · p. 5
  • An average of 69.2% of later-disclosed requirements appear in the first quarter of observed conversation time, though timestamps do not establish implementation timing or later client availability.Figure 4 reports the conversation-level average with pointwise 95% confidence intervals.Source: Figure 4 · p. 5, §2.5 · p. 5
  • Unlike repository and continued-feedback benchmarks, SWE-Clar tests independent completion of real, high-value greenfield commissions after early client interaction and separately checks understood requirements and delivered behavior.Table 1 compares benchmark task settings, client interaction, and evaluation.Source: §2.5 · p. 6, Table 1 · p. 6

3 EXPERIMENTS

The experiments evaluate nine coding models on requirement clarification and software implementation, then examine how clarification conditions and requirement-answer correctness relate to implementation outcomes.

3.1 EXPERIMENTAL SETUP

  • Models complete clarification before evaluation branches begin; implementation is scored by acceptance tests, while requirement-question answers are graded separately.The experiments use the same 100 tasks per model and report implementation and clarification scores as percentages.Source: §3.1 · pp. 6–7

3.2 OVERALL PERFORMANCE (RQ1)

  • Implementation scores range from 39.0% to 67.0%; GPT-6 Astra scores highest but still fails 33 of 100 tasks.The main comparison reports both implementation and clarification scores for 100 tasks per model.Source: Table 2 · p. 7, §3.2 · p. 7

3.3 EFFECT OF CLARIFICATION (RQ2)

  • With client interaction, GPT-5.6 Luna’s implementation score rises from 6.0% to 49.0% and GLM-5.3 Flash’s from 5.0% to 39.0%; complete requirements raise them to 75.0% and 71.0%.Clarification scores also increase by 30.4 and 38.1 percentage points, respectively; the standard protocol closes 62.3% and 51.5% of the implementation-score gaps to complete requirements.Source: §3.3 · pp. 7–8, Figure 5 · p. 8

3.4 REQUIREMENT ANSWERS AND IMPLEMENTATION (RQ3)

  • Across models, code-test pass rates are 33.9–68.4% for incorrect answers, 73.0–88.2% for partial answers, and 86.3–91.3% for correct answers.Correct answers have higher pass rates than partial answers in all nine models.Source: §3.4 · p. 8, Figure 6 · p. 8
  • Correct answers accompany failing implementations for 6.1–7.8% of RUs per model, showing that requirement answers and code tests provide complementary evidence.Partial answers can establish substantive correct behavior while leaving material omissions or uncertainty; an unstated detail need not be absent from the code.Source: §3.4 · pp. 8–9
  • Even fully disclosed initially hidden RUs receive Incorrect labels in 2.8–6.4% of cases, distinguishing information acquisition from correct understanding.The analysis matches client-disclosure records to requirement answers from the same runs.Source: §3.4 · p. 9

4 RELATED WORK

Prior benchmarks evaluate coding performance, clarification, and interactive processes across programming problems, repositories, and simulated tasks. SWE-Clar focuses on clarification followed by independent delivery for greenfield freelance jobs.

Software Development Benchmarks

  • Software-development benchmarks span function generation, repository issue resolution, paid freelance tasks, and complete repository generation from requirements or specifications.Examples include HumanEval, SWE-bench, SWE-Lancer, DevEval, and NL2Repo-Bench.Source: §4 · p. 9

Requirement Clarification

  • Clarification benchmarks range from programming problems and repository issues to from-scratch generation, while SWE-Clar uses real Upwork jobs and multiple client exchanges before unguided implementation.Earlier approaches include ClarifyGPT, HumanEvalComm, ClarifyCodeBench, Ambig-SWE, and ICAE-Bench.Source: §4 · p. 9

Interaction and Process Evaluation

  • Interactive-agent evaluation uses simulated users, task outcomes, intent coverage, development feedback, or validated requirement and planning references; SWE-Clar separately tests understanding from the shared clarification context.Its assessment questions do not guide the implementation run.Source: §4 · p. 9

5 CONCLUSION

SWE-Clar evaluates requirement clarification and code implementation for greenfield freelance software development using real, high-value Upwork jobs. Nine frontier models show that delivery remains challenging, clarification only partly closes the gap to complete requirements, and answer–code disagreements make the two evaluations complementary.

5 CONCLUSION

  • SWE-Clar pairs acceptance testing with a separate requirement-question assessment, with both runs starting from the same clarification context.The design prevents assessment questions from guiding coding.Source: §5 · p. 10
  • Future work can extend the benchmark to changing requirements and different patterns of client availability.The paper also identifies more effective clarification and reliable delivery as future-work directions.Source: §5 · p. 10
  • The public repository provides 100 tasks, 512 requirement units, tests, prompts, evaluation tools, build configurations, implementations, and offline validation tools.The release includes reference, behaviorally equivalent, and targeted incorrect implementations for every task.Source: REPRODUCIBILITY STATEMENT · p. 10

A TASK CONSTRUCTION

The benchmark selects locally reproducible, objectively testable jobs from high-value Upwork commissions and constructs linked requirement questions and acceptance tests. The appendix also specifies privacy protections and analyzes when client requirements become explicit.

A TASK CONSTRUCTION

  • From 122 candidate jobs meeting payment, milestone, and message thresholds, the authors select 100 locally reproducible jobs with testable deliverables, excluding difficult-to-evaluate design tasks.The US$4,000 threshold refers to total source-job payments; two reviewers cross-check task consistency, and acceptance tests are developed with reference implementations.Source: A TASK CONSTRUCTION · p. 13
  • Greenfield tasks may include starter structure or runnable baseline functionality, but none of the 512 requirement units passes its corresponding tests initially.Workspaces use synthetic records and controlled local services instead of original client accounts and deployment environments.Source: A TASK CONSTRUCTION · p. 13

A.1 PRIVACY PROTECTION AND RELEASE SCOPE

  • The release excludes original conversations and identifying information, while using synthetic records and controlled services for evaluation.Materials and metadata are inspected for identifying information and credentials; individual source-conversation contents are not released.Source: §A.1 · p. 13

A.2 REQUIREMENT DISCLOSURE IN SOURCE CONVERSATIONS

  • Requirement-disclosure analysis uses final confirmed requirements, merges equivalent repetitions, and tracks when requirements become explicit in each conversation.Unconfirmed freelancer proposals and scheduling, payment, and access logistics are excluded; disclosure curves are averaged across conversations with pointwise bootstrap intervals.Source: §A.2 · pp. 13–14

B EVALUATION PROTOCOL AND SCORING

The protocol freezes the clarification context before two independent evaluation branches assess implementation and requirement understanding. It also specifies execution failure handling and a requirement-level rubric that evaluates answers independently of code-test outcomes.

B.1 ROLES, CONTEXT, AND TOOLS

  • Both evaluation branches start from the frozen clarification context, while private requirement materials remain accessible only to client simulation and evaluation agents.The coding agent autonomously ends clarification and coding; the standard protocol sets no fixed limit on clarification rounds or question count.Source: Table 3 · p. 14, §B.1 · p. 14

B.2 TIMEOUTS AND FAILURE HANDLING

  • A 900-second default timeout limits inactivity rather than total task duration, and retryable execution errors receive at most five retries at the same checkpoint.Retryable evaluation errors trigger reevaluation without recoding; completed deliveries are not automatically retried for low scores, and execution errors are recorded separately from acceptance-test failures.Source: §B.2 · p. 14

B.3 REQUIREMENT-LEVEL SCORING

  • Requirement answers are judged against required behavior and the agreed delivery interface, independently of code-test outcomes.The judge considers evidence across all answers for the task, accepts equivalent wording, and does not treat generic commitments as evidence of specific knowledge.Source: §B.3 · p. 15
  • Questions go only to the independent answer branch after clarification is frozen, preventing assessment questions from supplying requirements to the coding branch.Answers receive Correct (1), Partial (0.5), or Incorrect (0); corresponding tests clarify behavior without introducing requirements.Source: §B.3 · p. 15

C ADDITIONAL EXPERIMENTAL RESULTS

The appendix supplements the main results with cost and requirement-unit pass rates. Across models, higher call cost does not consistently accompany higher task completion, and requirement-unit pass rates exceed whole-task pass rates.

C ADDITIONAL EXPERIMENTAL RESULTS

  • The appendix adds cost and requirement-unit pass rates to the main results, with supplementary rates and mean costs reported for 100 tasks per model.Cost includes coding-agent calls during clarification and implementation, excluding client, supervisor, requirement-answering, and judge-agent calls.Source: C ADDITIONAL EXPERIMENTAL RESULTS · p. 15, §C.1 · p. 15, Table 5 · p. 16

C.1 OVERALL PERFORMANCE AND COST

  • Requirement-unit pass rates exceed whole-task pass rates, separating success on individual requirements from satisfying all requirements together.Astra has a lower mean cost than either Claude model but a higher observed implementation score; Figure 7 compares estimated mean cost with implementation score.Source: §C.1 · p. 15, Figure 7 · p. 16

C.2 INFORMATION AVAILABILITY

Both models improve as information access increases, while clarification conversations leave some initially hidden requirements undisclosed. Even fully disclosed requirements are not always understood accurately in the agent’s answers.

C.2 INFORMATION AVAILABILITY

  • Both models improve from no interaction to the standard protocol and again with complete requirements, with all four paired 95% intervals excluding zero.The paired comparisons use matched tasks; Table 7 reports task-pass-rate differences and task-bootstrap intervals.Source: §C.2 · p. 16, Table 7 · p. 17

C.3 REQUIREMENT DISCLOSURE DURING CLARIFICATION

  • After filtering conversations with outlying question–answer counts separately per model, equal-weighted final mean disclosure ranges from 33.6% to 53.7% of initially hidden requirement units across models.Source: §C.3 · p. 17

C.4 REQUIREMENT ANSWERS AFTER FULL DISCLOSURE

  • Among initially hidden requirements fully disclosed during clarification, 2.8%–6.4% of answers are Incorrect and 2.4%–14.9% are Partial across models.The analysis pairs disclosure annotations with answer judgments from the same normally completed runs.Source: §C.4 · p. 18, Table 8 · p. 18
  • Disclosure rate and clarification score differ in scope and scoring, so they should not be directly subtracted and disclosure rate is not an upper bound on clarification score.Disclosure covers initially hidden requirements with complete information; clarification score covers all requirement questions and gives partial answers half credit.Source: §C.4 · p. 18

C.5 REQUIREMENT UNDERSTANDING AND IMPLEMENTATION

Answer and implementation outcomes do not always align: partial answers are more common than incorrect ones, and correct understanding does not ensure successful implementation. The results motivate improving both clarification and translation of requirements into code.

C.5 REQUIREMENT UNDERSTANDING AND IMPLEMENTATION

  • Partial answers outnumber Incorrect answers for every model, distinguishing incomplete knowledge from answers graded incorrect.The three-level rubric separates partial from complete knowledge, while strict correctness measures fully correct answers.Source: §C.5 · p. 18, Table 9 · p. 19
  • Correctly answered requirements have higher implementation success rates than partially or incorrectly answered requirements across all nine models.The joint distribution also includes cases where agents understand requirements but fail tests, or pass tests without fully demonstrating understanding; Figure 9 reports all 512 requirement units per model.Source: §C.5 · pp. 18–19, Figure 9 · p. 19
  • The authors recommend improving both clarification quality and the translation of requirements into code, including carrying clarified requirements into planning and verification.They suggest that additional inference during coding may explain successful implementations alongside incomplete answers, while accurate understanding may not be carried through to code.Source: §C.5 · pp. 19–20

C.5.1 ANSWER–IMPLEMENTATION CASES

The two cases show that correct implementation and accurate requirement answers can diverge in either direction, so neither is a substitute for checking the other.

C.5.1 ANSWER–IMPLEMENTATION CASES

  • Task 015 passes an acceptance test for an empty result even though the agent’s answer omits the no-match behavior.The test requires HTTP 200 with an empty result list.Source: §C.5.1 · p. 20
  • Task 053 fails on an invalid empty dictionary even though the answer correctly says malformed history should raise ValueError.Converting the input to a list makes the invalid dictionary appear to be a valid empty history.Source: §C.5.1 · p. 20
  • These cases motivate translating clarified requirements into concrete checks, including distinguishing valid empty results from invalid inputs.The paper recommends checking code behavior against requirements throughout implementation.Source: §C.5.1 · p. 20

D CLARIFICATION TIMING

The workflow comparison finds higher implementation and clarification scores when clarification ends before coding than when it continues during development. The authors suggest that integrating evolving information is a challenge for interleaved workflows.

D CLARIFICATION TIMING

  • Clarification-before-coding freezes the client context before implementation, while continued clarification interleaves questions and coding and retains existing code.The comparison uses GPT-5.6 Luna and GLM-5.3 Flash on the same 100 tasks per model.Source: D CLARIFICATION TIMING · p. 20
  • Implementation scores are higher by 23 and 26 percentage points for Luna and Flash, respectively, with clarification-before-coding; clarification scores are higher by 12.8 and 22.3 points.Paired outcomes also favor clarification before coding: 30 and 29 tasks pass only with it, versus seven and three only with continued clarification, for Luna and Flash.Source: D CLARIFICATION TIMING · p. 20, Table 10 · p. 20
  • The authors hypothesize that interleaving replies, prior assumptions, and code revisions may make it harder to maintain a coherent understanding of current requirements.Lower final clarification scores provide indirect support for this explanation; additional interaction alone does not ensure more accurate understanding.Source: D CLARIFICATION TIMING · p. 21
  • For interleaved agents, the authors suggest checking critical requirements before coding and updating requirement records, plans, and checks when clarification changes an assumption.They propose these coordination practices to help turn continued interaction into consistent software behavior.Source: D CLARIFICATION TIMING · p. 21

E EFFECT OF SUPERVISOR REVIEW

This section examines supervisor review in the clarification workflow, including its effect on revised client replies and the role-defining system prompts. It also describes a behavior-focused rubric for judging requirement answers, with scoring and output rules centered on correctness, scope, and boundaries.

E EFFECT OF SUPERVISOR REVIEW

  • Among 561 revised questions, 46.0% of replies changed from incorrect to correct, versus 2.9% from correct to incorrect, supporting review before replies reach the coding agent.Successful corrections substantially outnumbered harmful revisions; the other outcomes were preserved correctness and failure to correct an existing error.Source: Table 11 · p. 21, E EFFECT OF SUPERVISOR REVIEW · p. 21

F.1 CLARIFICATION AGENT

  • The clarification agent inspects available materials, asks focused questions about decisions and acceptance expectations, incorporates answers, and ends clarification before implementation.It must not implement during clarification or seek protected evaluator files.Source: §F.1 · pp. 21–22

F.2 CLIENT

  • The client answers focused questions truthfully without automatically revealing the full private specification, preserves indispensable conditions, and reports unresolved conflicts between authoritative sources.A broad request for the entire specification prompts the client to ask which workflow or decision to discuss first.Source: §F.2 · pp. 22–24

F.3 SUPERVISOR

  • The supervisor checks client drafts against authoritative facts and conversation, requiring revision for material errors, omissions, leakage, or out-of-scope disclosures while preserving valid content.It distinguishes genuine conflicts between authoritative obligations from conversational mistakes or compatible specific exceptions.Source: §F.3 · pp. 24–26

F.4 REQUIREMENT-ANSWER BRANCH

  • The requirement-answer branch uses frozen clarification context to answer questions about intended behavior; its answers do not enter the coding branch.Assessment measures understanding, including valid inference, and forbids using private tests or later coding-branch information.Source: §F.4 · p. 27

F.5 CODING BRANCH

  • The coding branch implements and tests from its independent frozen context, without access to audit answers or private evaluation files.Passing public checks does not establish that all requirements are complete.Source: §F.5 · pp. 27–28

F.6 CODING RUNTIME INSTRUCTION

  • The coding runtime instruction directs the agent to use native tools on the mounted repository, run project checks, and leave completed source in place.The prompt appendix says tool definitions and task-specific context accompany prompts at runtime, and identifies this addition as a runtime instruction rather than a client requirement.Source: F SYSTEM PROMPTS · p. 21, §F.6 · p. 28

F.7 JUDGE AGENT

  • The judge scores saved requirement answers rather than implementation, using the target question, reference, and agreed interface to assess correctness.The prompt defines correct as establishing all material requested behavior without unresolved material contradiction.Source: §F.7 · p. 28
  • The judge scores materially correct but incomplete or uncertain answers as partial (0.5), and explicit contradictions, unresolved conflicts, or no substantive correct answer as incorrect (0).Clearly retracted statements are not unresolved contradictions; equivalent wording is accepted, and missing details alone do not constitute contradictions.Source: §F.7 · pp. 28–29
  • The judge returns JSON with ru_id, label, score, reason, and reference_conflict, giving a concise behavioral reason without quotations, citations, evidence fields, or reasoning traces.A quotation-format check is explicitly excluded from scoring.Source: §F.7 · p. 29
  • The judge assesses only material behavior relevant to the target question, credits equivalent behavior established elsewhere or logically entailed, and downgrades to partial only for a genuinely unspecified behavior.Brevity, redundant omissions, unrelated implementation details, or absent repeated field names are not grounds for downgrading.Source: §F.7 · p. 29
  • Before scoring, the judge checks applicable boundaries and qualifiers, including types, nulls, empty values, equality, defaults, authorization, failures, state changes, and replay behavior.Contradictions take precedence over otherwise correct content, but the judge must not require boundaries unsupported by authoritative materials.Source: §F.7 · p. 29

Related papers

Loading 2610.31854v1…