PaperlayerView exact revision on arXiv

Source-linked AI summary

ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders

Zhongyuan Peng, Dan Huang, Chuyu Zhang, Caijun Xu, Changyi Xiao, Shibo Hong, David Lo, Lin Qiu, Xuezhi Cao, Jiyuan He, Yixin Cao

arXiv:2607.21217v1cs.AI

TL;DR

Coding agents are moving from local code completion toward building repositories from ambiguous product intent, while benchmarks have largely focused on fully specified tasks. ICAE-Bench evaluates this setting with controlled fuzzy requirements and reproducible interaction, finding that current agents achieve only 38.2% best overall functional correctness and that more information does not consistently produce more correct repositories.

  • Problem

    Coding agents increasingly build projects from high-level intent, but prior research largely focused on local tasks rather than incomplete implementation contracts.

  • Method

    ICAE-Bench evaluates fuzzy requirements using hidden User Agent records, controlled ambiguity levels, black-box behavioral cases, and multidimensional repository diagnostics.

  • Results

    38.2% is the best Overall functional-correctness rate, while GroundPRD outperforms recovered requirements for four of six models.

  • Takeaways & Limitations

    More requirement information alone does not consistently yield correct repositories; actionable information and preserving clarified constraints are more important.

Abstract

from arXiv · show

The recent emergence of vibe-coding workflows is changing what coding agents are expected to do. Instead of merely completing code under fully specified instructions, agents are increasingly expected to transform incomplete product intent into working software by combining various abilities including planning, requirement clarification, tool use, debugging, and repository-level construction. Yet existing benchmarks have not fully caught up with this shift, evaluating agents on static, fully specified tasks. In this paper, we introduce ICAE-Bench, a benchmark for evaluating coding agents under interactive project-building settings. The basic idea is to start from a fuzzy product requirement, simulating the dynamic paradigm with an automated User Agent. To make this setting both realistic and evaluable, ICAE-Bench introduces three key designs. First, to avoid the ambiguity of unconstrained fuzzy requirements, each task derives ambiguity from a precise real open-source repository with executable behavior. Second, to ensure high-quality and reproducible user simulation, ICAE-Bench grounds interaction through User Agent Data, allowing the User Agent to reveal hidden constraints without inventing new requirements or leaking implementation artifacts. Third, to evaluate open-ended repositories fairly, ICAE-Bench uses standardized black-box tests together with multi-dimensional diagnostics, including functional correctness, semantic and API similarity, structural fidelity, design quality, and interaction quality.

I. INTRODUCTION · II. RELATED WORK · A. Software-Agent Benchmarks

ICAE-Bench evaluates coding agents as interactive project builders that transform fuzzy product intent into working repositories through clarification, tool use, and iterative construction. It grounds ambiguity in real repositories, controls interaction with benchmark-authored User Agent Data, and combines black-box testing with multidimensional diagnostics across a large multilingual benchmark.

  • I. INTRODUCTION: Coding agents increasingly must turn incomplete human intent into functioning software, rather than only complete localized coding tasks.Vibe-coding workflows require clarification, repository inspection, command execution, dependency installation, and iteration over failures.
  • I. INTRODUCTION: A grounded User Agent answers clarification questions only from benchmark-authored requirement records, preventing invented requirements, leaked implementation details, and hallucinated answers.The coding agent operates in an ultimate image that preserves the runtime environment while removing golden code and original materials.
  • I. INTRODUCTION: Existing benchmarks largely assume static, explicit goals and therefore under-evaluate the coupled capabilities of planning, clarification, coding, and iterative project construction.Function-, repository-editing-, from-scratch-generation, and executable-reconstruction benchmarks expose more static task interfaces.
  • I. INTRODUCTION: ICAE-Bench starts each task from a fuzzy requirement derived from a precise real open-source repository, grounding ambiguity while preserving executable target behavior.This design reduces the risk that failures reflect an underdefined or infeasible prompt rather than coding-agent limitations.
  • I. INTRODUCTION: Three fuzzy levels, L1–L3, progressively expose information toward GroundPRD, enabling controlled comparisons of agents’ robustness to ambiguity.L1 provides the least information, L2 restores selected ambiguity points, and L3 retains most GroundPRD while removing limited details.
  • I. INTRODUCTION: ICAE-Bench evaluates diverse repositories by constraining required behavior rather than implementation, combining black-box functional tests with artifact-, structure-, and interaction-level diagnostics.The protocol is designed to accommodate different internal designs, module structures, dependencies, APIs, and implementation choices.
  • I. INTRODUCTION: 480 tasks span 12 programming languages, with a 50-task ICAE-Bench-Lite subset, fuzzy PRDs, ultimate images, recoverable User Agent Data, and authoritative black-box cases.The benchmark evaluates six coding models within the Claude Code and OpenHands frameworks.
  • A. Software-Agent Benchmarks: Software-agent benchmarks have progressed from isolated function synthesis toward repository-level, multi-file, executable-environment, and tool-using software-engineering evaluation.HumanEval and MBPP assess local programming, while RepoBench, M2RC-Eval, and issue-resolution benchmarks extend context to repositories and real software issues.

B. 0-to-1 Repository Generation … B. Benchmark Construction

ICAE-Bench evaluates from-scratch repository generation from fuzzy requirements by combining clarification-grounded interaction with language-agnostic behavioral testing. Its construction pipeline derives reproducible tasks from verified open-source behavior, progressively fuzzifies requirements, and augments evaluation with robust black-box cases and reproducible execution environments.

  • B. 0-to-1 Repository Generation: Existing 0-to-1 benchmarks expose substantial scaffolding or directly specified behavior, whereas ICAE-Bench hides decisive information and recovers requirements through interaction.Commit0 preserves repository structure, classes, and signatures; ProgramBench uses executables and documentation as static behavioral targets.
  • C. Ambiguity and Clarification: Ambiguity-focused benchmarks test clarification or classify unclear specifications, but ICAE-Bench extends this setting to interactive, from-scratch repository generation.HumanEvalComm evaluates clarification questions, while Orchid analyzes lexical, syntactic, semantic, and vagueness-related ambiguity in function generation.
  • III. ICAE-BENCH: Each ICAE-Bench task starts from a fuzzy PRD, supports User Agent clarification, and evaluates the resulting repository with authoritative Public and Hidden black-box cases.The five stages are repository filtering, GroundPRD synthesis and test refactoring, PRD fuzzification, ultimate-image packaging, and artifact verification.
  • A. Benchmark Instance Definition: A benchmark instance separates initially visible information, interactively recoverable requirements, and the behavioral target used for evaluation.The coding agent observes the fuzzy PRD and execution environment, queries the User Agent under a fixed budget, and produces repository R evaluated against B.
  • A. Benchmark Instance Definition: Public cases provide representative Native examples, while authoritative evaluation includes Native and Enhanced cases, preserving the distinction between requirement exposure and behavioral testing.In the file-parser example, clarification reveals representative formats while evaluation additionally tests unexposed malformed and boundary inputs.
  • B. Benchmark Construction: The construction pipeline retains repositories whose original tests pass in controlled Docker environments and converts externally observable behavior into standardized JSON black-box cases.Cases use task-specific dispatchers, expected outputs from golden implementations, and normalized repository-level error categories.
  • B. Benchmark Construction: GroundPRD embeds approximately one or two representative Native cases per leaf feature, mirrors them into Public, hides remaining Native cases, and fuzzifies requirement exposure across L1–L3.The behavioral target remains common across levels, while Fuzzy L1 hides selected API, edge-case, and architectural constraints.
  • B. Benchmark Construction: Enhanced cases target boundaries, special constraints, stress behavior, and logic-specific corner cases, while ultimate images package dependencies in explicit, reproducible language-specific environments.Enhanced cases are disjoint from Native cases and validated against the golden implementation; packaging removes golden code, original tests, and hidden construction artifacts.

C. Artifact Verification · D. Benchmark Statistics · E. Interactive User Agent

ICAE-Bench validates artifacts against executable repository behavior, spans 12 programming languages with 40 tasks per language, and uses a constrained User Agent to simulate stakeholder interaction while limiting information leakage. The benchmark combines iterative contract reconstruction, repository-scale statistics, and isolated query-based interaction.

  • C. Artifact Verification: Retained golden repositories must pass original and reconstructed Native cases, while every Enhanced case must pass against the golden implementation before acceptance.This upstream verification prevents benchmark-construction errors from being attributed to coding agents.
  • C. Artifact Verification: GroundPRD correctness is established through iterative reconstruction until GroundPRD, Public examples, and Native cases are mutually consistent.Claude-Opus-4.8 implements each repository, and failing cases trigger repairs or contract revisions before repetition.
  • C. Artifact Verification: On ICAE-Bench-Lite, Fuzzy L1 and L3 were equivalent to GroundPRD for all 50 repositories, with mean similarity scores of 0.952 and 0.942 and no likely same-test failures.GPT-5.5 jointly compares fuzzy PRDs and User Agent Data with GroundPRD under a fixed rubric.
  • D. Benchmark Statistics: ICAE-Bench covers 12 programming languages with 40 tasks per language, while ICAE-Bench-Lite prioritizes smaller repositories by golden-code LOC.The full benchmark contains 480 tasks, and the Lite variant supports rapid ablations and framework comparisons.
  • D. Benchmark Statistics: Golden-code LOC averages 35,533 for ICAE-Bench and 979 for ICAE-Bench-Lite, with maxima of 2,918,810 and 1,509, respectively.The minimum is 318 LOC for both benchmark variants.
  • D. Benchmark Statistics: Golden file counts average 258 for ICAE-Bench and 17 for ICAE-Bench-Lite, with maxima of 17,488 and 61, respectively.The minimum is four files for both variants.
  • E. Interactive User Agent: The User Agent is a model-based router using DeepSeek-V3.2 by default, with a budget of 16 queries and replies addressing at most three matched technical points.Coding agents submit a task identifier and one natural-language question through an HTTP endpoint, receiving a reply and remaining budget.
  • E. Interactive User Agent: Routing returns only benchmark-authored matched-record responses or a fixed fallback, while agents and the User Agent remain separated from source code, authoritative cases, and internal records.The User Agent receives generated records rather than golden artifacts, the coding agent sees only natural-language replies, and evaluation cases are injected afterward in a fresh container.

F. Evaluation Metrics … B. Main Results

ICAE-Bench combines functional, agentic, structural, and interaction metrics to evaluate repository-scale coding agents under fuzzy requirements. Main results show that agents achieve limited fuzzy-requirement performance, while interaction and greater requirement exposure recover only part of the gap to complete specifications.

  • F. Evaluation Metrics: Functional correctness tests externally observable behavior across Native versus Enhanced and Public versus Hidden cases, with Public cases forming a subset of Native cases.Native cases derive from original repository tests, whereas Enhanced cases add synthesized robustness coverage; Public cases are exposed in GroundPRD and Hidden cases are the remaining cases.
  • F. Evaluation Metrics: Agentic, structural, and interaction metrics diagnose semantic and API fidelity, design quality, repository construction strategy, and hidden-requirement retrieval.Structural assessment uses file-count and LOC ratios plus class, method, and namespace/package similarities; interaction quality measures constraint coverage, fallback rate, and budget usage.
  • IV. EXPERIMENTS: The experiments address four questions covering fuzzy-requirement implementation, interaction-based recovery, clarification-to-implementation bottlenecks, and sensitivity to system configurations.The reported sensitivity factors include agent framework, User Agent backbone, interaction budget, test scaffolding, thinking configuration, and execution environment.
  • A. Experimental Settings: The full benchmark measures realistic repository-scale performance, while ICAE-Bench-Lite’s 50-repository subset supports repeated controlled ablations.The Lite subset is selected by sorting full tasks by golden-code LOC and enables comparisons across models, frameworks, User Agent backbones, budgets, thinking settings, and execution environments.
  • B. Main Results: Main results first evaluate default Fuzzy L1 with User Agent interaction, then vary requirement exposure from Fuzzy L1–L3 and RecoveredPRD to complete GroundPRD.Six coding models are compared under the shared default framework and interaction protocol on both the full benchmark and ICAE-Bench-Lite.
  • B. Main Results: 38.2% is the best Overall rate, while Public rates consistently exceed Enhanced rates and similar functional scores conceal differences in design, repository scale, and requirement recovery.The leading model varies across programming languages; agentic rankings also diverge from functional rankings, and structural ratios indicate sharply different construction strategies.
  • B. Main Results: Interaction and greater initial exposure recover only part of the gap to GroundPRD because agents must organize, retain, and translate fragmented answers into implementation decisions.Even complete access to recoverable L1 records remains insufficient for several models, and the results distinguish a requirement-access gap from an information-to-execution gap.

C. Analysis · 1) Failure Mode Taxonomy: · 2) Effect of Test Scaffolding:

ICAE-Bench analysis localizes failures across execution, harness compliance, feature coverage, and behavioral fidelity, while executable test scaffolding substantially improves correctness without increasing semantic constraint coverage. These findings show that low performance reflects failures throughout the clarification-to-implementation chain, not requirement access alone.

  • C. Analysis: The analysis combines evaluator-visible failure categories with controlled executable feedback to localize breakdowns and test whether verification reduces them.It also varies interaction, reasoning, and execution settings to address the broader analysis questions.
  • 1) Failure Mode Taxonomy:: Mismatch, Missing, Exec., and No Test distinguish behavioral errors, incomplete feature coverage, build or runtime failures, and harness noncompliance.Mismatch indicates incorrect output; Missing indicates absent output in otherwise running suites; Exec. indicates recognized build or runtime errors; No Test indicates a missing rcb_tests/test.sh entry point.
  • 1) Failure Mode Taxonomy:: Failure labels apply separately to Public, Native, and Enhanced suites, can overlap within repositories, and define Clean only when every evaluated case passes.Only 1–13 of 480 repositories per model are Clean, because this repository-level criterion is stricter than case-level Overall pass rate.
  • 1) Failure Mode Taxonomy:: Stronger models more often reach execution but violate behavioral contracts, whereas weaker models more often omit the harness or fail during execution.Missing outputs further separate incomplete feature coverage from incorrect implemented behavior.
  • 2) Effect of Test Scaffolding:: The scaffolding treatment changes information representation and actionability rather than semantic coverage, making verification easier without recovering more constraints.Without files, the agent must recover case fragments through interaction with the User Agent.
  • 2) Effect of Test Scaffolding:: Ready-to-run Public case files raise Overall from 37.4% to 61.8%, while constraint coverage changes from 63.8% to 61.2%.The files preserve the same recoverable case content but package inputs, expected outputs, and invocation format into an executable structure.
  • 2) Effect of Test Scaffolding:: Overall, the analyses locate breakdowns at execution and behavioral fidelity and show that verification improves correctness without recovering more constraints.Low scores therefore reflect failures across the clarification-to-implementation chain rather than requirement access alone.

3) Impact of Agent Frameworks: … 7) Impact of Execution Environment:

The results show that coding-agent performance depends not only on requirements and reasoning, but also on framework behavior, interaction capacity, and execution environment. Across ablations, stronger information access does not reliably improve correctness when integration, verification, or implementation constraints remain limiting.

  • 3) Impact of Agent Frameworks:: Framework choice changes both absolute performance and model ranking, with OpenHands lowering Overall by 5.5–21.8 points across models.GPT-5.5 drops by 21.8 points, while Claude-Opus-4.8 drops by 5.5 points under OpenHands.
  • 4) Impact of User Agent Capability:: User-agent backbone affects retrieval and correctness non-monotonically: Gemini-3.1-Flash-Lite has the highest constraint coverage but the lowest Overall.Gemini-3.5-Flash and DeepSeek-V3.2 achieve similar correctness with lower coverage.
  • 5) Impact of Interaction Budgets:: Overall peaks at 37.4% with 16 queries, while coverage rises from 57.9% to 71.4% across budgets of 8, 16, and 24.Increasing the budget to 24 raises fallback from 27.7% to 31.9% and lowers Overall to 34.4%.
  • 6) Impact of Thinking Configuration:: Deliberation improves Overall in both tested comparisons, raising Gemini-3.1-Pro from 26.7% to 36.6% and GLM-5.1 from 37.4% to 40.0%.The larger effect occurs when thinking is enabled rather than disabled.
  • 7) Impact of Execution Environment:: The richer execution image lowers Overall from 37.4% to 28.4% despite increasing constraint coverage and repository size.Generated repositories reach 191.0% of golden file count and 674.1% of golden LOC.
  • 7) Impact of Execution Environment:: Richer environments permit more complex implementations but add dependencies, interfaces, modules, and verification burden without earning partial credit.The base image favors simpler, more self-contained implementations, whereas the provisioned image expands the available action space.

8) Cross-lingual Generalization: … VI. CONCLUSION AND FUTURE WORK

ICAE-Bench evaluates interactive repository construction across languages and shows that actionable, verifiable requirements matter more than information quantity alone. Its validation, efficiency analyses, and planned extensions define both the benchmark’s current scope and future direction.

  • 8) Cross-lingual Generalization:: Reimplementing non-Python tasks in Python changes Overall from 26.6% to 29.9%, showing cross-language implementation is feasible but source-task effects are mixed.The results do not establish a uniform Python advantage.
  • 8) Cross-lingual Generalization:: Framework support, executable Public scaffolding, and thinking improve correctness, whereas retrieval, interaction budgets, environments, and language changes have non-monotonic effects.Configurations matter chiefly when they make requirements actionable and verifiable.
  • V. DISCUSSION: Across RQ2–RQ4, more requirement information does not consistently improve correctness; the strongest gains occur when information becomes actionable through executable Public cases.This motivates preserving clarified constraints, linking them to implementation decisions, and verifying those decisions during construction.
  • A. Validity of ICAE-Bench-Lite.: Overall rankings have moderate consistency, with Spearman’s ρ = 0.71; Claude-Opus-4.8 leads the full benchmark, while GPT-5.5 leads ICAE-Bench-Lite.GPT-5.5 has higher constraint coverage on both splits, whereas Claude-Opus-4.8 is more robust as repository scale increases.
  • B. Reliability of Subjective Metrics.: Human–critic agreement is positive but moderate, so functional tests and subjective metrics are complementary rather than interchangeable.Host-verified cases assess externally observable correctness, while critic scores provide scalable diagnostics for semantic breadth, API fidelity, and design quality.
  • C. Runtime and Token Usage.: The full split increases runtime and token usage for every model, but these measures remain weak proxies for correctness.Claude-Opus-4.8 has the highest full-benchmark Overall pass rate without being the slowest or most token intensive.
  • VI. CONCLUSION AND FUTURE WORK: ICAE-Bench evaluates repository construction from ambiguous product requirements across 480 multi-language tasks, including requirement recovery and final-artifact correctness.Its fuzzy PRDs and reproducible User Agent protocol move evaluation beyond static code generation.
  • VI. CONCLUSION AND FUTURE WORK: Future work will extend ICAE-Bench to end-to-end, multimodal real-world scenarios with frontend tasks, complex visual inputs, and richer human-in-the-loop evaluations.The planned integration with frameworks such as OpenClaw targets next-generation interactive agents.

APPENDIX A PROMPT FOR TEST REFACTORING AND GROUNDPRD · APPENDIX B PROMPT FOR ENHANCED TEST CASE GENERATION

Appendix A defines how to refactor repository tests into implementation-independent black-box contracts and a self-contained PRD. Appendix B specifies how to generate disjoint, multidimensional enhanced cases and validate them against a reference solution.

  • APPENDIX A PROMPT FOR TEST REFACTORING AND GROUNDPRD: Retain only original tests that check externally visible behavior independent of implementation, converting them into JSON cases and input/output PRD descriptions.White-box and implementation-specific assertions are discarded, while each retained test becomes a standardized case.
  • APPENDIX A PROMPT FOR TEST REFACTORING AND GROUNDPRD: The PRD must use neutral, self-contained feature descriptions without repository identifiers, implementation details, concrete case values, or exact expected outputs.Each leaf feature includes a user story, plain-language behavior, a public-test pointer, and embedded example JSON cases.
  • APPENDIX A PROMPT FOR TEST REFACTORING AND GROUNDPRD: The refactoring deliverables include a core system description, a JSON stdin/stdout execution adapter, and a bash harness that runs selected case directories.The harness records per-case stdout files and supports a --cases-dir option with test_cases as the default.
  • APPENDIX B PROMPT FOR ENHANCED TEST CASE GENERATION: Enhanced test generation analyzes existing cases for missing boundary, special-constraint, performance-stress, and logic-specific dimensions.The agent reads the problem description, reference solution, and existing test data without modifying start.md.
  • APPENDIX B PROMPT FOR ENHANCED TEST CASE GENERATION: New enhanced cases must follow the native schema, contain only newly generated cases, and remain disjoint from the existing suite by excluding exact input/output duplicates.The enhanced suite is written separately rather than copying or merging native cases.
  • APPENDIX B PROMPT FOR ENHANCED TEST CASE GENERATION: Each enhanced case is evaluated against the reference solution, with logic or JSON-format errors fixed and rerun until all cases pass.The final materials include an execution log confirming successful validation against the golden implementation.
  • APPENDIX B PROMPT FOR ENHANCED TEST CASE GENERATION: The enhanced-test output includes a short coverage summary and the generated JSON cases saved to the specified output target path.The output target is supplied as an input file path, alongside paths for the description, solution, and existing cases.

APPENDIX C PROMPTS FOR FUZZY L1/L2 PRD CONSTRUCTION

The appendix defines prompts for generating fuzzy product requirements and oracle data from precise software artifacts. It emphasizes non-technical framing, deliberate ambiguity, hidden-constraint tracking, and structured JSON output.

  • Role Definition: The User Agent acts as a non-technical Operations PM who reverse-engineers a fuzzy business requirement and hidden interactive checkpoints from implementation, PRD, and tests.The required outputs include fuzzy_prd and oracle_data grounded in the source artifacts.
  • Generation Rules: Prompts translate missing parameters and technical details into non-technical business pain points rather than listing them directly.This reduces perspective-specific technical knowledge in the generated requirement.
  • Generation Rules: Generated requirements intentionally contain 1–2 vague internal references or omitted context, while hidden-constraint keywords remain absent from the public PRD.These rules require the tested agent to search the codebase without exposing exact hidden terminology.
  • Output Structure: Each omitted or blurred constraint must appear in oracle_data with semantic trigger keywords and an oracle response.The output structure also includes task_id, persona, golden_api_signature, context_pointers, and fallback_response.

APPENDIX D PROMPTS FOR FUZZY PRD SEMANTIC EQUIVALENCE · APPENDIX E PROMPTS FOR FUZZY L3 PRD CONSTRUCTION

Appendix D defines a joint audit of fuzzy PRDs and oracle data for semantic equivalence and execution-contract risk, while Appendix E specifies constraint extraction and controlled rewriting for fuzzy L3 construction.

  • APPENDIX D PROMPTS FOR FUZZY PRD SEMANTIC EQUIVALENCE: The semantic-equivalence judge evaluates the generated fuzzy_prd together with oracle_data rather than fuzzy_prd alone.This joint evaluation supports semantic-equivalence assessment while separately identifying execution-contract risks that may affect reuse of the same tests.
  • APPENDIX D PROMPTS FOR FUZZY PRD SEMANTIC EQUIVALENCE: The judge reports task_semantic_similarity, task_semantic_equivalent, execution_contract_risk, and likely_to_fail_same_tests.Similarity is scored in [0, 1], while execution-contract risk is classified as low, medium, or high.
  • APPENDIX D PROMPTS FOR FUZZY PRD SEMANTIC EQUIVALENCE: task_semantic_equivalent is true when similarity is at least 0.90 and no material task-level contradiction changes implementation or behavioral tests.The criterion distinguishes high semantic similarity from contradictions that alter the task or its tests.
  • APPENDIX E PROMPTS FOR FUZZY L3 PRD CONSTRUCTION: Constraint extraction reads the detailed PRD with all supporting files and produces a JSON object containing every distinct behavioral constraint.The extraction is framed as precision analysis by a senior engineer.
  • APPENDIX E PROMPTS FOR FUZZY L3 PRD CONSTRUCTION: The extracted constraints cover mapping and branching, exact output fields and sentinel values, action and dispatch contracts, optional parameters and defaults, and validation or error-rejection rules.These categories target both behavioral logic and exact interface contracts.
  • APPENDIX E PROMPTS FOR FUZZY L3 PRD CONSTRUCTION: Each constraint records an id, feature, input condition, output rule, exact values, and an ambiguous flag, with output restricted to {"constraints": [...]} JSON.The prescribed schema makes literal requirements and ambiguity explicit.
  • APPENDIX E PROMPTS FOR FUZZY L3 PRD CONSTRUCTION: At most three constraints may be hidden, limited to difficult-to-guess literals such as deliberate typos, obscure magic constants, and non-obvious sentinel values.Each hidden constraint receives a vague replacement, exact strings to replace, at least five trigger keywords, and a precise oracle answer.
  • APPENDIX E PROMPTS FOR FUZZY L3 PRD CONSTRUCTION: The rewriting process also converts one or two non-hidden constraints into vague internal references while preserving action names, parameter names, defaults, error categories, and feature descriptions.The final response uses only JSON with hide and vague_refs arrays.

APPENDIX F CODING AGENT TASK INSTRUCTION … APPENDIX H INIT FOR USER AGENT

The appendices specify a containerized coding-agent workflow in which agents implement functionality from PRDs, clarify intentionally vague requirements, and interact with a strictly grounded User Agent. They also define constrained interaction and JSON response rules for reproducible evaluation.

  • Coding Agent Task Instruction: The coding agent must enter the running Docker container and perform all development, compilation, execution, and self-testing inside it.The PRD is located inside the container, and deliverables must be written under the bind-mounted working directory.
  • Coding Agent Task Instruction: The agent must implement the full functionality itself and may not download the target repository or directly call its existing API.It must first read the PRD thoroughly and clarify intentionally incomplete requirements before coding when instructed.
  • Coding Agent Task Instruction: The container grants root access and network access, requiring the agent to install missing dependencies and repair build or runtime problems based on feedback.The instructions explicitly allow package installation through ecosystem-specific tools and caution against assuming dependencies are predeclared.
  • APPENDIX G CLARIFICATION BLOCK FOR FUZZY PRDS: Fuzzy PRDs expose a clarification endpoint where agents ask focused technical questions about hidden constraints, interface details, and edge cases.Replies are returned in data.reply, while the remaining query budget appears in status.remaining.
  • APPENDIX H INIT FOR USER AGENT: The User Agent acts as a strict requirements provider whose knowledge and hidden requirements are limited to injected JSON structured data.It is instructed to judge strictly, avoid overstepping, and follow the provided data rather than general programming knowledge.
  • APPENDIX H INIT FOR USER AGENT: The User Agent gives short, direct answers only to precise questions, withholding unsolicited hints and rejecting underspecified requests.It reveals information only when the agent explicitly asks about a relevant pain point.
  • APPENDIX H INIT FOR USER AGENT: Each User Agent reply may address at most three specific technical points, and its output must be a valid JSON string without Markdown or surrounding text.The response schema includes internal interaction logs and a natural-language reply field.

APPENDIX I CRITIC MODEL FOR AGENTIC EVALUATION

The critic model performs a static, source-tree comparison that scores generated code on semantic similarity, API similarity, and design quality. It enforces scoped source review, excludes non-source artifacts, and requires schema-compliant JSON with concrete rationales.

  • Review procedure: The reviewer compares original and generated source trees without running code, tests, or inferring behavior from tests.Only source files in the in-scope paths are read during the static review.
  • Review procedure: The review excludes tests, build artifacts, generated files, and configuration from both trees.Examples include bin/, obj/, target/, vendor/, .gradle/, node_modules/, and dist/.
  • Scoring dimensions: semantic_similarity measures behavioral equivalence, edge cases, algorithmic fidelity, side effects, and coverage of the original behavioral surface on a 0.0–1.0 scale.The score is holistic and evaluates observable behavior under the same inputs.
  • Scoring dimensions: api_similarity separately measures alignment with the original public API, including imported names and shapes, idiomatic correctness, and public-surface coverage on a 0.0–1.0 scale.It must not be conflated with semantic_similarity.
Loading 2607.21217v1…