PaperlayerView exact revision on arXiv

Source-linked AI summary

When Requirements Interact: Scaling Long-Horizon Coding via Interdependent Specifications

Yishuo Cai, Minzheng Wang, Liang Zhao, Linghao Zhang, Lei Li, Shicheng Li, Xu Sun, Fuli Luo

arXiv:2610.40256v1cs.SEcs.CL

TL;DR

Coding agents must meet interacting engineering requirements without breaking dependent behavior, while existing datasets often lack specification quantity and diversity. CodeSpecWeave synthesizes coupled tasks and tests from developer seeds and reviews model trajectories to improve test reliability. Reinforcement learning with its training tasks improves performance across evaluated benchmarks, and its benchmark distinguishes agents while exposing remaining challenges.

  • Problem

    Existing coding-task datasets often lack specification density and structural diversity, leaving requirements such as compatibility, state consistency, and error handling underrepresented.

    Source: §1 · p. 2

  • Method

    CodeSpecWeave iteratively synthesizes coupled task requirements and executable tests from real developer seeds, with a review agent auditing multiple models’ solution trajectories and patches.

    Source: §1 · pp. 2–3

  • Results

    Reinforcement learning on CodeSpecWeave tasks raises the base model’s success rate on DeepSWE-v1.1 and yields gains over the base model on all four evaluated benchmarks.

    Source: §5 · p. 10

  • Takeaways & Limitations

    CodeSpecWeave-Bench reveals performance differences across agents and continuing challenges in satisfying interacting requirements.

    Source: §5 · p. 10

    Training experiments require substantial computational resources; task containers require infrastructure to manage many pods concurrently, and generation relies on strong models and multiple filtering rounds that discard many candidates.

    Source: Limitations · pp. 10–11

Abstract

from arXiv · show

To succeed in real-world long-horizon software development, coding agents must satisfy multi-dimensional task specifications, such as interface compatibility and error handling, and do so without breaking existing dependencies. However, current works lack specification in both quantity and diversity, often emphasizing primary goals while ignoring complex constraints. Synthesizing tasks to bridge this gap presents two challenges: naively adding specifications risks reducing tasks to trivial checklists rather than compelling joint reasoning, and verifying these subtle interactions risks overlooking hidden flaws or rejecting valid alternatives. To address these challenges, we introduce CodeSpecWeave, a framework for synthesizing tasks with rich, interdependent specifications. Grounded in real developer seeds, a synthesis agent iteratively develops both coupled task requirements and executable tests, while a review agent audits solution trajectories from multiple models to guarantee test reliability. Using this pipeline, we construct 4,670 training tasks across 2,865 repositories. Applying Reinforcement learning on these tasks significantly improves performance over the base model on DeepSWE-v1.1, largely outperforming existing baselines. Furthermore, we curate CodeSpecWeave-Bench, a rigorous benchmark comprising 100 tasks and 1,044 specifications, which clearly separates frontier models and reveals significant headroom in handling interdependent engineering constraints.

1 Introduction

Real-world coding tasks require agents to satisfy multiple engineering requirements without breaking dependent behavior, but existing datasets often underrepresent those specifications. CodeSpecWeave synthesizes interdependent requirements and tests from real developer seeds, then uses reviewed training tasks and a benchmark to study agent performance.

1 Introduction

  • Existing executable-task datasets fall short in specification density and structural diversity, often leaving compatibility, state consistency, and error handling unaddressed.These expectations matter because agents must change code without breaking behavior that other components depend on.Source: §1 · pp. 1–2
  • Adding requirements can expose solution paths or create independent-edit checklists, while verifying interactions risks missed flaws or rejection of valid implementations.Reliable tests must align with both individual requirements and their interactions while allowing different valid implementations.Source: §1 · p. 2
  • CodeSpecWeave starts from real developer specification seeds and iteratively develops coupled requirements and executable tests together.A review agent audits multiple models’ solution trajectories and patches to filter unreliable tests.Source: §1 · pp. 2–3
  • 4,670 training tasks across 2,865 repositories support reinforcement learning that improves DeepSWE-v1.1 performance; the 100-task, 1,044-specification benchmark separates leading models and reveals remaining headroom.A review agent examines solution trajectories and patches from multiple models to filter tests that accept incomplete implementations or reject valid alternatives.Source: §1 · p. 3

2 CodeSpecWeave

CodeSpecWeave defines tasks around coupled requirements and synthesizes their tests alongside them. It validates task–verifier alignment using solver trajectories and produces training tasks and a manually reviewed benchmark.

2 CodeSpecWeave

  • The framework has two stages: synthesis builds interdependent requirements and linked tests from specification seeds, while validation reviews model trajectories to find and repair query–test misalignment.Figure 2 depicts many-to-many specification–test coverage and revalidation of selected repaired pairs.Source: §2 · p. 3, Figure 2 · p. 4

2.1 Preliminaries and Problem Formulation

  • A task is a repository, request, specification set, and execution environment with a verifier; valid patches satisfy every specification, and the verifier checks them through tests.The specification set is embedded in the request, and each specification evaluates whether a patch satisfies it.Source: §2.1 · p. 3

2.2 Design Principle: Interdependent Specifications

  • Interdependent specifications break the independent-choice factorization: satisfying one requirement restricts choices for others, requiring joint reasoning rather than checklist-style edits.Additive specifications, by contrast, admit largely separate decisions.Source: §2.2 · pp. 3–4

2.3 Task Synthesis

  • The synthesis agent combines requirements around shared state transitions or components, then iteratively drafts the request, specifications, and tests with runtime feedback.Tests can check one specification multiple times or cover several obligations in an integration test.Source: §2.3 · pp. 4–5

2.4 Validation and Repair

  • Reviewing patches and execution trajectories from multiple models reveals missing checks and tests that reject valid solutions, enabling iterative query–test reconciliation.In one batch, difficulty filtering retained 3,275 of 14,905 initial tasks, and review approved 766 verified tasks.Source: §2.4 · p. 5
  • 4,670 training tasks across 2,865 repositories and a manually reviewed benchmark of 100 tasks with 1,044 specifications are produced by the pipeline.The benchmark averages 10.44 specifications per task.Source: §2.4 · p. 6

3 Experiments

Experiments evaluate CodeSpecWeave on a benchmark of interdependent requirements and test reinforcement learning across coding benchmarks, specification groups, and training dynamics. CodeSpecWeave achieves the highest average scores across both backbones, while training also changes agents’ problem-solving behavior.

3.1 Evaluation on CodeSpecWeave-Bench

  • CodeSpecWeave-Bench pass rates range from 44.0% to 74.0%, distinguishing agent capabilities while leaving substantial headroom.GPT-6 Astra, the strongest model, achieves 74.0% and fails on 26% of attempts; higher task success does not require more output tokens.Source: §3.1 · p. 6

3.1.1 Failure analysis

  • Failure analysis finds incomplete handling of input boundaries or execution paths, and changes that satisfy one requirement while violating another in a different scenario.The two error patterns involve behavior that still violates a targeted specification, or a conflict between requirements constraining the same state.Source: §3.1.1 · p. 7

3.2.1 Training and evaluation setup

  • Training uses CodeSpecWeave, SWE-PI, and MiMo Coding data with two MiMo backbones and evaluates performance on four coding benchmarks.The CodeSpecWeave training set has 4,670 tasks across 2,865 repositories; Figure 5 presents learning curves for the three datasets, with MiMo-V2.5 on the left and MiMo-V2.6-Flash-SFT on the right.Source: §3.2.1 · p. 7, Figure 5 · p. 8

3.2.2 Evaluation results

  • 50.15% and 67.60% average scores across four benchmarks are highest for CodeSpecWeave across the two backbones, with especially strong results on CodeSpecWeave-Bench and DeepSWE.CodeSpecWeave scores 52.00% and 70.67% on CodeSpecWeave-Bench and 25.30% and 62.98% on DeepSWE; performance remains competitive on SWE-bench Verified and SWE-bench Pro.Source: Table 2 · p. 7, §3.2.2 · p. 8
  • After training, the models produce more thinking tokens and modify more files and lines, while edit–test loops increase from 3.48 to 4.34 and from 2.96 to 4.90.More than 70% of trajectories sustain at least two loops, each involving code edits followed by executable test feedback in a later turn.Source: §3.2.2 · p. 8, Table 3 · p. 8

3.2.3 Performance by specification count and category

  • Flash-SFT gains 10.2–12.5 percentage points across specification-count groups and 9.5–14.9 points across seven specification categories.The largest gains occur in the 9–11 and 12–31 specification groups; categories cover functional requirements and constraints on interfaces, execution order, state, and error handling.Source: §3.2.3 · p. 9, Figure 6 · p. 9

3.2.4 Training dynamics

  • Task scores rise steadily while policy entropy stays at 0.29–0.40; MiMo-V2.5 expands response length, whereas Flash-SFT nearly doubles its agent turns.The reported adaptation strategies are extensive single-turn elaboration for MiMo-V2.5 and more multi-turn interaction for Flash-SFT.Source: §3.2.4 · p. 9, Figure 7 · p. 9

4 Related Work

Prior coding-task work draws on source code, repository changes, tests, and solver feedback, while specification-focused work structures requirements and evaluation. CodeSpecWeave makes requirement coverage and interdependence explicit in task synthesis.

Coding Data Synthesis

  • Coding-task synthesis has used code snippets, repository commits, working code, tests, runtime dependencies, transferred issue semantics, recombined seed problems, and solver feedback.CodeSpecWeave instead explicitly targets engineering-requirement coverage and interaction, using developer specification seeds and repository context to require joint reasoning.Source: §4 · p. 10

Software Specifications

  • Specification-focused systems organize requirements, guide implementation, or structure evaluation and learning; CodeSpecWeave couples requirements so choices satisfying one constrain valid choices for others.The agent must explore the repository and develop an implementation that satisfies the coupled requirements together.Source: §4 · p. 10

5 Conclusion

The paper introduces CodeSpecWeave for synthesizing executable coding tasks with interdependent specifications in real repositories. Its training set supports reinforcement-learning gains, while its manually reviewed benchmark reveals performance differences and remaining challenges.

5 Conclusion

  • CodeSpecWeave jointly refines task requirements and tests from developer-interaction seeds, while a review agent examines multiple models’ solutions for verification errors.The resulting training set contains 4,670 tasks across 2,865 repositories; reinforcement learning yields gains over the base model on all four evaluated benchmarks.Source: §5 · p. 10
  • CodeSpecWeave-Bench contains 100 manually reviewed tasks and 1,044 specifications, and evaluations reveal differences across agents and continuing challenges with interacting requirements.Source: §5 · p. 10

Limitations

The paper identifies computational and infrastructure demands, as well as reliance on strong models and filtering, as practical limitations.

Limitations

  • Using synthesized tasks for reinforcement-learning improvements requires substantial computational resources for training experiments.Source: Limitations · p. 10
  • Each task uses a container image in a Kubernetes pod, so training and evaluation require infrastructure to manage many containers concurrently.High-quality task production also relies on strong models and multiple filtering rounds that discard many generated candidates.Source: Limitations · p. 11

A Data Construction

The synthesis and solver agents use specified MiMo models and Claude Code, while manual review checks task realism, requirement consistency, and verifier reliability. The training setup uses GRPO and tracks score, entropy, response length, and agent turns.

A.1 Agents

  • Synthesis and review use MiMo-V2.5-Pro, solver agents use MiMo-V2.5-Pro and MiMo-V2.5-Flash, and all use Claude Code as their harness.Source: §A.1 · p. 14

A.2 Manual review

  • Manual review checks whether tasks reflect realistic repository development, requirements are clear and consistent, and tests cover stated requirements without imposing unstated constraints.Reviewers inspect task descriptions, verifiers, model patches, and execution trajectories.Source: §A.2 · p. 14
  • Trajectory review checks passed solutions against requirements and separates failed solutions caused by implementation defects, verifier restrictions, or environment issues.Close-to-correct failures can expose hidden constraints or boundary-case problems; reviewers also consider missing dependencies and flaky tests.Source: §A.2 · p. 14

B.1 Reinforcement learning

  • Training uses GRPO, with score, masked-token entropy, response length, and main-agent turns tracked as training dynamics.The score averages within-task pass rates before batch selection, excludes invalid outcomes, assigns zero to groups with fewer than two valid outcomes, and sampled tasks change across updates.Source: §B.1 · p. 14

B.2 Evaluation settings

Evaluation uses three samples per task with a 512K context budget, temperature 1.0, and top-p 0.95. The benchmark’s 1,044 specifications are classified through a human-in-the-loop taxonomy with primary categories and cross-cutting secondary categories.

B.2 Evaluation settings

  • Evaluation uses three samples per task, a 512K context budget, temperature 1.0, and top-p 0.95.Source: §B.2 · p. 15

C.1 Categories and benchmark annotation

  • The taxonomy has five primary categories, with functional semantics split into three subcategories, for seven categories total.Table 5 summarizes the taxonomy and distribution of the 1,044 benchmark specifications; 470 are functional specifications.Source: §C.1 · p. 15, Table 5 · p. 15
  • MiMo-V2.5 proposes primary and secondary categories, auxiliary tags, and rationales for the 1,044 specifications, which the authors then audit.Each specification receives one primary category, while secondary categories capture cross-cutting aspects.Source: §C.1 · p. 15
  • Classification follows each requirement’s primary intent, distinguishing interface contracts, transformation logic, failed-transaction invariants, and explicit exceptions or retries.The 470 functional specifications are subdivided into parsing, algorithmic computation, and domain protocols.Source: §C.1 · p. 15

C.2 Training query distributions

CodeSpecWeave training queries are longer at the median than queries in the other listed training datasets. The example task shows how a feature request combines proportional allocation with compatibility and boundary constraints.

C.2 Training query distributions

  • 2,131 characters is the median CodeSpecWeave query length, compared with 731–1,361 characters for the other training datasets.Figure 8 compares query-length distributions, and Table 6 reports statistics over complete training splits.Source: §C.2 · p. 15, Figure 8 · p. 16, Table 6 · p. 16

D Task Example

  • The Skycoin task changes multi-recipient coin-hour allocation while preserving single-recipient behavior and existing transaction and wallet semantics.The existing equal split can give larger payments no more hours than tiny ones or leave a recipient with zero hours when the remainder is small.Source: D Task Example · p. 16
  • The task requires deterministic proportional integer allocation after fees, with request-order tie breaking and no zero-hour recipient when enough hours exist for all outputs.With change, the post-fee split remains 50/50, with an odd extra hour to change; without change, hours must not be burned or discarded.Source: D Task Example · p. 16

E Development Behavior

Development-behavior analysis measures edit–test loops and retesting after failed tests in main-agent trajectories. The reported rates describe changes in these behaviors and a higher likelihood of editing and testing again after CodeSpecWeave training.

E Development Behavior

  • Within the first 40 turns, trajectories with at least two edit–test loops change from 44.0% to 54.7% for MiMo-V2.5 and from 47.7% to 45.7% for Flash-SFT.An edit–test loop is one or more edits followed by a test invocation in a later turn; edits include source code, tests, and auxiliary files.Source: E Development Behavior · p. 17
  • Both models more often edit and test again after a test invocation explicitly reports failure following CodeSpecWeave training.Table 8 reports task-averaged rates of editing and retesting after a failed test.Source: E Development Behavior · p. 17, Table 8 · p. 17

F Failure Analysis

The failure analysis identifies incomplete enforcement across input boundaries and interacting requirements that one update cannot satisfy together.

F Failure Analysis

  • The analysis checks whether the prompt requires the failed behavior, then examines the implementation and execution evidence for its violation.A frontier agent inspects solver prompts, final patches, and verification outputs, consulting tests and tool records where needed.Source: F Failure Analysis · p. 17
  • A URL after “See(” remains plain text despite link recognition and a successful parenthesized-URL probe, exposing incomplete enforcement across input settings.Figure 9 depicts the boundary mismatch; underlining denotes a URI span.Source: F Failure Analysis · p. 18, Figure 9 · p. 19
  • Adding a required signature through a whole-state copy also changes the original because the copy retains shared transaction objects, violating isolation.The update fulfills signature merge while violating state isolation; Figure 10 measures the original-state count with the verifier.Source: F Failure Analysis · p. 18, Figure 10 · p. 20
Loading 2610.40256v1…