PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-Nexus: Automated Composition of Software Engineering Tasks from Test Execution

Songwei Yu, Ziqi Miao, Jingxuan Xu, Junhao Fan, Haoyang Huang, Yuan Xiong, Zihang Xu, Weihao Li, Wenqiang Zhu, Yuechen Luo, Xinshun Feng, Yanan Wu, Ken Deng

arXiv:2610.20038v2cs.SEcs.AI

TL;DR

Historical code changes do not identify which production functions selected tests exercise or support composing validated behavior beyond one change. SWE-Nexus constructs tasks from traced, validated test–function scopes and evaluates a benchmark spanning repositories, languages, models, and scaffolds. The best configuration resolves 37.8% of tasks in one attempt, while scope-weighted scoring measures partial recovery.

  • Problem

    Historical changes provide task boundaries but do not show which production functions selected tests exercise or support composing validated behaviors beyond one change.

    Source: §1 · pp. 1–2

  • Method

    SWE-Nexus traces test execution to identify production-function scopes, validates them against original and masked repositories, and deterministically composes compatible scopes with revalidation.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    37.8% of tasks are resolved in one attempt by the best configuration across a 600-task benchmark from 105 repositories and six languages.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    Composing validated execution footprints extends tasks beyond individual historical changes to span functions and modules.

    Source: §7 · p. 8

    Because SWE-Nexus relies on existing repository tests, behaviors and edge cases those tests do not exercise may be underrepresented; the benchmark currently covers six languages.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Historical code changes provide convenient task boundaries, but do not identify which production functions selected tests exercise or support composing validated behaviors beyond a single change. We present SWE-Nexus, a fully automated pipeline that constructs repository-level implementation tasks from test execution. Four construction agents select tests, adapt tracing and evaluation protocols to each repository, and generate behavioral specifications. For each selected test file, tracing records an observed execution footprint of production function bodies. A scope is retained only if all expected scoring points pass in the original repository and masking the traced bodies causes at least one failure or error. Compatible scopes are composed deterministically, with shared functions counted once and every constituent test file revalidated under the complete combined mask. The benchmark contains 600 tasks from 105 repositories across six programming languages, including 116 atomic and 484 composite tasks. We evaluate 12 models under three agent scaffolds. The best configuration resolves 37.8% of tasks in one attempt, and scope-weighted scoring measures partial recovery. SWE-Nexus enables task construction from validated behavior beyond individual historical changes.

1 Introduction

SWE-Nexus addresses the gap between historical changes and the production functions tests actually exercise by constructing and validating repository-level tasks from execution. Its benchmark spans atomic and composite tasks, and the best evaluated configuration resolves 37.8% in one attempt.

1 Introduction

  • Historical PR boundaries do not show which modified or unchanged production functions a test exercises, and patch extraction can include unrelated tests.These limitations motivate constructing implementation scopes validated through test execution.Source: §1 · pp. 1–2
  • A four-agent pipeline traces test execution to identify production-function scopes, validates each scope, and deterministically composes compatible scopes with revalidation.A scope is retained when all expected scoring points pass originally and masking its recorded function bodies causes at least one failure or error.Source: §1 · p. 2
  • 37.8% of tasks are resolved in one attempt by the best configuration across 12 models and three agent scaffolds.The benchmark contains 600 tasks from 105 repositories across six programming languages.Source: §1 · p. 2
  • The benchmark includes 116 atomic tasks and 484 composite tasks, while evaluation introduces scope-weighted scoring for partial recovery.Composite tasks span two to nine test files.Source: §1 · p. 2

2 Related Work

Related repository-level benchmarks derive tasks from historical changes, reproducible bugs, or synthesized requirements and tests. Other work focuses on combining executable tasks, developing agent scaffolds, or training agents with execution feedback.

Repository-level evaluation

  • Repository-level benchmarks range from reproducible bug datasets and issue-based patch evaluation to fresh-task collection and longer-horizon changes with explicit requirements.Examples include Defects4J, BugsInPy, SWE-bench, Multi-SWE-bench, SWE-rebench, and SWE-Bench Pro.Source: §2 · p. 2

Constructing executable tasks

  • Task-construction approaches combine bugs, inject bugs iteratively, synthesize tests and descriptions from commits, or trace tests across multiple PRs.The cited examples include SWE-smith, Self-Play SWE-RL, R2E-Gym, and FeatureBench.Source: §2 · p. 2

Agent scaffolds and training

  • Agent research includes code-inspection interfaces, fixed repair pipelines, interface ablations, and training with test-based or patch-similarity rewards.The passage describes SWE-agent, OpenHands, Agentless, SWE-Gym, R2E-Gym, SWE-RL, and DeepSWE.Source: §2 · p. 2

3 Dataset Overview

SWE-Nexus tasks ask solvers to restore functionality from a natural-language specification while selected production function bodies and scoring tests are withheld. The benchmark contains 600 tasks spanning six languages, with task difficulty based on test-file count and restoration-patch size.

3 Dataset Overview

  • The solver works in an available surrounding codebase with selected production function bodies masked and scoring tests withheld.Unselected repository tests may remain available during development.Source: §3 · p. 3

3.1 Task Definition

  • A task pairs selected test files and a validated scope of production functions whose bodies are masked in a fixed-revision repository.The solver receives a natural-language specification and the masked repository; selected tests are withheld for evaluation.Source: §3.1 · p. 3
  • A scoring point is an evaluation unit, typically a test case or parameterized instance; a selected test file serves as one point when finer outcomes are unavailable.Individual assertions are not counted separately.Source: §3.1 · p. 3

3.2 Dataset Characteristics

  • 600 tasks across 105 repositories cover six languages, including 116 atomic tasks and 484 composite tasks combining two to nine test files.Tasks span behaviors such as configuration parsing, application interfaces, and data serialization.Source: §3.2 · p. 3
  • Tasks have a median of five selected test files, 35 masked production function bodies, and 433 changed lines in reference restoration patches.The scoring-point range is 1–202, totaling 15,975 task–point pairs across the release.Source: §3.2 · p. 3
  • Difficulty is the higher tier assigned by selected test-file count (1–3, 4–6, or 7–9) and reference restoration patch size (1–300, 301–999, or at least 1,000 changed lines).Every language includes all three difficulty groups.Source: §3.2 · p. 4

4 Dataset Construction

SWE-Nexus constructs repository-level implementation tasks by tracing tests to production functions, validating those scopes, and composing compatible scopes into tasks with behavioral specifications and fixed evaluation.

4 Dataset Construction

  • Four construction agents select tests, adapt tracing and evaluation protocols, and generate specifications; fixed procedures validate scopes and recheck constituent tests after deterministic composition.Tracing links selected tests to observed production function bodies, from which atomic and composite tasks are constructed.Source: §4 · p. 4

4.1 Execution-Grounded Scope Validation

  • For each selected test file, tracing records observed production function bodies; a scope is retained only when original scoring points pass and masking causes a failure or error.The protocol adapts test execution and result parsing to each repository, then independently validates original and masked runs.Source: §4.1 · pp. 4–5

4.2 Deterministic Scope Composition

  • Compatible scopes are unioned with shared functions deduplicated, then every constituent test file is revalidated under the complete combined mask.Each file must pass in the original repository and have at least one scoring point fail or error under the combined mask; composite tasks combine two to nine files.Source: §4.2 · p. 5

4.3 Behavioral Specification

  • The Query Agent generates a natural-language specification from original implementations, selected tests, and repository context while selected tests and execution metadata remain hidden.Requirements describe observable inputs, outputs, boundary cases, defaults, error handling, and side effects.Source: §4.3 · p. 5

4.4 Fixed Evaluation and Scoring

  • A task resolves only when every expected scoring point passes without additional detected failures; scope-weighted scoring allocates function-body span across associated points, counting each function span once.Weights are fixed before solving, and the evaluation protocol is applied consistently to submissions.Source: §4.4 · p. 5

5 Evaluation

Across 12 models and three agent scaffolds, the leading model depends on whether performance means complete or partial recovery. Performance also declines with task difficulty, while results vary across languages.

5.1 Evaluation Setup

  • The evaluation covers 12 proprietary and open-weight model variants under Claude Code, Cline, and OpenCode, with up to 200 turns and 6,600 seconds per solver attempt.Overall and language results use P@1; the Claude Code task-scope analysis also reports P@4. Table 1 presents resolved rate and scope-weighted score by task scope.Source: §5.1 · pp. 5–6, Table 1 · p. 6

5.2 Main Results

  • 37.8% resolved rate and 51.3% weighted score for Claude Opus 4.8 under Claude Code contrast with GPT-5.5’s 59.7% weighted score and 33.7% resolved rate.The leading resolved-rate model varies by scaffold: Claude Opus 4.8 leads under Claude Code and OpenCode, while GPT-5.5 and GPT-5.4 tie under Cline at 31.2%.Source: Figure 4 · p. 6, §5.2 · p. 6

5.3 Performance by Task Scope

  • Across Claude Code models, P@1 resolved rates fall from 55.2% on Easy tasks to 20.4% on Medium and 7.4% on Hard; weighted scores fall from 69.3% to 50.0% and 28.1%.At P@4, Hard-task means rise to 10.2% resolved and 36.3% weighted, but most Hard tasks remain unresolved; the difficulty ordering persists for every model.Source: §5.3 · p. 7, Figure 5 · p. 7

5.4 Performance across Languages

  • Across languages and scaffolds, mean P@1 resolved rates range from 12.7% on JavaScript to 37.1% on Java, and weighted scores from 33.1% on Go to 52.6% on Java.Under Claude Code, Claude Opus 4.8 leads resolved rates on Go, Python, JavaScript, and TypeScript; GPT-5.5 leads on Java and Kotlin. Averaged across models, Claude Code leads four languages on both measures.Source: §5.4 · p. 7

6 Analysis

Scope-weighted scores reveal substantial partial recovery even when tasks remain unresolved, while resolved attempts generally use fewer tools and modify fewer files. A composition case shows how multiple test files can yield a larger, validated implementation scope.

6.1 Interpreting Partial Recovery

  • 128 of 141 FlowFuse scoring points pass for a 91.6% scope-weighted score, while Jackson’s Kotlin module passes 18 of 20 points for 90.4%; both patches remain unresolved.The score captures partial recovery even when the strict resolution criterion is not met.Source: §6.1 · p. 7

6.2 Solver Activity in Recorded Trajectories

  • Resolved attempts have lower median tool calls in 122 of 126 configurations and fewer modified files in 123, with median differences of −35.5 calls and −4 files.The comparisons cover 126 model, scaffold, and language configurations; task difficulty and stopping after completion may partly explain the association.Source: §6.2 · p. 7, Figure 6 · p. 8

6.3 Case Study: Task Composition

  • Seven test files expand the OpenAPI task scope from at most 40 functions in any individual footprint to 94, with all 39 expected scoring points passing originally and failing under the complete mask.Shared functions connect document assembly with routing and type conversion; every selected file is affected by the combined mask.Source: §6.3 · pp. 7–8

7 Conclusion

SWE-Nexus constructs repository-level tasks from test execution, validating candidate scopes before deterministic composition. Across its benchmark, the best configuration resolves 37.8% of tasks in one attempt, while scope-weighted scoring captures partial recovery.

7 Conclusion

  • SWE-Nexus validates candidate scopes against original and masked repositories, then rechecks every constituent test file after deterministic composition.Evaluation uses the same hidden tests and fixed protocol across submissions.Source: §7 · p. 8
  • 37.8% of tasks are resolved in one attempt by the best configuration across 12 models and three agent scaffolds.The benchmark contains 600 tasks from 105 repositories across six languages.Source: §7 · p. 8
  • Scope-weighted scoring captures partial recovery, and composing validated execution footprints extends tasks beyond individual historical changes.The score weights are fixed from observed test–function associations before solving.Source: §7 · p. 8

Limitations

SWE-Nexus relies on existing repository tests to identify and validate implementation scopes, so untested behaviors and edge cases may be underrepresented.

Limitations

  • Behaviors and edge cases that existing repository tests do not exercise may be underrepresented in the resulting tasks.Extending the approach to functionality with limited test coverage requires developing and validating additional tests.Source: Limitations · p. 8
  • The benchmark currently covers six programming languages.Broader coverage requires adapting tracing, source mapping, and result parsing to additional runtimes and test frameworks.Source: Limitations · p. 8
  • Functionality with limited test coverage requires additional tests to extend the approach.Those tests must be developed and validated.Source: Limitations · p. 8

Ethical Considerations

The paper notes licensing and isolation considerations for repository code, tests, and derived tasks. Its appendices describe tracing, masking, evaluation, specifications, task forms, and example cases.

Ethical Considerations

  • Repository code, tests, and derived tasks remain subject to original licenses; isolated environments are advised because tests and setup commands can have side effects.Benchmark results verify selected test outcomes but do not establish security or deployment readiness; the study involves no human participants.Source: Ethical Considerations · pp. 8–9

A Execution and Task Construction Details

  • Repository-specific tracing adapts to each test runner and source layout, while masking preserves declarations and signatures and independently checks that the patch applies correctly.Execution protocols specify setup, test runs, report collection, and parsing; file-level outcomes use the same scoring unit for validation and evaluation.Source: A Execution and Task Construction… · p. 11

B Task Specifications and Case Studies

B.1 Concrete Task Forms

  • An atomic task uses one selected test file and may mask multiple functions; a composite task combines validated scopes from multiple files, counting shared functions once.Tracing mechanisms are adapted to each repository’s test runner and source layout.Source: §B.1 · pp. 11–12, Table 3 · p. 11

B.2 Recorded Execution Cases

  • Recorded execution cases contrast a fully resolved task with a high-scoring patch that leaves several behaviors incomplete.The excerpts include visible tool actions and recorded results.Source: §B.2 · p. 12

B.3 Cases across Languages

  • First-attempt cases under Claude Code cover five languages and all three scope tiers, including full resolution, partial recovery, and zero recovery.The table reports selected-file count, unique functions and body spans, and scope-weighted score.Source: §B.3 · p. 12, Table 4 · p. 12

C Dataset Scope and Statistics

The release combines validated test-file scopes into repository-level tasks, with shared-function links and complete-mask revalidation. Its evaluation configuration and scope-weighted scoring define how complete and partial recovery are assessed.

C Dataset Scope and Statistics

  • 255 of 484 composite tasks have connected test-file graphs, while 229 have multiple disconnected components; every constituent file is revalidated under the complete mask.Tasks select between one and nine test files, with counts of 116, 56, 59, 61, 61, 67, 63, 55, and 62, respectively.Source: C Dataset Scope and Statistics · p. 12
  • Task outcomes may be statistically dependent because repositories and validated parent tests recur, so future splits should control both sources of reuse.The maximum parent-test reuse ranges from 8 to 24 tasks across the six language sets.Source: C Dataset Scope and Statistics · p. 13

D.1 Evaluation Configuration

  • All evaluated configurations share fixed execution limits, including per-request output limits and per-attempt solver deadlines.The evaluated Claude Code and OpenCode versions are 2.1.158 and 1.3.17, respectively.Source: §D.1 · p. 13, Table 8 · p. 14

D.2 Complete Results

  • Tables 9 and 10 report complete P@1 results, with resolved rate and scope-weighted score reported as paired percentages.The supplied passages do not give the table cell values.Source: §D.2 · p. 14, Table 9 · p. 14

E.1 Scope-Weighted Scoring

  • Equation 2 allocates each function’s recorded span equally among its associated scoring points; coarser attribution can associate a function with all points in its execution record.When individual outcomes are unavailable, a file supplies one point; scoring configurations are fixed before solving.Source: §E.1 · p. 14
  • Passing two of three points in the synthetic example yields an unweighted score of 2/3 but a scope-weighted score of 0.5, and the task remains unresolved.The example assigns point weights of 10, 40, and 50 based on associated function spans.Source: §E.1 · p. 14
  • 128 of 141 scoring points with a scope-weighted score of 0.915988, and 18 of 20 with a score of 0.904475, still left both recorded patches unresolved.The examples show that partial recovery is distinct from meeting the complete-resolution criterion.Source: §E.1 · p. 14

E.2 Trajectory Analysis

Trajectory analysis compares solver activity between resolved and unresolved attempts across retained model, scaffold, and language configurations. The paper also illustrates composite tasks that combine complementary constraints into coherent features across programming languages.

E.2 Trajectory Analysis

  • Resolved attempts have median differences of +4.39 percentage points in tool calls and +2.11 percentage points in modified file counts versus unresolved attempts.The retained configurations have at least five attempts of each outcome.Source: §E.2 · p. 14

F Task Examples across Six Languages

  • The six language examples combine complementary constraints into coherent software features while preserving functional goals, boundary cases, and errors.The example specifications’ wording was edited for presentation.Source: F Task Examples across Six Languages · p. 14

Knative Kafka Broker: reconciliation and finalization

Django REST framework: requests, representations, and versioned URLs

Shepherd: rendering and running interactive tour steps

Element Plus: tab navigation, editing, and reactive updates

S3Mock: object request handling and multipart integrity

Related papers

Loading 2610.20038v2…