Source-linked AI summary
WEAVE: Weaving Real-World Code Evolution into Dependency-Structured Tasks for Benchmarking and Training Code Agents
Weihao Li, Zihang Xu, Zhongqiu Wang, Songwei Yu, Wenqiang Zhu, Yuechen Luo, Jingxuan Xu, Yanan Wu, Haoyang Huang, Ken Deng
TL;DR
Most executable repository benchmarks use a single issue or PR as the task unit, while real engineering work can span dependent changes and existing multi-PR settings often use coarse aggregation. WEAVE builds executable tasks from verified PR histories using repository-calibrated typed change loci and dependency DAGs, with Graph Credit for dependency-consistent partial progress. Experiments report coherent tasks, more faithful progress measurement, and effective supervision and verifiable rewards for training code agents.
Problem
Most executable repository benchmarks retain a single issue or PR as the task unit, while multi-PR settings rely on coarse aggregation and structure-agnostic evaluation that does not capture dependencies among changes.
Source: §1 · pp. 1–2Method
WEAVE uses repository-calibrated typed change loci to build bounded dependency DAGs from verified PR histories, instantiate unified executable tasks, and measure dependency-consistent partial progress with Graph Credit.
Source: §1 · p. 2Results
Experiments report more coherent task groupings, more faithful progress measurement than flat scoring, and effective supervision and verifiable rewards for training code agents.
Source: §1 · p. 2Takeaways & Limitations
WEAVE provides a framework for executable dependency-structured tasks that supports evaluation and training over realistic multi-PR software evolution.
Source: §6 · pp. 7–8
Abstract
from arXiv · showhide
Real-world software engineering rarely consists of isolated changes. Features, migrations, and systematic fixes often unfold across multiple pull requests (PRs), requiring coordinated modifications across a repository. Yet most executable repository-level benchmarks retain a single issue or PR as the task unit, while existing multi-PR settings rely on coarse aggregation and structure-agnostic evaluation. This mismatch obscures dependencies among changes and limits progress measurement and learning signals for long-horizon code agents. We introduce WEAVE, a framework that transforms verified PR histories into executable, dependency-structured tasks. WEAVE leverages repository-calibrated typed change loci to organize related PRs into bounded DAGs and instantiate each as a unified executable task. We further introduce Graph Credit, a deterministic, test-grounded metric that measures dependency-consistent partial progress and supports dense, verifiable rewards for reinforcement learning. Experiments on task construction, agent evaluation, supervised fine-tuning, and reinforcement learning show that WEAVE yields coherent executable tasks, enables more faithful progress measurement than flat scoring, and provides effective supervision and verifiable rewards for training code agents.
1 Introduction
Most executable repository benchmarks treat one issue or PR as a task, although real engineering objectives often span dependent PRs. WEAVE builds executable tasks from verified PR histories and uses dependency structure for evaluation and training.
1 Introduction
- Most existing executable benchmarks use a single issue or PR as the task unit, while real features, migrations, upgrades, and fixes may span sequential, parallel, or converging PRs.The missing abstraction is the dependency structure connecting changes into a coherent engineering objective.Source: §1 · p. 1
- Aggregation proxies can merge unrelated changes or miss dependencies, and related change sets alone do not provide a reproducible task with a valid base state, specification, and executable oracle.Flat scoring may overlook dependencies or over-credit downstream behavior built on incomplete upstream changes; binary rewards are sparse, while patch-similarity and learned rewards may not align with executable semantics.Source: §1 · pp. 1–2
- WEAVE organizes verified PRs into bounded dependency DAGs using repository-calibrated typed change loci, then instantiates each graph as a unified executable task.The graph remains evaluator-side metadata unavailable to the agent.Source: §1 · p. 2
- WEAVE’s four-part study finds more coherent task groupings, more faithful progress measurement, and effective supervision and verifiable rewards for training code agents.The study covers task construction, agent evaluation, supervised fine-tuning, and reinforcement learning.Source: §1 · p. 2
- Graph Credit measures dependency-consistent partial completion with deterministic, test-grounded credit, complementing binary resolution and providing dense, verifiable reinforcement-learning rewards.The framework also uses its constructed corpus for long-horizon supervised fine-tuning trajectories.Source: §1 · p. 2
2 Related Work
Prior executable benchmarks largely retain individual issues or PRs as their task unit, while newer multi-PR settings aggregate broader changes. WEAVE combines executable PR-DAG tasks with graph-conditioned partial credit.
2.1 Single-PR Software Engineering Benchmarks
- Single-PR benchmarks pair an issue with a repository state and executable oracle, and later work scales this formulation for evaluation and training while generally retaining an individual issue, commit, or PR as the task unit.Table 1 compares representative benchmarks by task unit, construction, retained structure, evaluation granularity, and training support.Source: §2.1 · p. 2, §2 · p. 2
2.2 Multi-PR Software Engineering Benchmarks
- Multi-PR benchmarks aggregate changes across releases, features, roadmaps, temporal spans, release chains, milestone DAGs, or explicitly referenced PR chains, often using flat or stage-level progress measures.Related work also explores denser rewards, enhanced test oracles, and graph structures for agent workflows or program analysis.Source: §2.2 · p. 3
- WEAVE complements these directions by constructing executable PR-DAG tasks from repository-calibrated code coupling and deriving graph-conditioned partial credit from node-attributed tests.The resulting signal can also serve as a dense, verifiable outcome signal during optimization.Source: §2.2 · p. 3
3 Dependency-Structured Task Construction
WEAVE turns validated atomic PRs into executable tasks whose dependency DAGs support graph-conditioned credit and reinforcement-learning methods. It constructs those DAGs from repository-calibrated typed-locus coupling, then builds a unified task state, specification, and test oracle for each retained graph.
3.1 Overview and Task Definition
- WEAVE represents each PR through modified program entities, aggregates strongly coupled PRs into bounded dependency DAGs, and instantiates each graph as a task with a unified specification, executable tests, and node-level test attribution.The graph is retained as evaluator-side metadata for later graph-conditioned credit and reinforcement learning.Source: §3.1 · p. 3
3.2 Repository-Calibrated DAG Construction
- Typed change loci capture AST declarations, lightweight symbols, structured configuration or schema entries, and dependency keys rather than treating touched files as the unit of overlap.Type-specific extraction avoids treating configuration files, manifests, and lockfiles as single shared locations; test, generated, vendored, and binary changes are excluded from aggregation.Source: §3.2 · p. 4
- Repository-local inverse-frequency weighting emphasizes specific, infrequently modified loci and suppresses frequently modified hubs when scoring shared evidence between PRs.The coupling score is used to form candidate relations within a repository.Source: §3.2 · p. 4
- Candidate edges are limited by a search radius and coupling threshold; merge time constrains search and orients edges but does not itself indicate coupling.Repeated or reversed changes are removed, and oversized components are reduced to high-scoring connected subgraphs while retaining surviving edges among selected nodes.Source: §3.2 · p. 4
3.3 Executable Task Instantiation
- For each retained graph, WEAVE reverts constituent PR changes to create a counterfactual base state, consolidates objectives into one specification, and requires a combined executable oracle that preserves required regression behavior.Candidates are discarded if repository reconstruction, gold-patch application, or executable validation fails.Source: §3.3 · p. 4
- Valid F2P tests are attributed to PR nodes when high-confidence attribution is possible; retained instances require adequate attribution coverage and exclude recoverable target commits or repository history.Train and benchmark splits enforce temporal separation and remove overlapping target PRs and duplicate patches; WEAVE-Bench spans language coverage, task scales, dependency structures, and patch scopes.Source: §3.3 · p. 4
4 Graph Credit
Flat test scoring captures partial progress but ignores dependencies, allowing downstream passes to receive unsupported credit. Graph Credit conditions node completion on upstream prerequisites and supplies a structural signal for evaluation and reinforcement learning.
4 Graph Credit
- Flat test scoring can reward downstream tests even when an upstream prerequisite remains unresolved, creating structurally unsupported phantom credit.Binary resolution, by contrast, cannot distinguish progress among unsolved tasks.Source: §4 · pp. 4–5
4.1 Dependency-Conditioned Scoring
- Graph Credit recursively caps each node’s credit by its own completion and the least-complete upstream prerequisite, then averages effective credit over evaluable nodes.Ambiguous and unattributed tests remain in the full-task oracle but are excluded from node-level scoring.Source: §4.1 · p. 5
4.2 Evaluation and Agent Optimization
- Graph Credit complements full-task resolution and Flat-F2P by diagnosing dependency-consistent partial progress on unsolved tasks.The evaluation also examines alternative graph operators, linearized scoring, and randomized-graph controls.Source: Evaluation Protocol · p. 5
- RWEAVE combines flat functional progress, dependency-conditioned credit, full-task resolution, and regression preservation into a reward computed from executable test outcomes.Graph Credit is a structural reward component, not a standalone objective; the resulting feedback is dense and verifiable.Source: Reinforcement-Learning Reward · p. 5
5 Experiments
The experiments assess task construction, agent evaluation, supervised fine-tuning, and reinforcement learning under temporally separated train and benchmark splits. Across these studies, WEAVE’s dependency structure supports stronger construction results, more informative progress scoring, and improved training outcomes.
5 Experiments
- The study evaluates four questions across construction, agent performance, SFT utility, and RL reward, using temporally separated training and held-out benchmark splits.The construction study compares typed-locus coupling with temporal, traceability, file-overlap, and symbol-overlap baselines.Source: §5 · p. 5, §5.1 · p. 5
5.1 RQ1: Construction Validity
- WEAVE achieves 37.7% AP, 11.1 points above File Overlap, and 84.5% Coverage, outperforming all four single-signal baselines.Its P@50 is 38% versus File Overlap’s 10%; repository-calibrated IDF weighting suppresses hub-file false edges, while multi-channel evidence recovers dependencies missed by single signals.Source: §5.1 · pp. 5–6, Table 3 · p. 6
5.2 RQ2: Benchmark Performance
- Across 16 agent configurations, Flat-F2P exceeds Graph Credit by 10.5 pp on average, revealing substantial phantom credit in dependency-structured tasks.The gap ranges from 7.2–16.8 pp, and node-permuted random graphs reduce it to under 2 pp.Source: §5.2 · p. 6, Table 2 · p. 6
5.3 RQ3: SFT Utility and Token Efficiency
- WEAVE SFT improves WEAVE-Bench Resolved over Atomic-PR by 3.0/3.7 pp at 4B/35B and transfers gains to single-PR benchmarks.On WEAVE-Bench, Graph Credit gains are 5.4/6.2 pp; SWE-bench Verified improves by 3.6/2.8 pp over Atomic-PR, while external corpora yield smaller or inconsistent gains.Source: §5.3 · p. 7, Figure 4 · p. 7
5.4 RQ4: Graph-Conditioned Reinforcement Learning
- RWEAVE improves Resolved over SFT by 3.7/4.3 pp and over flat F2P by 2.0/2.6 pp on Qwen3.5-4B/35B-A3B, respectively.Graph Credit gains over SFT are 5.6/6.2 pp, and SWE-bench Verified gains are 3.2/4.2 pp; all settings keep P2P regression below 2.3%.Source: §5.4 · p. 7, Figure 5 · p. 7
6 Conclusion
WEAVE represents verified PR histories as dependency DAGs and uses that graph representation to unify executable task construction, partial-credit evaluation, and reward design. The appendix details how loci are extracted, DAGs assembled, and executable base states and test attributions constructed.
6 Conclusion
- WEAVE uses dependency DAGs to unify executable task construction, dependency-conditioned partial credit, and structured reward design for long-horizon code agents.The conclusion presents WEAVE as both an evaluation workbench and a substrate for improving code agents.Source: §6 · pp. 7–8
A.1 Typed Change Loci Extraction
- Each PR is represented by typed change loci, including AST declarations, symbols, configuration or schema entries, declarations or signatures, and dependency keys.The representation targets concrete program entities and structured entries rather than files alone.Source: §A.1 · p. 10
A.2 Edge Filtering and DAG Assembly
- DAG assembly filters mechanical changes, partitions connected components into bounded tasks, orients edges by merge time, and removes structurally uninformative graphs.Tasks contain at most K=8 PR nodes; star graphs and weak, trivially serial chains are filtered.Source: §A.2 · pp. 11–12
- At K=8, benchmark tasks range from 2 to 8 PR nodes, with mean 4.91 and median 5.The task set spans linear chains and branching dependency structures.Source: §A.2 · p. 11
A.3 Rollback Construction and Test Attribution
- About 35% of candidate DAGs are discarded when neither rollback nor agent-assisted carving yields a valid, test-passing base state.Among successfully constructed instances, 73% use clean git revert and 27% require agent-assisted carve.Source: §A.3 · p. 12
- Test attribution uses isolated reverts to link failing tests to individual PR nodes, marking tests ambiguous or unattributed when evidence does not identify a unique node.Agent-based attribution supplements the procedure when isolated reverts are infeasible.Source: §A.3 · p. 12
B Graph Credit Analysis
- The appendix examines how the dependency graph interacts with scoring operators and task topologies in its extended Graph Credit analysis.Source: B Graph Credit Analysis · p. 13
B.1 Alternative Graph Operators
The appendix compares Graph Credit operators and tests whether its diagnostic signal depends on the true dependency graph. It also clarifies depth conventions and documents the experimental computing setup.
B.1 Alternative Graph Operators
- The min-bottleneck operator has the highest correlation with task resolution and eliminates phantom credit, supporting its conservative dependency-gating design.Mean-parent scoring leaves 3.2 pp residual phantom credit, while depth weighting penalizes deeper nodes regardless of upstream completion.Source: §B.1 · p. 13, Table 5 · p. 13
B.2 Randomized-Graph Controls
- With the true DAG, mean OverCredit is 10.5 pp; under randomized graphs it falls to 1.6 pp, indicating the true-graph gap reflects dependency violations rather than arbitrary constraints.Across permutations, the standard deviation is 0.7 pp, and 89% of tasks are within 2 pp of zero.Source: §B.2 · p. 14
B.3 Chain-Depth Interpretation
- Graph Credit’s recursive minimum caps downstream completion at each edge, creating more opportunities for unresolved ancestors to cap local test passes on longer chains.The passage cautions that this structural property alone does not establish an empirical relationship between chain depth and OverCredit.Source: §B.3 · p. 14
C Experimental Setup
- The appendix identifies computing environment, agent evaluation configuration, and construction-hyperparameter sensitivity as its experimental setup topics.Source: C Experimental Setup · p. 14
C.1 Computing Infrastructure
- Agent evaluations run in isolated Docker containers, while training uses distributed configurations; evaluation models use official provider endpoints.The stated software stack includes Python 3.12 and PyTorch 2.x, with BF16 optimization for training.Source: §C.1 · p. 14
C.2 Evaluation Configuration
The evaluation configuration keeps the two scaffolds’ underlying capabilities aligned while preserving their different interaction formats. Sensitivity analyses select a coupling threshold and temporal search radius based on construction quality and coverage.
C.2 Evaluation Configuration
- Agents face no per-turn timeout, file-read limit, or patch-size restriction, and provider-default context windows and sampling temperatures are retained.Source: §C.2 · p. 14
- OpenCode and Cline expose the same underlying capabilities but differ in system prompts and tool-invocation formats, enabling comparisons of scaffold-invariant and scaffold-specific behavior.OpenCode uses structured XML actions with explicit planning; Cline uses conversational interaction with implicit planning.Source: §C.2 · p. 14
C.3 Threshold and Hyperparameter Sensitivity
- At θ=0.20, AP stays within 0.4 pp of its peak at θ=0.15 while executable construction rate is maximal, making it the selected precision–recall tradeoff.Lower thresholds add spurious edges and rollback conflicts; higher thresholds miss genuine dependencies.Source: §C.3 · p. 15, Figure 6 · p. 15
- The selected search radius is 90 days: coverage saturates by 60 days and gains only 0.6 pp from 60 to 90 days.The corpus median inter-PR interval is about 2–3 weeks; 90 days is described as a conservative upper bound for long-lived feature arcs.Source: §C.3 · p. 15, Table 8 · p. 15
D Additional Experimental Results
- The extended results section organizes findings across four research questions, with the dependency graph connecting construction, agent evaluation, and training results.Source: D Additional Experimental Results · p. 15
D.1 RQ1 Extended Results
Extended RQ1 results show that WEAVE leads in precision across ranking cutoffs, while repository-local weighting helps align pooled and per-repository evaluations. Ablations find that declaration loci are especially informative and that the full combination of locus types performs best.
D.1 RQ1 Extended Results
- WEAVE attains the highest precision at every cutoff, while no baseline is consistently second: Traceability Links is strongest at K ≤20, with 12.8% coverage, but falls below Symbol Overlap by K =50.Source: §D.1 · p. 15
- WEAVE’s repository-local weighting yields similar per-repository AP and pooled P@50 scores: 37.7% and 38.0%, respectively.The two metrics differ because pooled P@K additionally rewards cross-repository score calibration; File Overlap has AP 26.6% but P@50 10.0%.Source: §D.1 · pp. 15–16
- WEAVE’s median AP is 36.2% (IQR: 28.4–47.1%), versus File Overlap’s 24.8% (IQR: 18.2–35.6%).WEAVE has AP above 50% in 6 repositories and below 20% in 3 repositories, each with fewer than 5 gold edges.Source: §D.1 · p. 16
- Last is the strongest single locus type at AP 28.4%, and removing it causes the largest drop (−6.5 pp); no single type matches the full combination.Configuration and dependency loci add coverage of relationships not captured by code-structural analysis alone.Source: §D.1 · p. 16, Table 10 · p. 16
- Last alone achieves AP 28.4%, exceeding File Overlap’s 26.6%; the like-for-like comparison with Symbol Overlap is 28.4% versus 24.0%.Symbol Overlap spans both Last and Lsym, while Lsym alone covers exported symbols, type aliases, constants, and enum members.Source: §D.1 · p. 16
- WEAVE reaches the Graph Credit achieved by Atomic-PR at 100% of its token budget using 25%, indicating at least ∼4× token efficiency.The token-scaling curves differ in slope rather than offset, indicating denser supervision per token rather than merely better initialization.Source: Figure 7 · p. 16, §D.1 · p. 16
D.2 RQ3 Full Numerical Results and Token Scaling
The section reports token-budget scaling for three data-construction strategies and identifies a substantial token-efficiency advantage for WEAVE at the tested operating point.
D.2 RQ3 Full Numerical Results and Token Scaling
- The token-scaling study trains Qwen3.5-4B at 25%, 50%, 75%, and 100% of a ∼158 M-token budget and reports WB-GC scores.Table 12 reports the scores and Figure 7 plots the curves.Source: §D.2 · p. 16, Table 12 · p. 17
- At 25% budget, WEAVE reaches 12.6% WB-GC, compared with Atomic-PR’s 12.4% at 100% budget, establishing at least ∼4× token efficiency at this operating point.The analysis does not evaluate budgets below 25%, so it does not establish an exact efficiency ratio.Source: §D.2 · p. 16
- Per budget doubling, WB-GC rises by ∼2.6 pp for WEAVE, ∼1.8 pp for Temporal-Window, and ∼1.6 pp for Atomic-PR, with no strategy saturating in the tested range.The approximately log-linear curves separate by slope rather than offset.Source: §D.2 · p. 16
D.3 RQ4 Component Ablation and Curriculum
The reward ablation identifies Graph Credit as the most impactful individual component, while the curriculum yields modest gains concentrated early in training.
D.3 RQ4 Component Ablation and Curriculum
- Removing Graph Credit reduces WB-GC by 4.4 pp and WB-Res by 1.7 pp; it is the most impactful individual reward-component removal.Table 13 compares progressive component removals from the same WEAVE-SFT checkpoint with matched rollout budgets and renormalized remaining coefficients.Source: §D.3 · p. 17, Table 13 · p. 17
- Without RegP2P, P2P regressions rise from 1.8% across the other four reward variants to 4.7%, despite minimal impact on performance metrics.The no-penalty ablation is the only compared setting above the main-text 2.3% regression ceiling.Source: §D.3 · p. 17
- Binary Resolved yields at most 0.7 pp improvement over the SFT checkpoint because most rollouts receive identical zero reward, while Flat F2P densifies the signal but credits downstream passes regardless of prerequisite state.Removing Graph Credit from the full reward still outperforms Flat F2P alone, indicating that the remaining reward components contribute independently.Source: §D.3 · p. 17
- The three-phase DAG-complexity curriculum improves metrics by ∼1.0 pp, primarily during the first 30% of training, as it prevents early divergence on overly complex tasks.The schedule progresses from depth ≤2 and |V| ≤3 to depth ≤3 and |V| ≤5, then all tasks; mean patch size rises from 166 to 289 lines, a ∼1.7× increase.Source: §D.3 · pp. 17–18, Table 14 · p. 18
E Training Details
WEAVE uses dependency DAGs across trajectory generation, supervised fine-tuning, and reinforcement learning, then reports dataset statistics for its training and benchmark splits.
E Training Details
- The same PR dependency DAG defines evaluation tasks and Graph Credit scoring and structures the SFT trajectory and RL reward pipeline.This connects task construction through agent optimization within WEAVE’s training framework.Source: E Training Details · p. 18
E.1 Trajectory Generation and Quality Filtering
- GLM-5.2 is selected as an openly weighted, self-hostable generator; best-of-3 attempts help offset its 5.7 pp WEAVE-Bench Resolved gap from Claude Opus 4.7.GLM-5.2 incurs 3.6× generation overhead, motivating the three-attempt cap.Source: §E.1 · p. 18
- Filtering retains 2,304 trajectories covering 72.0% of WEAVE-Train tasks; coverage is 74.1% for depth ≤2 and 69.4% for depth ≥4.The five filters require dependency-consistent success, adequate length, no P2P regressions, no degenerate patterns, and no recovered target changes; Graph Credit gates success.Source: §E.1 · pp. 18–19
E.2 SFT Configuration
- SFT compares data-construction strategies with matched token budgets and identical training configurations; runs use a single fixed random seed and are single-run unless stated otherwise.The loss is masked to assistant-generated tokens, and the 131,072-token sequence window matches the generator context budget.Source: §E.2 · p. 19
E.3 RL Configuration
- RL uses a cumulative three-phase curriculum over 3,200 tasks, expanding eligibility from 1,048 tasks in Phase 1 to 1,864 in Phase 2 and all tasks in Phase 3.Phases progress from depth ≤2 and |V| ≤3, to depth ≤3 and |V| ≤5, then uniform sampling of all tasks.Source: §E.3 · pp. 19–20
- GRPO samples eight rollouts per task and normalizes each reward against its group; groups with identical rewards provide no gradient, making reward density consequential.Graph Credit’s continuous, dependency-conditioned signal almost always produces within-group spread.Source: §E.3 · p. 20
F Dataset Statistics and Case Studies
- The dataset section presents WEAVE-Bench and WEAVE-Train statistics and case studies illustrating Graph Credit on concrete dependency structures.The supplied text identifies language, DAG structure, node-count, patch, and test statistics without giving their full table values.Source: F Dataset Statistics and Case Studies · p. 20, §F.1 · p. 20
F.1 WEAVE-Bench and WEAVE-Train Statistics
- WEAVE-Bench DAGs comprise 44.3% linear chains, 15.7% diamonds, 15.0% general DAGs, 13.3% fan structures, and 11.7% cross structures; it spans 52 repositories.The largest repository contributes 23 tasks (7.7%), while the top 12 contribute ∼60%; WEAVE-Train covers ∼120 repositories.Source: §F.1 · p. 20
F.2 Qualitative Examples
The examples show how dependency-conditioned scoring discounts test passes unsupported by upstream work, including both a complete prerequisite failure and partial progress across branches.
F.2 Qualitative Examples
- In the four-node migration case, mocked downstream tests let the agent pass Nodes C and D while leaving migration and ORM prerequisites unresolved; Flat-F2P gives 58.3% credit, but Graph Credit gives 0.0%.The graph is a chain from database migration through ORM model and API serializer to frontend; Figure 8 marks completed and unresolved nodes.Source: §F.2 · pp. 21–22, Figure 8 · p. 21
- In the three-node fork, Graph Credit is 55.6% versus Flat-F2P’s 61.1%; it caps the primary-consumer branch at its utility prerequisite’s 2/3 credit while leaving the secondary branch’s lower local score binding.The resulting 5.6 pp OverCredit comes entirely from the primary branch; the secondary consumer passes 2 of 6 tests.Source: Figure 8 · p. 21, §F.2 · p. 22
- 68% of runs exhibit >0 pp OverCredit, 23% exceed 15 pp, and Graph Credit correlates more strongly with actual Resolved than Flat-F2P (ρ = 0.82 vs. 0.69).The correlation is a 19% improvement in diagnostic fidelity.Source: §F.2 · p. 22
Related papers
This paper cites
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?2023
- Training Software Engineering Agents and Verifiers with SWE-Gym2024
- SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale2026
- FeatureBench: Benchmarking Agentic Coding for Complex Feature Development2026
- SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration2026