PaperlayerView exact revision on arXiv

Source-linked AI summary

Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks

Mengyu Zheng, Kai Han, Boxun Li, Haiyang Xu, Yuchuan Tian, Wei He, Hang Zhou, Jianyuan Guo, Hailin Hu, Lin Ma, Chao Xu, Guohao Dai, Lixue Xia, Yunchao Wei, Yunhe Wang, Yu Wang

arXiv:2606.12344v1cs.LGcs.CL

TL;DR

Repository-level coding ability of general-purpose agents remains difficult to measure comparably. Claw-SWE-Bench introduces a multilingual benchmark and adapter protocol that controls the evaluation stack while varying the harness. Harness choice changes Pass@1 by up to 27.4 percentage points under a fixed model, and can reorder system rankings.

  • Problem

    Evidence on whether general-purpose agents can solve repository-level software-engineering tasks remains limited, while existing evaluations conflate model, harness, and task effects.

  • Method

    Claw-SWE-Bench combines an adapter with a shared orchestrator that fixes the task, runtime, budget, patch, prediction, and evaluation protocols while varying the harness.

  • Results

    27.4 pp is the claw spread on Qwen 3.6-flash under a fixed model, large enough to reorder leaderboard conclusions when the harness is unspecified.

  • Takeaways & Limitations

    Harness choice is a first-order variable in SWE-style coding-agent evaluation, so comparable results require explicit reporting of harness, budget, cost, and cache accounting.

  • Takeaways & Limitations

    Experiments use single-run aggregates and only five claws with two representative models, limiting conclusions about stable superiority and harness–model interactions.

Abstract

from arXiv · show

General-purpose agents such as OpenClaw are increasingly used as autonomous tool users, but their coding ability is difficult to measure under SWE-bench: a generic agent does not by itself satisfy the clean Docker workspace, patch, and prediction contract required for scoring. We introduce Claw-SWE-Bench, a multilingual SWE-bench-style benchmark and adapter protocol that makes heterogeneous agent harnesses, or claws, comparable under fair settings including a fixed prompt, runtime budget, workspace contract, patch extraction procedure, and evaluator. The full benchmark contains 350 GitHub issue-resolution instances across 8 languages and 43 repositories, drawn from SWE-bench-Multilingual and SWE-bench-Verified-Mini after future-commit cleanup. We also release Claw-SWE-Bench Lite for faster validation, which is an 80-instance subset selected by a cost-aware, rank-aware procedure over 17 calibration columns. On the full benchmark, OpenClaw with a minimal direct-diff adapter scores only $19.1\%$ Pass@1, whereas the full adapter reaches $73.4\%$ with the same GLM 5.1 backbone, showing that adapter design is essential for enabling OpenClaw-style harnesses to perform coding tasks effectively. Across an OpenClaw $\times$ nine-model sweep and a five-claw $\times$ two-model sweep, model choice changes Pass@1 by $29.4$ pp and harness choice by $27.4$ pp under fixed models; systems with similar accuracy can differ substantially in total API cost. Claw-SWE-Bench therefore treats harness and cost accounting as first-class axes of SWE-style coding-agent evaluation, providing both a full benchmark and a low-cost reference set for reproducible comparison. The data is available at https://github.com/opensquilla/claw-swe-bench and https://huggingface.co/datasets/TokenRhythm/Claw-SWE-Bench.

1. Introduction

Claw-SWE-Bench addresses the difficulty of evaluating OpenClaw-style coding agents by isolating harness effects under a fixed SWE-bench-style protocol and treating evaluation cost as a first-class metric. It also provides an 80-instance Lite subset for lower-cost iteration and comparable validation.

  • Motivation: Prior SWE-bench-style evaluations conflate harness design with prompts, scaffolds, budgets, termination policies, models, and task sets, making cross-system differences difficult to attribute.The introduction identifies this as a gap in measuring repository-level coding ability of general-purpose agents.
  • Cost accounting: Accuracy and cost do not move in lockstep across the full 350-instance five-claw × two-model sweep, motivating cost-aware benchmark reporting.Figure 1 plots Pass@1 against full-run total API cost and marks the non-dominated Pareto frontier.
  • Benchmark design: Claw-SWE-Bench isolates harness effects by fixing the prompt, task set, container, timeout, patch extraction, and evaluator while exposing a replaceable harness slot.Harnesses use a shared adapter protocol with defined lifecycle methods.
  • Lite subset: 80 instances comprise Claw-SWE-Bench Lite, a low-cost subset designed to preserve the full set’s scale, language distribution, key rankings, and cost structure under limited budget.Lite supports shorter feedback loops for model replacement and harness iteration.
  • Empirical motivation: 27.4 pp is the harness-choice spread on Qwen 3.6-flash under fixed-model evaluation, compared with 12.5 pp on GLM 5.1.The study fixes two representative models and evaluates five claws.

2. Claw-SWE-Bench

Claw-SWE-Bench adapts general-purpose and specialized coding harnesses to SWE-bench’s patch-based contract through a shared adapter and standardized execution protocol. It combines 350 multilingual real-world issue-resolution tasks with fixed evaluation conditions for fair, comparable harness assessment.

  • Contract mismatch: Native OpenClaw cannot directly enter SWE-bench because the evaluator reads only the model_patch field and general-agent artifacts can contaminate git diff.The adapter constrains behavior to concrete repository edits and converts the final repository state into a scorable patch.
  • Adapter protocol: The adapter converts heterogeneous harness interactions into repository edits and evaluator-readable patch predictions.Candidate patches are collected from repository state rather than parsed from final messages, making output independent of a harness’s native response format.
  • Evaluation protocol: The shared orchestrator fixes the task set, repository state, prompt, Docker runtime, outer budget, patch extraction, prediction format, and downstream evaluation.These controls standardize execution while leaving each harness’s internal agent loop and design choices unconstrained.
  • Benchmark composition: 350 real GitHub issue-resolution tasks span 8 programming languages and 43 repositories in the full benchmark.The workload combines 300 multilingual instances with 50 human-validated Python instances.
  • Runtime and workspace: Future commits are removed from multilingual task containers so agents cannot inspect fixes beyond the issue’s base_commit history boundary.All tasks run in their corresponding SWE-bench Docker images with repositories reset to base_commit and mounted at /testbed.

3. Claw-SWE-Bench Lite

Claw-SWE-Bench Lite is an 80-instance, low-cost companion to the 350-instance benchmark, designed to accelerate iteration while preserving multilingual difficulty balance and evaluation comparability. Its selection procedure matches full-benchmark behavior across models and claws, with validation showing close Pass@1 parity and roughly one-quarter cost.

  • Motivation: Lite-80 provides a low-cost feedback loop for adapter debugging, prompt changes, model replacement, and regression testing when repeated full-350 runs become a bottleneck.The full benchmark requires substantial token usage, API cost, wall-clock time, and log inspection effort.
  • Construction: Lite-80 selects 10 instances from each of 8 languages, using a fixed 2/3/3/2 quota across difficulty quartiles within every language.The subset contains 70 non-Python instances from SWE-bench-Multilingual and 10 Python instances from SWE-bench-Verified-Mini.
  • Selection objective: The binary selection objective minimizes resolve-rate differences, penalizes pairwise ranking reversals, and enforces cost parity over 17 calibration columns spanning model and claw variation.Difficulty quartiles use mean resolved rate over the calibration pool; ranking penalties use RANK_EPS = 0.03 and λ = 1.0.
  • Validation: 0.639 versus 0.643 mean Pass@1 on full-350 and Lite-80, respectively, yields a difference of about +0.4 pp across the 17 calibration columns.Per-language deviations are within 1 pp for Go, JS/TS, PHP, and Python, while C/C++ and Ruby deviate by +2.94 pp and +2.65 pp.
  • Validation and release point: 22.9% of a full run is Lite-80’s approximate cost, while cross-claw validation shows a 1.88 pp mean absolute Lite–full difference and a 3.68 pp maximum.The K-sweep identifies K*∈[8, 10], and the released K*=10 point passes all resolve, cost, and operational gates.

4. Experimental Setup

The experiments isolate variation from language models and agent harnesses under standardized conditions, using complementary model and claw sweeps, accuracy and efficiency metrics, held-out Lite validation, and an adapter diagnostic.

  • Experimental grids: Two complementary grids fix either a reference claw while sweeping models or two models while sweeping claws, then test whether Lite preserves full-set trends.The study does not perform an exhaustive claw × model grid over all 350 instances.
  • Claws: Five claws share the same task prompt, SWE-bench Docker workspace, and outer budget.The evaluated claws are OPENCLAW, HERMES-AGENT, ZEROCLAW, NANOBOT, and GENERICAGENT.
  • Models: The model sweep evaluates nine LLMs with OPENCLAW, while the claw sweep uses GLM 5.1 and QWEN 3.6-FLASH as representative models spanning capability and cost.The nine-model sweep includes GPT 5.5, Claude Opus 4.7, GLM 5.1, DeepSeek-V4 Pro, DeepSeek-V4 Flash, Kimi 2.6, Qwen 3.6-flash, MiniMax M2.7, and Seed 2.0-mini.
  • Evaluation metrics: PASS@1 is the primary accuracy metric, supplemented by total USD cost, mean wall-clock duration, and cache hit rate diagnostics.Total cost covers the full 350-instance run, while cache hit rate reflects provider policy, adapter call paths, and context reuse rather than coding capability.
  • Lite held-out validation: OPENSQUILLA provides held-out validation by comparing PASS@1 on Lite-80 and full-350 under the same adapter protocol, budget, and evaluator.OPENSQUILLA is not used to construct or calibrate the Lite subset; approximation quality is measured by the percentage-point gap.
  • Adapter diagnostic: The bare-vs-full adapter diagnostic uses GLM 5.1 on the same full-350 workload to quantify the effect of workspace preparation, shared prompting, Git patch extraction, and patch cleaning.The bare adapter provides minimal Docker access and asks for a unified diff directly, whereas the full adapter uses the complete protocol.

5. Results

Results show that adapter design, model choice, and harness choice substantially affect coding-agent performance, while cost and cache hit rate provide essential context for interpreting Pass@1. The benchmark’s cleanup protocol also prevents future-commit visibility from inflating reported results.

  • Adapter effect: 73.4% Pass@1 with the full adapter versus 19.1% resolved rate for the bare adapter shows that direct unified-diff generation is an unreliable scoring interface.The full adapter lets the model edit repository files and exports the patch from Git state, whereas the bare adapter requires final-response diff text.
  • Model axis: 29.4 pp Pass@1 separates models under the same OpenClaw scaffold, from 78.0% for GPT 5.5 to 48.6% for Seed 2.0-mini.Claude Opus 4.7 reaches 77.1% (270/350), while GPT 5.5 reaches 78.0% (273/350).
  • Cost–accuracy: $8.2 is DeepSeek-V4 Flash’s total cost at 70.3% Pass@1, while GPT 5.5 costs $1399.1 at the highest Pass@1 of 78.0%.DeepSeek-V4 Pro reaches 71.7% Pass@1 at $81.3, and GLM 5.1 reaches 73.4% at $277.0 under cache-fixed accounting.
  • Future-commit cleanup: 84.7% →76.7% is Claude Opus 4.7’s largest cleanup drop, while no model’s Pass@1 increases after future-commit cleanup.Kimi 2.6 drops 5.0 pp and Qwen 3.6-flash drops 2.0 pp; cleanup results are used as the main accounting basis.
  • Harness axis: 27.4 pp is the Pass@1 spread across claws with Qwen 3.6-flash, exceeding the 12.5 pp spread observed with GLM 5.1.Changing only the harness-specific loop, tools, workspace management, and stopping policy can materially change performance.
  • Cost accounting: 96.5%, 91.3%, and 90.4% are the GLM 5.1 cache hit rates for OpenClaw, hermes-agent, and zeroclaw, respectively, but cache rate does not measure coding strength.Pass@1 measures coding outcome, cost measures evaluation resources, and cache hit rate diagnoses one mechanism affecting API billing.

6. Related Work

Claw-SWE-Bench builds on repository-level SWE-bench issue resolution, multilingual and verified subsets, and prior harness evaluations. It complements function-level and other coding benchmarks that measure code-generation skill rather than harness-mediated behavior.

  • Foundational SWE benchmarks: SWE-bench established resolving real GitHub issues against repository-level test suites, while multilingual and verified subsets supply Claw-SWE-Bench’s 350 instances.SWE-bench-Multilingual contributes 300 instances, and SWE-bench-Verified-Mini contributes the remaining 50.
  • Single-harness SWE evaluations: Prior harnesses span SWE-agent’s agent-computer-interface scaffold, AutoCodeRover’s code-aware retrieval, OpenHands’ SWE-bench adapter, and mini-SWE-agent’s minimal design.These systems represent a growing line of individual harnesses reporting per-system SWE-bench results.
  • Other coding benchmarks: HumanEval, MBPP, and APPS evaluate function-level synthesis, while CrossCodeEval, CodeClash, and PinchBench probe other coding-capability axes.These benchmarks are treated as orthogonal background because they assess model code-generation skill rather than harness-mediated behavior on real repositories.

7. Conclusion and Discussion · A.1. Code release

The paper introduces a multilingual benchmark and adapter protocol that makes OpenClaw-style coding agents repeatable, comparable, and scorable, alongside a cost-efficient Lite subset. It also releases adapters, selection tooling, and figure-generation scripts, while identifying replication, auditing, and generalization as future needs.

  • 7. Conclusion and Discussion: 350 instances comprise the multilingual CLAW-SWE-BENCH benchmark, complemented by an 80-instance CLAW-SWE-BENCH LITE subset.The conclusion presents both as the paper’s principal benchmark releases.
  • 7. Conclusion and Discussion: An adapter constrains OpenClaw to the SWE-bench execution environment, patch contract, and scoring protocol, enabling repeatable, comparable, and scorable evaluation.This establishes the adapter as the mechanism connecting a general-purpose agent to the benchmark protocol.
  • 7. Conclusion and Discussion: 0.4 pp is the approximate full-set aggregate Pass@1 difference reproduced by Lite, while its full-run cost is about 23% of full-350.Lite uses a 17-column cost-aware calibration spanning model and harness variation.
  • 7. Conclusion and Discussion: Single-run aggregates limit interpretation of differences of only a few percentage points, motivating future multi-seed replication to estimate randomness and run-to-run variance.The limitation concerns stability claims from the main experiments.
  • 7. Conclusion and Discussion: Future releases should retain raw token traces, alongside total API cost, input/output tokens, and cache hit rate, to support auditing and re-pricing.The paper reports these accounting quantities together because cost differences require auditability.
  • 7. Conclusion and Discussion: Whether model–harness non-separability generalizes to web or computer-use agents, and how loop, tools, parser, and stopping rule affect accuracy–cost trade-offs, remains open.These are identified as broader future research questions.
  • A.1. Code release: Five claw-adapter packages are released with run_infer.py, run_eval.py, orchestrators, agent adapters, and workspace modules.The registry also hosts the minimal bare adapter used for the diagnostic in §5.1.
  • A.1. Code release: The release includes Node.js Lite-construction tooling and matplotlib scripts regenerating selection analyses, cost–accuracy, cleanup-comparison, and Lite-parity figures.Named scripts include generate_pareto_figure.py, generate_leak_fix_figure.py, and generate_lite_figures.py.

A.2. Data release … B.4. API providers

The release provides curated instance IDs and metadata for the full and Lite benchmarks, alongside a uniform CLI and execution protocol across five claws. Experiments use single-run, cost-conscious evaluation on one CPU host, with remote API inference and equal runtime conditions across claws.

  • A.2. Data release: 350 full-benchmark instance IDs and 80 Lite instance IDs are released as JSON with metadata, while underlying issues and repositories remain hosted upstream.Only curated ID sets and metadata are redistributed from SWE-bench-Multilingual and SWE-bench-Verified-Mini sources.
  • A.3. Reproduction protocol: Predictions are passed to the evaluator through a standardized JSONL artifact path with the SWE-bench Multilingual dataset name and configurable worker count.The prediction command specifies artifacts/<run_id>/predictions.jsonl, the dataset name, run ID, and max_workers 8.
  • A.3. Reproduction protocol: The CLI surface is identical across all five claws, with harness-specific adapter flags documented separately.The reproduction protocol exposes shared harness, dataset, model, run, timeout, and worker parameters.
  • A.4. Random seeds and runs: 1 repeat per instance–harness–model cell is executed with 3-thread concurrency as a cost-driven trade-off.Multi-seed validation on a 50-instance slice is left to future work.
  • A.5. Compute requirements: One 16-core / 61 GiB-RAM Linux server is sufficient for the experiments, with no GPU required.All model inference is routed through external APIs, while the host supports orchestration and evaluation.
  • B. Compute and Environment Details: All five claws are executed on a single host with identical runtime parameters, making claw implementation the only experimental variable.The evaluated claws are OPENCLAW, HERMES-AGENT, ZEROCLAW, NANOBOT, and the GENERIC baseline.
  • B.2. Software stack: The software stack bind-mounts standalone Python and harness virtual environments into the SWE-bench evaluation container.This lets the agent loop run inside the same container as the patched code.
  • B.3. Run-time parameters (held equal across all five claws): Runtime parameters are overridden via CLI flags so all five claws receive the same per-instance budget and identical prompt template.Model inference runs through remote provider APIs, so API providers supply inference while the local host handles orchestration and patch evaluation.

B.5. Compute cost (aggregate) … D.3. K-sweep Decision

The benchmark standardizes compute accounting and harness configuration while evaluating diverse claws under shared prompts and runtime budgets. Its Lite-80 subset uses cost-aware, rank-aware calibration and conservatively releases 80 instances after all specified gates pass.

  • B.5. Compute cost (aggregate): ≈1,148 hours of end-to-end wall-clock cover 17 unique (claw, model) columns, including ≈671 model-sweep hours and ≈477 non-openclaw claw-cell hours.This equals ≈47.8 days of single-thread execution or ≈15.9 days on the actually used 3-thread schedule.
  • C. Harness Configurations: All five claws share the identical prompt template, 3600 s per-instance timeout, concurrency 3, repeats 1, and outer build–docker exec–git diff orchestration.Only the inner harness implementation varies, including its CLI surface, agent loop, tool set, and model adapter.
  • D.0 Shared prompt template (verbatim): The shared prompt directs agents to read, run tests, explore, reproduce, analyze, minimally fix, verify, and review issues without modifying test files.The rendered template is handed verbatim to openclaw, hermes-agent, zeroclaw, and nanobot; GenericAgent receives a documented tool-specific variant.
  • D.1 openclaw: openclaw uniquely creates isolated per-instance agents, workspaces, session stores, and memory directories while enforcing a tool deny-list and structured JSON output.The adapter disables memory, web, session-spawning, sub-agent, cron, and image tools before extracting run metadata.
  • D.2 hermes-agent: hermes-agent is a stateless CPython harness invoked with –yolo, terminal/file toolsets, and a 300-turn loop, with outcomes classified from exit code and stdout.Its 3600 s timeout matches the other claws, so differences arise from the CLI surface and toolset rather than a different wall-clock budget.
  • D.3 zeroclaw: zeroclaw is a stateless Rust single binary with an opaque 50-turn loop and native costs.jsonl logging of per-turn input, output, and total tokens.The orchestrator sums logged turns and token counts into per-instance metrics.
  • D.4 nanobot: nanobot runs a stateless Python loop of up to 30 turns, then copies its session JSONL and scrubs workspace metadata before collecting the source-code diff.This cleanup addresses nanobot-specific workspace pollution while preserving the full conversation and tool-call audit trail.
  • D.5 generic (GenericAgent): GenericAgent uses a file-based input.txt/output.txt contract with sentinel polling, fixed function-calling tools, and prompt-disabled web access under the shared adapter protocol.It is distinct from the bare adapter diagnostic and exports patches through git diff against the base commit.

D.4. Distribution and Cross-claw Validation · D.5. Coverage and Cost · E. Per-Language Breakdown

Lite-80 closely matches full-350 across calibration distributions and cross-claw evaluations while covering most repositories at roughly one-quarter of the full benchmark’s resource cost. The appendix provides per-language breakdowns computed directly from released result workbooks under the specified future-commit cleanup setting.

  • D.4. Distribution and Cross-claw Validation: 0.639 is full-350 Pass@1 and 0.643 is Lite-80 Pass@1, averaged over all 17 calibration columns, a +0.4 percentage-point difference.Table 9 reports the per-language distribution match between the full and Lite benchmarks.
  • D.4. Distribution and Cross-claw Validation: 1.88 percentage points is the mean absolute Lite-vs-full gap on the 5-claw × 2-model grid.The direct cross-claw parity check compares Lite-80 with full-350 across the shared claw and model configurations.
  • D.4. Distribution and Cross-claw Validation: 3.68 percentage points is the maximum Lite-vs-full gap on the 5-claw × 2-model grid.This maximum complements the reported mean absolute gap in the direct parity check.
  • D.5. Coverage and Cost: 34 unique repositories in Lite-80 cover 34/43 = 79% of the repositories in full-350.The released Lite-80 is designed to preserve broad repository coverage while reducing evaluation cost.
  • D.5. Coverage and Cost: 22.9% is Lite-80’s true-cost ratio relative to full-350, supporting an approximately four-times cheaper evaluation surface.The intended uses include debugging, regression testing, and preliminary model or claw evaluation.
  • D.5. Coverage and Cost: 22.2% input tokens, 23.6% output tokens, 22.6% cache-read tokens, and 23.0% wall-clock duration are Lite-80’s full-run resource ratios relative to full-350.These ratios are close to the raw instance ratio.
  • E. Per-Language Breakdown: Per-language breakdown tables are computed directly from released result workbooks, with no values estimated.Multilingual results use the future-commit cleanup setting of §5.3; the workbooks cover the combined-350 model sweep and cache-fixed 5-claw cross report.

E.1. Per-Language Resolved Rate (openclaw × 9 models)

Under the openclaw harness, resolved rates vary substantially by language across nine models on the 350-instance benchmark. Rust and Java most often lead, while Go is consistently weakest, and model-specific peaks differ across languages.

  • Per-language resolved rate: Table 11 reports per-language resolved rates for 9 models evaluated under the openclaw harness on the 350-instance benchmark.Models are ordered by overall resolve rate, and totals match Table 2.
  • Per-language resolved rate: Go is the worst language for all 9 models, with resolved rates ranging from 33.3–61.9%.Go also sits 11–22 pp below each model’s overall mean.
  • Per-language resolved rate: GPT 5.5 peaks on Rust at 93.0%, while Claude Opus 4.7 leads JS/TS at 81.4% and Python at 82.0%.The two leading models therefore exhibit different language profiles.

E.2. Per-Language Resolved Rate (claw sweep, 5 claws × 2 models)

The five-claw, two-model sweep reveals strongly language-dependent performance: Go is usually hardest, spreads widen on Qwen 3.6-flash, and claw rankings can invert across languages.

  • Sweep setup: The claw sweep evaluates five claws on GLM 5.1 and Qwen 3.6-flash across 10 (claw, model) cells and eight languages.Each table row represents one claw–model cell, with language-specific and overall resolved rates.
  • Language variation: Go is the hardest language in 8 of 10 cells; exceptions are nanobot × Qwen 3.6-flash, where JS/TS is weakest at 37.2%, and zeroclaw × Qwen 3.6-flash, where Go ties C/C++ at 47.6%.These exceptions identify the only cells where Go is not uniquely weakest.
  • Language variation: 19.0 pp is the largest per-language claw spread on GLM 5.1, occurring on Go from 38.1–57.1%, while Qwen 3.6-flash reaches 41.8 pp on Java from 41.9–83.7%.The within-model claw spread is therefore substantially wider on the smaller model, especially for Java.
  • Language variation: 19.0% on Go and 25.0% on Ruby are the generic baseline’s lowest Qwen 3.6-flash results, while zeroclaw × GLM 5.1 achieves the grid’s best Java cell at 83.7%.The Java result occurs despite zeroclaw’s mid-pack overall rate, demonstrating that rankings can invert across languages within one model.

F. License and Ethics

Claw-SWE-Bench derives from two MIT-licensed upstream datasets while preserving upstream licenses and citations, alongside heterogeneous repository-level licensing obligations. Its evaluation of real software bugs has dual-use implications, mitigated by releasing the protocol and instance_id list rather than vulnerable patches as targets.

  • License: The benchmark derives from SWE-bench Multilingual and SWE-bench Verified-Mini, both released under MIT.The sources are hosted through the official SWE-bench project and the Verified-Mini repository, respectively.
  • License: Released instances retain original repository licenses, spanning BSD, Apache 2.0, MIT, and some GPL-licensed projects.Notable examples include Django and Flask under BSD, requests and xarray under Apache 2.0, and pylint under GPL-2.
  • License: Users redistributing patches or derivative work must comply with each repository’s license.The benchmark retains both upstream LICENSE files and citations, while task source code preserves its original repository license.
  • Ethics: Evaluating coding agents on real software bugs can support maintenance and accessibility but may also enable autonomous exploitation of vulnerable software.The benchmark inherits this dual-use ethical surface from upstream SWE-bench.
  • Ethics: The release mitigates this risk by providing the benchmark protocol and instance_id list rather than vulnerable patches as targets.This mitigation is stated as part of the benchmark’s broader-impact and ethical considerations.
Loading 2606.12344v1…