SWE-Duel: Towards a Self-Scaling Arena for Coding Agents

Motivation

• Static benchmarks risk leakage and saturation because their publicly accessible task sets are fixed at release while coding agents continue to improve.The paper contrasts these with self-scaling benchmarks whose tasks and oracles are created at evaluation time and whose difficulty is set by participating agents.Source: §1 · p. 2

Approach

• SWE-DUEL pairs two coding agents in mirrored turns: Red adds a feature and hidden bug, and Blue reviews and patches the pull request.Tasks are admitted only if they pass feature and regression tests while failing the bug tests, and a fresh session of their author can repair them blind.Source: §1 · p. 2

Results

• In a 1,500-turn tournament, GPT-5.5 with Codex CLI repaired 98.3% of reviewed defects, while Claude Opus 4.8 with mini-swe-agent had the highest Red win rate, 27.3%.The tournament covered 11 participants, 15 repositories, and five languages.Source: §1 · p. 2

Contributions

• The authors provide an open arena toolkit for launching tournaments or adding agents to an existing leaderboard.They also report a 1,500-turn tournament across 11 model-harness pairs.Source: §1 · p. 3

Repository-level coding benchmarks

• Repository-level benchmarks use public, fixed tasks that can enter training data, while refreshed tasks remain drawn from human development history and do not necessarily become harder.The paper contrasts this with SWE-DUEL, where agents create tasks and oracles at evaluation time.Source: §2 · p. 3

Automatic bug generation

• SWE-DUEL bugs stay hidden from visible tests and are admitted only after the author repairs them blind, unlike BugPilot’s visible-test-breaking regressions and CDBench’s shown mutant diffs [Sonwane et al., 2025].Its deliberate bug injection also avoids relying on accidental agent mistakes for task yield.Source: §2 · p. 4

Arena-style evaluation

• SWE-DUEL retains execution-based scoring but replaces human-written task pools and gold-patch validation with agent-authored tasks, execution gates, and blind self-repair.SwingArena uses fixed human-written issues and validates reviewer tests against a human gold patch [Xu et al., 2026].Source: §2 · p. 4

3 SWE-DUEL: A SELF-SCALING CODING ARENA

• Each duel has Red create a feature pull request with a hidden bug and Blue review it; Blue wins only when its revision passes existing, feature, and sealed bug tests.Because the bug is embedded in the feature, reverting the pull request breaks the feature tests.Source: §3 · p. 4

3.1 PARTICIPANTS AND ENVIRONMENT

• Each participant is a model-harness pair, and repository tasks run in pinned Docker environments with the same image used for Red sessions, validation, and Blue runs.The existing test suite is rerun on later artifacts to detect regressions.Source: §3.1 · p. 4

3.2 RED AGENT: TASK CREATION

• Red creates features, embeds bugs, and undergoes blind self-repair in separate sessions; gates verify feature correctness, hidden-bug behavior, and solvability.Each slot allows at most five attempts, and exhausted slots remain empty; admitted tasks are limited by the author’s own bug-finding ability.Source: §3.2 · pp. 5–6, Figure 3 · p. 5

3.3 BLUE AGENT: TASK SOLVING

• Blue receives the public task bundle without bug tests or notice that a bug exists, then wins only if all three test suites pass on its revision.There is no partial credit; leaving the pull request unchanged loses because the bug tests fail.Source: §3.3 · p. 6

3.4 TOURNAMENT

• Matches give each participant turns as both Red and Blue on the same repositories; missing tasks from exhausted slots count as Blue wins.A new participant can join a finished tournament by playing only ⌈log2 M⌉+1 incumbents while reusing earlier task banks and results.Source: §3.4 · p. 6

4.1 EXPERIMENTAL SETUP

• The tournament covers 15 repositories in five languages and 11 model–harness participants across 50 matches and 1,500 turns.The original ten participants played a round robin; Gemini 3.1 Pro joined later as a late entrant. Missing Red challenges count as automatic Blue wins.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

• GPT-5.5 with Codex CLI leads the full-schedule Bradley–Terry ranking, followed by Claude Opus 4.8, Kimi-K2.7 Code, and DeepSeek-V4-Pro with mini-swe-agent.MiniMax-M3 and the five-match Gemini 3.1 Pro entrant rank lowest.Source: §4.2 · p. 7, Table 1 · p. 7
• GPT-5.5 is the strongest solver, while Claude Opus 4.8 with mini-swe-agent is the strongest task creator; MiniMax-M3 and Gemini have the weakest task-creation rates.Claude Opus 4.8’s Red win rate is 27.3%, versus 11.3% and 10.4% for the two GPT-5.5 participants; MiniMax-M3 and Gemini often fail to generate admitted tasks, giving Blue an automatic win.Source: §4.2 · pp. 7–8
• Blue succeeds on 87.9% of all 1,500 turns and 85.1% of the 1,219 contested turns.Contested turns include 1,037 clean repairs, 158 cases where the latent bug ships, and 24 collapses that break the feature or existing behavior.Source: §4.2 · p. 8
• Gemini 3.1 Pro joined with five active-sampled matches, finished 0–1–4, and ranked last, while the incumbent ordering remained unchanged.Its contested Blue success was 61.8%; refitting Bradley–Terry on the 45 incumbent-only matches reproduced the full-fit ordering exactly (Kendall’s τ = 1.0).Source: §4.2 · p. 8

4.3 ADDITIONAL ANALYSIS

• A fixed solver’s failure rate rises from 0.0% on GLM-5.2 tasks to 42.9% on GLM-5.3 tasks, compared with 8.3% on GLM-5.1 tasks.The upward trend is significant (exact trend test p=.015); the head-to-head results also show GLM-5.3 tasks defeating GLM-5.1 in 7/14 and GLM-5.2 in 4/14 contested turns.Source: Table 2 · p. 8, §4.3 · p. 9
• Against a fixed reference solver, Claude Opus 4.8 with mini-swe-agent remains the strongest task creator, with a 27.3% endogenous Red win rate and a 26.7% reference Red win rate.Across participants, the two task-difficulty scales agree at Spearman ρ = 0.66; the comparison uses the fixed solver on tasks from the main tournament.Source: §4.3 · p. 9, Table 3 · p. 9
• Admitted tasks were mostly reliably self-repaired, while a sampled bug-test audit found a 6.0% false-negative rate; the harness ablation placed mini-swe-agent highest in BT score.Of 44 tasks, 35 were self-repaired in all three reruns, eight in two, and one in none. In 50 sampled Blue losses, reviewers found 45 genuine misses, three valid repairs rejected by tests, and two ambiguous cases. Harness scores were +0.73 for mini-swe-agent, −0.25 for Codex CLI, and −0.49 for Claude Code.Source: §4.3 · p. 9

• SWE-DUEL is an arena in which coding agents test each other.The tournament included 11 participants and 1,500 turns.Source: §5 · p. 10
• GPT-5.5 with Codex CLI was the strongest solver, repairing 98.3% of reviewed defects, while Claude Opus 4.8 with mini-swe-agent was the strongest task author, with a 27.3% Red win rate.Source: §5 · p. 10
• Tasks written by the newest version in one model family defeated a fixed-reference solver more often than tasks from older versions, providing early evidence that task difficulty scales with authoring ability.A new participant joined with five matches without changing the existing order.Source: §5 · p. 10

• Generative AI was used to generate tasks and attempt solutions for evaluating AI coding agents.Source: AI USE STATEMENT · p. 16
• The authors report not using generative AI to conceptualize the framework, design experiments, implement methods, or interpret results.Source: AI USE STATEMENT · p. 16
• AI-generated code was manually verified in pair-programming sessions and with unit and integration tests, and the authors take responsibility for the final content.Source: AI USE STATEMENT · p. 16

ETHICS STATEMENT

• The 27.3% attack success rate demonstrates that plausible, test-passing contributions can retain defects despite automated review, but is not an exploit rate for deployed software.Prompts, trajectories, sealed tests, and admitted challenges are dual-use artifacts.Source: ETHICS STATEMENT · p. 16
• Generation and evaluation used isolated local containers at pinned revisions; the study did not contact maintainers deceptively or submit generated artifacts to real repositories.If future use reveals an independently existing vulnerability, the authors require coordinated disclosure before publication; safeguards do not eliminate dual-use risk.Source: ETHICS STATEMENT · p. 16

A TASK FRESHNESS AND TASK BANK RELEASE POLICY

• A disclosed task bank must be marked retired because public release prevents it from supporting future unseen-task claims.The authors encourage generating a new bank for each replication study.Source: A TASK FRESHNESS AND TASK BANK RELEASE… · p. 16

D.1 HARNESS SETUP

• Each session had a 50-step initial budget and 30-minute limit; recovery could add up to three turns of six steps.The step counts do not represent equal compute across models and harnesses.Source: §D.1 · p. 26

E EVALUATION AND OUTCOME CLASSIFICATION

• Blue wins only when its patch passes the repository’s existing tests, feature tests, and hidden bug tests; failures are classified by which visible suites fail.Existing-test failure is a regression, feature-test failure after existing tests pass is feature collapse, and hidden-test failure after both pass means the defect shipped.Source: E EVALUATION AND OUTCOME CLASSIFICATION · p. 27

G MANUAL AUDIT FOR SEALED-ORACLE FALSE NEGATIVES

• In a 50-case audit, 45 failures were genuine misses, three were false negatives, and two were ambiguous; the estimated false-negative rate was 6.0%.Excluding ambiguous cases, it was 6.25%; all three false negatives arose from two Go JWT tasks.Source: G MANUAL AUDIT FOR SEALED-ORACLE FALSE… · pp. 28–29

H MAIN TOURNAMENT BRADLEY–TERRY CONFIDENCE INTERVALS

• No adjacent participants in the main tournament ranking were distinguishable at 95% confidence.The intervals use 1,000 match-bootstrap draws.Source: H MAIN TOURNAMENT BRADLEY–TERRY… · p. 29, Table 6 · p. 29

I SESSION-C ADMISSION STABILITY

• The probe repeated Session C three times per sampled task with the same model–harness pair and settings, using 44 tasks sampled from the 134 tournament-contested tasks.Each run launched a fresh reviewer instance; the study sampled four tasks per participant, round-robined across repositories.Source: I SESSION-C ADMISSION STABILITY · p. 29, Table 7 · p. 29
• 79.5% of sampled tasks were repaired in all three re-runs, while one task (2.3%) was repaired in none; pooled repair was 121/132 = 91.7%.Eight tasks were repaired 2/3 times, none 1/3; the study identifies the single 0/3 task as a lucky admission.Source: I SESSION-C ADMISSION STABILITY · p. 30, Figure 6 · p. 30
• The 2/3 group marks a reviewer-ability boundary where repair depends on sampling, while the single 0/3 task marks the lucky-admission tail.The authors conclude admission noise does not materially attenuate reported Blue success rates.Source: I SESSION-C ADMISSION STABILITY · p. 30

J FIXED-REFERENCE TASK SOLVER

• The study used GLM-5.2 with mini-swe-agent as the fixed reference, blind-solving all 134 admitted tasks through the match evaluation path.Its in-match results covered 111 tasks; the remaining 23 received fresh blind runs.Source: J FIXED-REFERENCE TASK SOLVER · pp. 30–31
• The strongest task creator remained first on the fixed-reference scale: Claude Opus 4.8 with mini-swe-agent had 27.3% endogenous and 26.7% reference Red success.Across participants, the two difficulty scales agreed at Spearman ρ = 0.66 and Kendall τb = 0.58.Source: J FIXED-REFERENCE TASK SOLVER · p. 31, Figure 7 · p. 31
• The fixed solver failed on 15 of 134 tasks (11.2%), and Kimi-K2.7-Code’s 17.9% scheduled Red success fell to 0% against the reference.The authors interpret Kimi’s scheduled success as exploiting weaknesses of its opponents rather than intrinsic task difficulty.Source: J FIXED-REFERENCE TASK SOLVER · p. 31
• With only 5–15 tasks per participant, confidence intervals are wide, limiting claims about finer-grained ranking order.The authors plan a larger study of rank-level agreement.Source: J FIXED-REFERENCE TASK SOLVER · p. 31

K SCALING TASK DIFFICULTY WITH TASK GENERATOR CAPABILITY

• The ladder compared GLM-5.1, GLM-5.2, and GLM-5.3 using the same mini-swe-agent harness and DeepSeek-V4-Pro with mini-swe-agent as the blind reference solver.The 39 tasks were evaluated through the same match-subturn path and reused existing in-match results for GLM-5.2 tasks.Source: K SCALING TASK DIFFICULTY WITH TASK… · p. 32
• Solver failures were 1/12 for GLM-5.1, 0/13 for GLM-5.2, and 6/14 for GLM-5.3; the upward trend test gave z=2.32, one-sided p=.015.All seven failures were complete collapses, and the solver used more steps on tasks it lost than on tasks it solved.Source: K SCALING TASK DIFFICULTY WITH TASK… · p. 32
• The difficulty increase was not explained by defect-category mix or admission yield: security-only tasks defeated the solver in 5/7 GLM-5.3 cases versus 0/6 older-version cases.Admission yields were non-monotonic across the three versions.Source: K SCALING TASK DIFFICULTY WITH TASK… · p. 32
• In a sibling round robin, GLM-5.3 tasks defeated GLM-5.1 in 7/14 turns and GLM-5.2 in 4/14, while the older versions rarely defeated opponents.The authors caution that shared training lineage may produce shared blind spots, so this comparison corroborates rather than replaces the fixed-solver measurement.Source: K SCALING TASK DIFFICULTY WITH TASK… · p. 32
• 42.9% of GLM-5.3 tasks defeated the fixed solver, versus 4.0% of pooled GLM-5.1 and GLM-5.2 tasks.The newest-versus-older comparison was 6/14 versus 1/25 (Fisher p=.005).Source: K SCALING TASK DIFFICULTY WITH TASK… · p. 32
• The ladder used only 12–14 tasks per level and one draw per task; adjacent-version resolution is limited, and the two older levels’ intervals largely overlap.The authors describe the scaling trend as approximate rather than strictly monotone.Source: K SCALING TASK DIFFICULTY WITH TASK… · p. 33

L ROLE DECOMPOSITION AND CONTESTED-ONLY BT REFITS

• Role decomposition separates Red generation from Blue solving; Nemotron-3-Ultra’s Red win rate of 10/70 rests on one task and has the widest task-creation interval [0.000, 0.450].Coverage is admitted tasks divided by 15 scheduled repository slots; an unsuccessful chain of five attempts counts as an exhausted slot.Source: L ROLE DECOMPOSITION AND… · p. 33, Table 8 · p. 33
• The authors warn that shrinking gaps in task-solving ability could also shrink authoring gaps and risk arena saturation; they propose larger repositories and a less restrictive solvability check.The proposed check could tell the self-review session that a bug exists or identify its type, allowing more subtle bugs.Source: L ROLE DECOMPOSITION AND… · pp. 33–34
• Removing 80 auto-win concessions moves Nemotron-3-Ultra from ninth to third (+0.145), but its interval [−0.35, +0.75] includes zero and nine wins come from one task.Its contested-only refit deletes the concessions and reweights the record toward contested Red wins.Source: L ROLE DECOMPOSITION AND… · p. 34

M RED–BLUE TRANSFER

• Each observed Red–Blue transfer cell represents one scheduled turn per repository, so its x/15 values are raw counts rather than precise estimates.The late entrant has matchups with only a subset of incumbents; future transfer studies could freeze a common task subset per author.Source: Figure 8 · p. 33, M RED–BLUE TRANSFER · p. 34

N TURN-LEVEL BRADLEY–TERRY ANALYSIS

• Full-schedule turn-level rankings are closer to match-level full-schedule rankings (Kendall’s τ = 0.782) than to contested-only rankings (τ = 0.382).The contested-only turn-level and match-level orderings have Kendall’s τ = 0.527.Source: N TURN-LEVEL BRADLEY–TERRY ANALYSIS · p. 35, Table 9 · p. 36
• Under turn-level full-schedule BT, GPT-5.5 with Codex drops to fourth (+0.18), while Claude Opus 4.8 with mini-swe-agent rises to first (+0.30) and DeepSeek-V4-Pro to second (+0.25).The top seven ranking intervals overlap; the per-turn weighting reflects Opus’s field-best 27.3% Red win rate.Source: N TURN-LEVEL BRADLEY–TERRY ANALYSIS · p. 35

O A CONCENTRATED RED-WIN RECORD

• Nine of Nemotron-3-Ultra’s ten contested Red wins came from one task, and the section reports that task’s ten solving runs to examine what its difficulty measures.The task’s sealed defect concerns a prefix-matching condition: the configured prefix path itself fails validation instead of being skipped.Source: O A CONCENTRATED RED-WIN RECORD · p. 36
• Only one of ten solvers repaired the defect; the other nine passed existing and feature suites but failed the sealed suite, although most inspected the relevant condition.The successful solver distinguished the unslashed path from the prefix path and added matcher tests; the paper says the task is genuine but the concentrated Red record makes the capability estimate high-variance.Source: O A CONCENTRATED RED-WIN RECORD · p. 36

P DEFECT COMPOSITION

• The taxonomy includes logic, security, off-by-one, type-confusion, resource, input-validation, and race-related defects, but its frequencies describe generated tasks rather than human-software prevalence.The resource-leak bar is based on one task (nine turns) and is not interpreted; Missing-Red auto-wins are excluded from the contested Blue win-rate figure.Source: P DEFECT COMPOSITION · p. 37, Figure 11 · p. 37, Figure 10 · p. 37

Q HARNESS ABLATION STUDY

• The harness ablation resolves no ordering: aggregate score-rate estimates span 0.03 points with intervals about 0.17 wide, and even the largest within-model contrast overlaps.For DeepSeek V4 Pro, Codex CLI is 0.62 [0.46, 0.78] versus Claude Code at 0.40 [0.27, 0.57]; paired score margins also include zero, and within-model intervals overlap.Source: Q HARNESS ABLATION STUDY · p. 38, Table 10 · p. 38, Figure 12 · p. 38
• The ablation cross-pairs three harnesses with DeepSeek V4 Pro, GLM 5.2, and Kimi K2.7 Code, avoiding native pairings that could confound the harness-effect analysis.Each harness pools 45 tasks from the three base models and records 180 total turns.Source: Q HARNESS ABLATION STUDY · pp. 37–38

R COST

• GPT-5.5 with Codex cost $641.89, 2.6 times the next-highest total and 7.4 times MiniMax-M3; totals also vary with generation yield and match count.Generation costs include prior failed attempts within a slot and exhausted slots that do not produce contested turns.Source: R COST · pp. 38–39, Table 12 · p. 39
• Nine of eleven Blue-spend correlation intervals include zero; Nemotron-3-Ultra has a positive association (r = .384 [.213, .532]), while GPT-5.5 with mini-swe-agent has a small negative one (r = −.107 [−.197, −.004]).The paper characterizes this as descriptive and exploratory because difficult tasks can induce longer Blue trajectories and easier tasks shorter ones.Source: R COST · p. 39
• At a matched output-token budget of 638,976, GPT-5.5/Codex and GPT-5.5/mini-swe remain first and second, with matched and complete-case solving orders at Kendall’s τ = 1.000.This supplementary sensitivity analysis covers contested task solving only, not Red task creation.Source: R COST · p. 39

S NEWCOMER ONBOARDING AND MARGINAL COST

• The entrant’s marginal cost was $243.40: $125.95 for task creation and $117.45 for match evaluation.Evaluation included $61.81 for the entrant’s 55 fresh solving runs and $55.64 for incumbents’ 48 recorded runs on its admitted tasks.Source: S NEWCOMER ONBOARDING AND MARGINAL COST · pp. 40–41, Table 15 · p. 41
• Gemini-3.1-Pro finished last with a 0–1–4 record and a full-schedule BT estimate of −4.55 [−12.57, −2.45]; the incumbent ordering was unchanged (Kendall’s τ = 1.0).Its contested solving success was .618 [.491, .741], the lowest in the pool, and its contested Red win rate was .042.Source: S NEWCOMER ONBOARDING AND MARGINAL COST · p. 41
• A ten-incumbent entrance was estimated at about $379; active sampling saved about a third of evaluation spend by using ⌈log2 N⌉+1 matches instead of O(N) pairings.The observed five-match entrance reused incumbent banks, completed matches, and cached solving runs.Source: S NEWCOMER ONBOARDING AND MARGINAL COST · pp. 39–41

T TEXTUAL INTERFACE

• The framework includes an interactive terminal interface for task generation and tournament management, with separate views for generation slots and tournament scheduling.The generation view tracks feature and bug sessions, gate outcomes, and attempt history; the scheduling view uses persisted pairings and cached solving runs.Source: T TEXTUAL INTERFACE · p. 42, Figure 14 · p. 42, Figure 15 · p. 42