PaperlayerView exact revision on arXiv

Source-linked AI summary

Pairing Matters: Performance, Failures, and Efficiency in Lightweight Coding Agents

Lu Zhao, Christopher M. Poskitt, David Lo

arXiv:2610.32131v1cs.SE

TL;DR

Existing evaluations provide limited systematic evidence on how harness choice affects a lightweight model’s coding-agent performance. This study evaluates six harnesses paired with three models across 100 SWE-bench Verified tasks, finding that performance, failure patterns, and efficiency depend on the pairing.

  • Problem

    Existing evaluations largely report fixed harness–model configurations, leaving limited systematic evidence about how pairing choices affect resolution, failures, and cost.

    Source: §I · p. 1

  • Method

    The authors conduct a controlled full-factorial evaluation of six open-source harnesses and three lightweight coding models on 100 tasks, yielding 1,800 graded runs.

    Source: §I · pp. 1–2

  • Results

    Harness choice changes the same model’s resolution accuracy by up to 44 percentage points, while harness rankings, failure profiles, and token efficiency vary across pairings.

    Source: §IV · p. 8, §VIII · p. 10

  • Takeaways & Limitations

    Pairing selection should be based on resolution and workload coverage, while token budgets and harness improvements should be informed by pairing-specific efficiency and failure evidence.

    Source: §I · p. 2

Abstract

from arXiv · show

Self-hosting coding agents is often necessary for software-engineering teams that must keep proprietary code and development traces within controlled infrastructure. Lightweight, open-weight models make such deployment more feasible by reducing hardware requirements, while strong results on repository-level benchmarks demonstrate their potential for demanding coding tasks. Deploying these models as coding agents requires pairing them with an agent harness, making their joint performance a key deployment consideration. Yet existing leaderboards primarily report fixed harness-model configurations rather than controlled comparisons across alternative pairings. To address this gap, we investigate harness-model interactions through a controlled full-factorial evaluation of six open-source harnesses and three lightweight coding models across 100 SWE-bench Verified tasks, producing 1,800 graded runs under common serving, metering, and grading conditions. For the same model, resolution rates differ by up to 44 percentage points across harnesses, while harness rankings change across models. We apply an evidence-based failure taxonomy to 1,100 unsuccessful runs and find substantial differences in failure patterns across harness-model pairings. A model's dominant failure modes under one harness need not dominate under another. Efficiency is similarly dependent on the pairing: even among model-directed loop harnesses, charged tokens per resolved task vary by more than 20-fold. These findings make harness-model selection a central deployment decision for lightweight coding agents. Strong performance with one harness or model does not guarantee similar results with another. We translate these findings into practitioner guidance for pairing selection, token-efficiency assessment, and trajectory-informed harness engineering, and release the complete experimental records and analysis artifacts.

I. INTRODUCTION

Lightweight open-weight models make self-hosted coding agents more feasible, but deployment requires choosing a harness–model pairing. The study evaluates pairing performance, failures, and efficiency to inform that choice.

I. INTRODUCTION

  • Existing evaluations offer limited systematic evidence about how changing the harness affects the same lightweight model’s performance and related outcomes.The audit found that five submissions reported fixed configurations using off-the-shelf models with fewer than 100B total parameters and established harnesses.Source: §I · p. 1
  • The study evaluates six open-source harnesses with three lightweight coding models on 100 SWE-bench Verified tasks, producing 1,800 graded runs under common conditions.The analyses cover performance, failure mechanisms, and efficiency.Source: §I · p. 1, Figure 1 · p. 2
  • Harness choice changes the same model’s resolution rate by up to 44 percentage points, and harness rankings do not transfer consistently across models.Failure mechanisms and charged tokens per resolved task also vary by pairing.Source: §I · pp. 1–2
  • The study develops a five-mechanism failure taxonomy and applies it to all 1,100 unsuccessful runs to compare pairing-specific failure profiles.The analyses compare failures across models, harnesses, and individual pairings.Source: §I · p. 2
  • Practitioners are advised to test alternative harnesses and models, select using resolution rate and workload task coverage, and budget tokens per resolved task.Pairing-specific failure evidence can guide improvements to progress control, fault localization, and validation.Source: §I · p. 2
  • The authors release records for all 1,800 graded runs, including available trajectories, candidate patches, serving telemetry, failure labels, configurations, and analysis scripts.Source: §I · p. 2

II. STUDY DESIGN

The study uses a full-factorial design, pairing every selected model with every selected harness. Harnesses were chosen to represent contrasting, practically relevant execution-control designs.

II. STUDY DESIGN

  • Every selected coding model is evaluated with every selected harness in the full-factorial design.The study covers six harnesses and three models.Source: §II · p. 2

A. Agent Harness Selection

  • Harness eligibility required a public, version-pinnable implementation with headless execution and model substitution through a locally served OpenAI-compatible endpoint.Adapters were used when needed while preserving each harness’s interaction workflow.Source: A. Agent Harness Selection · p. 2
  • The six selected harnesses balance adoption with contrasting execution control, grouped as model-directed loops, phased retrieval and repair, or a fixed repair pipeline.The selection prioritizes practical relevance and design contrast rather than exhaustive coverage.Source: A. Agent Harness Selection · pp. 2–3

B. Model Selection

The study selects three coding-specialized open-weight models using deployment eligibility and reported performance, adoption, and design diversity. It evaluates them on a fixed 100-task subset of SWE-bench Verified.

B. Model Selection

  • Models had to have 10–100B total parameters and public checkpoints servable on one H100-class GPU; selection also considered reported benchmark performance, adoption, and design diversity.The selected models span dense and sparse architectures and differ in training emphasis, reasoning modes, and tool-use protocols.Source: B. Model Selection · p. 3
  • GLM-4.7-Flash is evaluated with thinking enabled, unlike the non-thinking configurations of the other two models.It is a 30B-total-parameter mixture-of-experts model with approximately 3B active parameters.Source: B. Model Selection · p. 3

C. Benchmark Selection

  • The study evaluates a fixed-seed subset of 100 tasks from the 500-task SWE-bench Verified benchmark, sampling for repository coverage and official difficulty categories.The allocation gives smaller repositories greater representation than in the full benchmark; grading defects were excluded or replaced through documented seeded procedures.Source: C. Benchmark Selection · pp. 3–4

D. Experimental Runtime

A common runtime, serving setup, and measurement protocol support comparison across pairings, while harness-specific execution limits remain part of the evaluated configurations. The paper records run artifacts and defines shared grading and efficiency measures.

D. Experimental Runtime

  • All pairings share the task set, serving infrastructure, grading procedure, and a cumulative threshold of 262,144 completion tokens per run.Additional harness-specific limits and timeouts are not directly equivalent, so results characterize the evaluated deployment configurations.Source: D. Experimental Runtime · p. 4
  • Models are served sequentially through vLLM with each model’s serving configuration fixed across harnesses, using a common metering infrastructure on one NVIDIA H100 NVL GPU.Harnesses run on a separate CPU host, and requests and token usage are recorded.Source: D. Experimental Runtime · p. 4

E. Grading and Metrics

  • A run is one pairing’s attempt on one task, and its trajectory is the sequence of model requests and responses, including tool calls and observations.Run records associate available trajectories with logs, metadata, candidate patches, and benchmark grades.Source: E. Grading and Metrics · p. 4
  • Trajectory length counts model calls, while charged-token cost sums prompt and completion tokens reported by the serving engine across calls.Source: E. Grading and Metrics · p. 4
  • Resolution uses the official SWE-bench Docker grader after filtering specified edits; missing or empty patches count as unsuccessful.Resolution rate is the proportion of a pairing’s 100 tasks resolved.Source: E. Grading and Metrics · p. 4

III. RQ1: PAIRING PERFORMANCE

Resolution and task overlap vary across harness–model pairings, so component-level strengths do not reliably identify the best combination. The combined coverage across harnesses exceeds the strongest individual pairing for every model.

III. RQ1: PAIRING PERFORMANCE

  • 79 tasks form Qwen3-Coder’s combined coverage versus 63 for OpenHands; GLM-4.7-F covers 62 versus 51 for its top harnesses, and Devstral-S covers 64 versus 54 for OpenHands.The combined coverage exceeds the strongest individual pairing by 16, 11, and 10 tasks, respectively.Source: §III · p. 5
  • 28–44 percentage points separate same-model resolution rates across harnesses, and the leading harness changes with the model.Even the best harness for each model resolves 10–16 fewer tasks than the combined coverage across all six harnesses.Source: §III · p. 5
  • OpenCode and OpenHands share 34 and 33 tasks, respectively, across all three models, compared with 7 for SWE-agent.SWE-agent resolves 17 tasks exclusively with GLM-4.7-F, illustrating model-dependent task overlap.Source: §III · p. 5, Figure 2 · p. 5
  • 79 tasks are in Qwen3-Coder’s union across harnesses, but only 5 are resolved by all six; GLM-4.7-F’s corresponding counts are 62 and 15.Devstral-S covers 64 tasks, including 16 resolved by exactly one harness; broader union coverage does not necessarily mean a larger shared core.Source: §III · p. 5, Figure 2 · p. 5
  • OpenCode and OpenHands show comparatively consistent task coverage across models, while GLM-4.7-F shows greater consistency across harnesses; these strengths do not establish universal superiority.Component-level strengths can guide selection but cannot replace evaluating the intended pairing.Source: §III · p. 5

IV. RQ2: FAILURE MECHANISMS

Failure profiles vary across harness–model pairings, and aggregate profiles alone do not identify which mechanisms dominate an individual pairing. The analysis applies a five-category, evidence-based taxonomy to unsuccessful runs and finds distinct model-level tendencies whose severity changes across pairings.

Failure taxonomy

  • The five mutually exclusive failure types are assigned by reviewers using ordered criteria, with diagnostic indicators supporting rather than automatically determining labels.The taxonomy distinguishes exchange breakdowns from progress, localization, validation, and repair-correctness failures.Source: §IV · p. 6, Table IV · p. 6
  • 784 of 1,100 runs received matching initial reviewer labels (Cohen’s κ = 0.600); disagreements were adjudicated, and 48 matching cases were rechecked, with one label reassigned.Indicators from proxy records, trajectories, and patches were assembled into evidence packs for blinded independent review.Source: §IV · p. 6

Model-level failure profiles

  • 386 failures (35.1%) were no-progress loops or budget exhaustion, and 362 (32.9%) were wrong or incomplete fixes; the other three categories accounted for 12.8%, 11.7%, and 7.5%.Figure 3 aggregates the five failure types by model, harness, and pairing; segment widths encode failure counts and bar-end numbers give unsuccessful-run totals.Source: §IV · p. 6, Figure 3 · p. 7
  • 57% of Qwen3-Coder’s unsuccessful runs involved loops or budget exhaustion, even though it achieved the highest overall resolution rate among evaluated models.The progress-related failures involved stalled execution, ineffective repeated actions, or exhausted step or token budgets.Source: §IV · pp. 6–7
  • Devstral-S’s 405 failures included 35% wrong or incomplete fixes, 21% loops or budget exhaustion, 20% insufficient validation, and 19% fault mislocalization; GLM-4.7-F’s 356 failures were led by 36% wrong or incomplete fixes and 30% loops.These model-level tendencies recur across harnesses, but their severity varies by pairing.Source: §IV · p. 7

Agent harness and pairing effects

  • 95% of Qwen3-Coder’s unsuccessful SWE-agent runs involved loops or budget exhaustion, versus 64% with AutoCodeRover, 51% with OpenCode, and 32% with OpenHands.With SWE-agent, the corresponding shares were 95% for Qwen3-Coder, 35% for GLM-4.7-F, and 25% for Devstral-S; AutoCodeRover’s shares were 64%, 43%, and 11%.Source: §IV · pp. 7–8
  • 41% of Devstral-S’s Agentless failures involved insufficient validation, versus 24% with OpenHands and 12% with mini-SWE-agent; mini-SWE-agent also had 50% loop failures and left 78 tasks unresolved, versus 7% and 46 under OpenHands.A lower share for one failure category therefore does not by itself indicate better overall performance; pairing-specific profiles are needed to identify prominent mechanisms.Source: §IV · p. 8
  • 3%–95% is the range of loop or budget failures across Qwen3-Coder pairings; across models paired with SWE-agent the range is 25%–95%, while Devstral-S validation failures range from 4% to 41%.Pairing can attenuate or amplify a model-level tendency and redistribute failures among mechanisms.Source: §IV · p. 8

V. RQ3: PAIRING EFFICIENCY

Token efficiency depends substantially on the harness–model pairing, and fewer model calls do not necessarily mean fewer tokens per resolved task. The section compares proxy-metered expenditure across evaluated pairings and estimated frontier references, with those reference comparisons limited to token efficiency.

V. RQ3: PAIRING EFFICIENCY

  • 5.1 million tokens per resolved task is Devstral-S’s aggregate across harnesses, compared with 6.0 million for GLM-4.7-F and 19.8 million for Qwen3-Coder.These aggregate figures pool tokens and resolved-task counts across harnesses.Source: §V · p. 8
  • 87.8 million tokens per resolved task for Qwen3-Coder, 11.0 million for GLM-4.7-F, and 6.7 million for Devstral-S are the reported SWE-agent values.Under mini-SWE-agent, GLM-4.7-F has the lowest expenditure at 6.2 million, followed by Qwen3-Coder at 9.5 million and Devstral-S at 9.9 million.Source: §V · p. 8
  • 62 versus 92 median calls under mini-SWE-agent correspond to Devstral-S and GLM-4.7-F, but their tokens per resolved task are 9.9 versus 6.2 million, respectively.Efficiency depends on token volume per call and tasks resolved, not call count alone.Source: §V · p. 8
  • 50 to 169 median calls and 4.8 to 87.8 million tokens per resolved task are Qwen3-Coder’s changes from AutoCodeRover to SWE-agent.The section reports that harness choice changes median calls by 2.6–3.4× and charged tokens per resolved task by 9–18× within the same model.Source: §V · pp. 8–9
  • 68 tasks at an estimated 2.8 million tokens per resolved task for OpenHands with Claude 4 Sonnet and 70 tasks at 1.0 million for OpenHands with GPT-5 are the frontier reference values.13 of 18 lightweight pairings require more tokens per resolved task than both references; the comparison is indicative because frontier token counts are estimated and configurations differ.Source: Figure 4 · p. 8, §V · p. 9
  • 1.1 to 87.8 million charged tokens per resolved task span iterative harness pairings, nearly an 80-fold range; 13 of 18 pairings resolve fewer tasks while consuming more tokens per task than both estimated frontier references.Higher expenditure does not consistently yield higher resolution accuracy.Source: Figure 4 · p. 8, §V · p. 9

A. Implications for Coding-Agent Deployment

Deployment choices should be evaluated at the harness–model pairing level, using both resolution performance and the tasks resolved on the intended workload. The guidance also calls for budgeting against resolved tasks and using pairing-specific failure evidence to guide harness investigation.

A. Implications for Coding-Agent Deployment

  • A model can perform substantially better under another harness, and a leading harness does not retain its advantage across all models; selection should consider both resolution and intended-workload task coverage.Neither a weak result nor a leading score should be treated as intrinsic to either component alone.Source: A. Implications for Coding-Agent… · p. 9
  • Lightweight models are not necessarily token-efficient, so execution budgets should account for unsuccessful-run expenditure and be assessed against resolved tasks.Pairing-specific failure evidence can guide investigation of progress control, fault localization, and validation; these are potential priorities, not verified remedies.Source: A. Implications for Coding-Agent… · p. 9

B. Threats to Validity

The study notes limits to causal attribution and outcome interpretation, alongside boundaries on generalizing its findings across tasks, systems, and serving conditions.

1) Internal and Construct Validity:

  • A single final attempt per pairing–task combination prevents estimation of run-to-run variance, while execution limits and timeouts may affect outcomes.These factors limit attribution of resolution, failure profiles, and token expenditure to harness design alone.Source: 1) Internal and Construct Validity · pp. 9–10
  • Failure labels involve interpretation, especially for insufficient validation and wrong or incomplete fixes, and uncertainty may affect aggregate failure profiles.Independent model review does not eliminate this uncertainty.Source: 1) Internal and Construct Validity · p. 10

2) External Validity:

  • Resolution rates describe a fixed, stratified 100-task SWE-bench Verified subset, not full-benchmark performance.Rankings and effect sizes may differ across systems or versions, non-Python projects, task distributions, and activities beyond repository-level issue resolution.Source: 2) External Validity · p. 10
  • Different sampling settings, quantization schemes, context limits, caching, serving engines, hardware, and completion-token thresholds may alter performance, failure profiles, efficiency, or timeout-constrained outcomes.All models were served through vLLM on a single NVIDIA H100 NVL GPU.Source: 2) External Validity · p. 10

VII. RELATED WORK

Prior evaluations and trajectory studies characterize selected configurations, while this study compares harnesses and models systematically. Its findings show pairing-dependent performance and motivate further work to diagnose interactions that shape outcomes.

VII. RELATED WORK

  • Prior SWE-bench evaluations characterize selected harness–model configurations, while the full-factorial comparisons show which component-level strengths persist and which depend on pairing.Trajectory studies examine thought–action–result patterns and repair processes; this study compares failure profiles across the pairing matrix.Source: §VII · p. 10
  • Harness adaptation studies show that harness changes can improve performance without replacing the model, while this pairing matrix compares existing harnesses’ task coverage, failure profiles, and token efficiency.The comparisons provide evidence for prioritizing model-specific harness improvements and inform model–harness co-evolution.Source: §VII · p. 10

VIII. CONCLUSION AND FUTURE WORK

  • Harness choice changes resolution accuracy by up to 44 percentage points for the same model, and no harness performs best across all models.Across 18 pairings, even strong pairings differ in the tasks they resolve; among iterative configurations using the same model, harness choice changes model calls by 2.6–3.4× and charged tokens per resolved task by 9–18×.Source: §VIII · p. 10
  • Future work could use controlled interventions to identify which interactions between model capabilities and harness policies drive success, failure, and token expenditure.A proposed co-evolution loop would adapt the harness or train the model, evaluate the revised pairing, and use the evidence to guide the next change.Source: §VIII · p. 10

Related papers

Loading 2610.32131v1…