PaperlayerView exact revision on arXiv

Source-linked AI summary

SERA: Soft-Verified Efficient Repository Agents

Ethan Shen, Daniel Tormoen, Saurabh Shah, Ali Farhadi, Tim Dettmers

arXiv:2601.20789v3cs.CLcs.LGcs.SE

TL;DR

Training open-weight coding agents for private repositories has been costly and complex. SERA introduces SVG, which generates synthetic trajectories from any repository without unit tests and trains agents with supervised finetuning. It achieves leading fully open-source results, matches strong open-weight models, and reaches equivalent performance at much lower cost.

  • Problem

    High training cost and complexity have kept the repository-specialization advantage of open-weight coding agents largely theoretical.

  • Method

    SERA uses Soft Verified Generation to create trajectories from any repository without unit tests, then trains a coding agent using supervised finetuning.

  • Results

    49.5%/54.2% on SWE-bench Verified at 32K/64K context makes SERA state-of-the-art among fully open-source models, with equivalent performance costing 26× less than SkyRL and 57× less than SWE-smith.

  • Takeaways & Limitations

    Repository specialization becomes practical at low cost, including agents that can match or exceed the teacher model on a target codebase.

  • Takeaways & Limitations

    The authors evaluate only on SWE-bench Verified and do not know how broadly the model performs across other coding benchmarks or tasks.

Abstract

from arXiv · show

Open-weight coding agents should hold a fundamental advantage over closed-source systems because they can specialize to private codebases, encoding repository-specific information directly in their weights. Yet the cost and complexity of training has kept this advantage theoretical until now. We present Soft-Verified Efficient Repository Agents (SERA), an efficient method for training coding agents that enables the rapid and cheap creation of agents specialized to private codebases. Using Soft Verified Generation (SVG), we generate thousands of trajectories from any code repository, without requiring unit tests. Beyond repository specialization, we apply SVG to a larger corpus of codebases, generating 200,000+ synthetic trajectories. Using only supervised finetuning (SFT), SERA achieves leading results among fully open-source (open data, method, code) models while matching the performance of open-weight models like Devstral-Small-2. Creating SERA models is 26x cheaper than reinforcement learning and 57x cheaper than previous synthetic data methods to reach equivalent performance. We use our dataset to provide detailed analysis of scaling laws, ablations, and confounding factors for training coding agents. Overall, we believe our work will greatly accelerate research on open coding agents and showcase the advantage of open-source models that can adapt to private codebases. We release SERA as the first model in Ai2's Open Coding Agents series, along with all our code, data, and Claude Code integration to support the research community.

1 Introduction

SERA makes repository-specialized open-weight coding agents practical by replacing costly training infrastructure with efficient synthetic trajectory generation and supervised finetuning. SVG supports broad data generation without test infrastructure, while SERA achieves strong open-source performance at substantially lower cost.

  • SERA addresses the high cost and complexity that have kept repository specialization for open-weight coding agents largely theoretical.
  • $2,000 trains a 32B coding agent with state-of-the-art fully open-source results using simple supervised finetuning.This corresponds to 40 GPU days.
  • Soft verification matches full test-based verification in training-data quality while removing the need for test infrastructure and enabling generation from any repository.
  • 49.5%/54.2% on SWE-bench Verified at 32K/64K context makes SERA state-of-the-art among fully open-source models.
  • 26× lower cost than SkyRL and 57× lower cost than SWE-smith achieves equivalent performance when self-hosting inference via vLLM.Using the z.ai API, the advantages increase to 53× and 115× respectively.
  • SERA releases code, data, and Claude Code integration alongside more than 200,000 synthetic coding-agent trajectories.

2 Background

Coding-agent training uses synthetic teacher-student data generation or reinforcement learning, with long trajectories and tool calling adding practical difficulty. Existing approaches trade infrastructure simplicity against dependence on teacher capability or the instability and complexity of online reinforcement learning.

  • 32K tokens or more of context and reliable tool calling are practical requirements that make coding-agent experimentation slower and more expensive.
  • Synthetic data generation distills trajectories from a strong teacher solving programmatically constructed tasks into a smaller student model.This separates data generation from training so each can be optimized independently.
  • SWE-smith injects bugs, generates issue descriptions, solves them, and verifies patches with tests, requiring valid bug generation and execution infrastructure.
  • $0.52 is the approximate cost of each SWE-smith trajectory using Sonnet 3.7, excluding the additional complexity and iteration slowdown from CPU-based test execution.
  • Teacher-student methods are largely bounded by teacher capability, so synthetic data may be insufficient for frontier progress when no stronger teacher exists.Students can sometimes exceed teachers, but the gains are described as modest.
  • Reinforcement learning can improve through self-play but requires online rollouts, sandboxed execution, distributed coordination, and sensitive reward and optimization settings.

3 Method

SVG generates repository-specific coding trajectories through two teacher rollouts: an initial change is transformed into a synthetic pull request, then reproduced and scored by patch overlap. This design broadens synthetic tasks beyond test failures while avoiding test infrastructure.

  • SVG treats a valid synthetic pull request as any instruction seeking a desired repository change, not only a fix for a failed unit test.
  • First Rollout: In the first rollout, the teacher receives a randomly selected function and sampled bug type, producing trajectory T1 and patch P1.The pipeline uses 51 bug types and runs once for every function in the codebase.
  • Synthetic PR: The first trajectory is converted into a synthetic pull request using its traces and patch plus a demonstration pull request.
  • Second Rollout: In the second rollout, the teacher receives only the synthetic pull request and produces trajectory T2 and patch P2 while attempting to reproduce the initial change.
  • Soft Verification: Soft verification scores P2 against P1 using line-level recall: r=1 is hard-verified, 0<r<1 is soft-verified, and r=0 is unverified.
  • Setup: SVG uses open-weight and locally deployable teacher infrastructure to support scalable, reproducible coding-agent experiments.The experiments use GLM-4.5-Air as teacher and 121 codebases for data generation.

4 Main Results

SERA is evaluated across controlled comparisons, scaling studies, and repository-specialization benchmarks. It achieves leading fully open-source performance, remains competitive with open-weight systems, and shows that repository-specific data improves specialization efficiency.

  • Controlled Comparisons: SERA outperforms SWE-smith in a head-to-head comparison using the same teacher model, sample size, verification method, and evaluation context.The comparison trains on hard-verified trajectories, controlling the data distribution for synthetic issue descriptions and working code.
  • Controlled Comparisons: SERA nearly matches BugPilot despite using approximately 10% fewer training samples.The comparison uses BugPilot BaseMix, whose data is not public, at 64K context.
  • Data Generation: SERA simplifies large-scale data generation by avoiding synthetic bug injection and unit-test execution.The method uses this property to generate datasets from repositories with GLM-4.5-Air and GLM-4.6 teachers.
  • Main Results: SERA achieves a new state-of-the-art among fully open-source 32B models at 32K context on SWE-Bench Verified.At 64K context, it again leads fully open-source models and matches some open-weight systems, although Devstral-Small-2-24B outperforms it among similarly sized models.
  • Evaluation Considerations: At 64K evaluation, SERA underperforms some baselines because it was trained only at 32K context.The evaluation methodology treats context length as a major comparison factor because it affects both memory cost and model performance.
  • Repository Specialization: Repository-specific data improves specialization: pure Django data matches GLM-4.5-Air at 51.2% with 8,000 samples, whereas pure general data fails to match it at 16,000.Intermediate mixtures show higher asymptotic performance as the specialized-data proportion increases.

5 Ablations and Analysis

SERA’s ablations examine verification, truncation, specialization, filtering, mixing, and teacher-model choices. The results show that soft or absent verification can suffice, high truncation ratios and ordered truncation help, repository-specific data matters, and reasoning traces and trajectory mixing affect performance.

  • 5.1 Verification: Verification thresholds from r = 0.0 to r = 1.0 perform similarly, including unverified T1 trajectories, through 7,400 samples.The study finds no significant downstream benefit from stricter verification in this regime.
  • 5.1 Verification: Line-level recall correlates extremely highly with unit-test resolution across 1,036 SWE-Bench Verified evaluations.The evaluations comprise over 500,000 individual tasks.
  • 5.3 Truncation: 24.83% of Sera-4.5A-Lite trajectories exceed 32K tokens, motivating truncation for base-model training.SWE-smith has 23.26% of trajectories above the same context limit.
  • 5.3 Truncation: Trajectories with high but incomplete 32K-token coverage perform best, while performance decreases as truncation ratios fall.The authors suspect longer trajectories reflect harder tasks and that later steps focus on final changes.
  • 5.3 Truncation: 43.00 versus 37.47: ordered truncation significantly outperforms random truncation, t(4) = 4.81, p = .009, d = 3.93.The result supports preserving earlier trajectory turns rather than truncating randomly.
  • 5.5 Teacher Models: GLM-4.5-Air is a better teacher than Claude 3.7 Sonnet despite similar SWE-Bench Verified performance, and removing reasoning traces significantly degrades results.The findings associate stronger teacher data with longer, more elaborate reasoning traces.
  • 5.6 Dataset Mixing: Combining 16,000 T2 and 9,224 T1 trajectories improves performance over using 16,000 T2 trajectories alone, with little to no degradation from mixing.The mixed-data result supports improved sampling efficiency during specialization.

6 Robustness of Evaluations

The evaluation analysis emphasizes that run-to-run variance can match reported improvements, so multi-seed testing, uncertainty reporting, cross-model checks, and scaling-law fits are important for reliable conclusions.

  • Statistical Analysis: 78 experimental conditions evaluated with three random seeds each yield 234 individual evaluations across the paper’s analyses.The aggregation covers scaling laws, verification, truncation, specialization, filtering, and baseline comparisons.
  • Observed Variance: 0.5% to 3.0% standard deviations, with a 1.2% median, overlap the typical 1–3% improvement size in coding-agent research.Many reported gains fall within one standard deviation of run-to-run noise.
  • Signal-to-Noise Analysis: SNR between 1–2 is borderline, while SNR > 2 indicates a likely significant effect and SNR < 1 means noise dominates.The framework divides absolute method differences by typical run-to-run variance.
  • Signal-to-Noise Analysis: SNR > 3 supports high confidence for specialized versus general data (+4.3%) and SERA versus SWE-smith with the same teacher (+4.7%).Scaling-law predictions have mean error 0.4%.
  • Seed Requirements: With only 3 seeds, improvements below 2–3% should be treated skeptically under the observed 1.2% median standard deviation.Table 10 estimates the seeds needed to reach SNR ≥2 for different effect sizes.
  • Single-Seed Problem: Different random seeds can reverse the preferred truncation setting: seeds 1 and 2 favor 0.95, while seed 3 favors 0.92 and makes 0.95 2.2% worse.The authors conclude that single-seed ablations cannot be trusted.
  • Cross-Model Generalization: SWE-smith falls from 32.6% with Qwen 2.5-32B to 25.3% with Qwen 3-32B, while base-model generalization for SERA remains untested outside Qwen and GLM families.The authors caution that observed improvements may not transfer broadly.
  • Scaling Laws: Scaling-law fits achieve R2 > 0.95 with mean prediction error of 0.4%, supporting their use for robustness checks and cost estimation.The authors also recommend at least three seeds and reporting standard deviations.

7 Deployment

SERA can be deployed behind a lightweight proxy that connects Claude Code to an OpenAI-compatible serving endpoint. Deployment requires exact SWE-agent tool-format compatibility, translation, and path normalization.

  • Deployment: The release includes a lightweight proxy server that enables Claude Code to use SERA as its backend.The proxy section focuses on practical implementation considerations.
  • Tool Format Compatibility: Exact SWE-agent tool-format matching is required because different scaffolds or subtle formatting changes significantly degrade performance.The proxy translates Claude Code’s Read, Edit, Write, and Bash tools to SWE-agent’s formats and normalizes paths.
  • Infrastructure: The proxy connects Claude Code to any OpenAI-compatible SERA endpoint and supports vLLM, Modal, cloud GPUs, and on-premises infrastructure.It handles API translation, tool mapping, and response streaming.

8 Related Work

Related work spans synthetic data generation, environment construction, training procedures, benchmarks, agent architectures, and reinforcement learning for software-engineering agents.

  • Synthetic Data Generation: Training-data generation methods synthesize bugs, debugging workflows, or repair trajectories using language-model agents and test cases.Examples include BugPilot, SWE-Synth, and SWE-Mirror.
  • Environment Construction: Environment-construction methods build executable repositories or training environments through feedback, multi-agent collaboration, or memory pools.Examples include SWE-Gym, Repo2Run, SWE-Factory, and RepoST.
  • Training Methods: Recent training-process work improves supervised fine-tuning, synthesizes projects from scratch, or defines agent-data protocols without necessarily using reinforcement learning.Examples include SWE-Lego, SWE-Playground, and Agent Data Protocol.
  • Evaluation Benchmarks: SWE-bench evaluates coding agents on real GitHub issues and pull requests, while SWE-rebench and Multi-SWE-bench extend benchmark coverage and contamination handling.SWE-bench remains useful for broad method and model comparisons.
  • Agent Architectures: Agent architectures include SWE-agent’s repository-navigation interface, OpenHands’ modular standardized interfaces, and Agentless’ alternative approach.These systems differ in their interfaces and execution organization.
  • Training Methods: Training approaches include reinforcement learning with rule-based rewards, efficient multi-turn RL infrastructure, and retrieval-plus-editing systems.Examples include SWE-RL, SkyRL-Agent, and SWE-Fixer.

9 Limitations

The paper identifies uncertainty about how broadly its empirical findings generalize, especially across verification levels, repository types, model families, and evaluation settings.

  • Verification: Soft verification may stop sufficing for larger models or more training data, where hard verification with correct code could become essential.The authors could not test this hypothesis at their scale.
  • Teacher matching: Benefits beyond teacher performance are modest and may level off, with stronger teachers potentially becoming more efficient than generating additional data.The authors could not verify whether the advantage scales further because of compute limitations.
  • Evaluation scope: Evaluation is limited to SWE-bench Verified, so broader performance across coding benchmarks and tasks remains unknown.The model also exhibits undesirable behaviors in the authors’ coding-task use, including attempting to call a nonexistent submit tool.
  • Specialization scope: Private-repository specialization is demonstrated only on Django, Sympy, and Sphinx, which may have appeared in base-model training data.The authors lack evaluation data for truly private codebases that models have never seen.
  • Statistical robustness: Some comparisons use only n = 3 seeds, so effects smaller than 3% may be noise rather than genuine improvements.The authors recommend skepticism toward small differences.
  • Model scope: All experiments use Qwen-3 base models with GLM-4.5-Air or GLM-4.6 teachers, leaving generalization to other model families uncertain.Experiments with Claude models only hint at broader generalization.

10 Broader Impact

SERA is presented as a way to lower the resource barrier to coding-agent research and enable local specialization to private codebases. The release also emphasizes reproducibility through public research components.

  • Enabling Private Codebase Specialization: The authors expect private specialization to affect how small organizations use coding agents.This impact is presented alongside the broader claim that the release can support lower-resource research.
  • Democratizing Coding Agent Research: SERA aims to democratize coding-agent research by reducing the cost and infrastructure complexity associated with competitive training.The paper contrasts its approach with reinforcement-learning systems requiring teams of 12 or more researchers, 64+ GPUs, and months of engineering.
  • Open Science and Reproducibility: The Ai2 Open Coding Agents initiative is positioned as a shared foundation for research without recreating expensive infrastructure from scratch.SERA is described as the initiative’s first release.
  • Enabling Private Codebase Specialization: SERA enables individuals and small companies to train small local models on private codebases instead of sending proprietary code to cloud providers.The paper frames this as an alternative workflow addressing privacy and intellectual-property concerns.
  • Open Science and Reproducibility: The release includes training data, generation code, model weights, and evaluation scripts for replication and extension.Cost analyses and scaling laws are also intended to support resource planning and experimental design.

A Scaling Law and Data Points

The paper fits cost-performance power laws for SERA and uses them to estimate the cost of matching baseline performance. The fits distinguish inference-cost regimes and rely on extrapolation beyond observed data.

  • Scaling Law: The scaling law models SWE-bench Verified resolve rate as a power-law function of total training cost.Total cost includes data generation and training.
  • Scaling Law: The fitted parameters represent the asymptotic performance ceiling, initial distance below that ceiling, and the rate of diminishing returns.The parameters are denoted c, a, and b, respectively.
  • Cost Regimes: Separate fits use vLLM self-hosting at $0.187/sample and the z.ai API at $0.092/sample because identical sample counts imply different costs.Each cost regime therefore receives different fitted parameters.
  • Baseline Matching: The paper estimates matching Devstral-Small-2 at $7K with z.ai or $15K with vLLM, and matching GLM-4.5-Air at $9K or $19K, respectively.The baseline resolve rates are 50.0% for Devstral-Small-2 and 50.5% for GLM-4.5-Air.
  • Extrapolation: The fitted asymptote is approximately 70%, but this extrapolation extends well beyond the observed data range and is uncertain.The estimates come from solving the scaling law at target performance levels.
  • Data Points: The underlying experiments use Qwen 3-32B, GLM-4.5-Air-generated SERA data, 32K context, and three random seeds.Table 11 reports the exact data points used for the curve.

C Specialization Results at 64K Context

At 64K context, SERA models underperform competing baselines despite matching or exceeding them at 32K. The paper attributes this gap to a context-length mismatch during training.

  • 64K Evaluation: SERA models trained at 32K context underperform baselines at 64K evaluation, despite matching or exceeding them at 32K.The comparison concerns specialization results and competing models such as Devstral-Small-2 trained at longer contexts.
  • Context-Length Mismatch: The performance gap is attributed to SERA models not learning to effectively use the additional context available at 64K tokens.Competing models were trained at 64K or longer contexts.
  • 64K Evaluation: Table 13 reports specialization results at 64K context, averaged over three seeds.The table contrasts fine-tuned SERA models with longer-context competitors.

D Cost Breakdown

SERA’s cost analysis compares reinforcement learning and synthetic-data training across self-hosted and API inference. The results emphasize substantially lower costs for SERA, while API-based estimates depend on changing provider conditions.

  • Reinforcement Learning: 4,601 H100-hours ($9,202) trains SkyRL-Agent to 39.4% on SWE-bench Verified, while SERA matches DeepSWE’s performance at 960 H100-hours ($1,920).DeepSWE requires 9,180 H100-hours ($18,360) for similar performance.
  • Reinforcement Learning: $352 via self-hosted vLLM or $173 via the z.ai API reaches SkyRL’s 39.4% performance, corresponding to 26× or 53× cost-to-performance efficiency.These estimates come from SERA’s fitted cost-performance scaling curve.
  • Synthetic Data Generation: 749K cached-context tokens across 35 API calls dominate API-based inference costs.The cumulative context accounts for 54.4% of inference cost, while output tokens total 7.7K per trajectory.
  • Synthetic Data Generation: $0.1699 per trajectory makes GLM-4.6 through z.ai 3.1× cheaper than SWE-smith and 1.1× cheaper than vLLM self-hosting.GLM-4.5-Air through z.ai is 5.7× cheaper than SWE-smith and 2.0× cheaper than vLLM self-hosting.
  • Scaling Costs: $6K in GLM-4.6 data generation via z.ai is predicted to match Devstral-Small-2, versus $23K with GLM-4.5-Air and $47K using vLLM self-hosting.At 16,000 trajectories, generation costs are approximately $1.5K for GLM-4.5-Air via z.ai, $2.7K for GLM-4.6, $3.0K via vLLM, and $8.4K via Sonnet 3.7.
  • Caveats: Commercial API estimates are less reproducible because pricing, model quality, rate limits, and availability may change without notice.The paper encourages vLLM for rigorous scientific work requiring full reproducibility.
Loading 2601.20789v3…