One to More, More to One: Category-Aware Iterative Expert Training for Software Engineering Agents

1 Introduction

• Pooled updates can help one task category while hurting another, and aggregate metrics can conceal these opposing changes; the paper calls this the category see-saw.Prior expert-splitting and fusion methods use conventional domain boundaries, whereas this work organizes experts within repository-level SWE using observable task categories.Source: §1 · p. 2
• Unlike pipeline decomposition into stages such as localization and repair, this approach partitions the training distribution into categories, develops category-specific experts, and then integrates them.The paper describes this category-based decomposition as complementary to workflow-stage approaches such as Agentless [Xia et al., 2024], SWE-Fixer [Xie et al., 2025], and AutoCodeRover [Zhang et al., 2024].Source: §1 · p. 2
• The study organizes repository-level SWE tasks into semantic categories, trains category-specific RL experts, and integrates them into one policy through label-routed multi-teacher on-policy distillation.Its stated objective is to improve overall and per-category resolution in a single deployable policy.Source: §1 · pp. 2–3
• Initial expert RL improves average training-instance success but leaves regressions, motivating Refresh–Repair–Expand (RRE) to refresh mastery, reuse verified successes for Repair SFT, and select a new training frontier.The experts alternate stable long-horizon agentic RL with mastery refresh and Repair SFT, without external models supplying solution trajectories or action targets.Source: §1 · pp. 2–3
• SWE Labeler uses evidence-grounded definitions and explicit rules to label issues, changes, and agent trajectories across hierarchical semantic and ordinal scale axes.The labels support category analysis, training-pool construction, and teacher routing.Source: §1 · p. 3

Executable data and training environments for SWE agents

• Executable repository tasks support category-expert training, while audits and runtime anti-hacking controls protect outcome-reward integrity.Prior SWE environments include collected, generated, and hybrid-verified tasks; audits identify specification, environment, grading, and reward-hacking failures, including incorrect patches receiving positive verifier outcomes.Source: §2 · p. 4

Post-training software engineering agents

• Within SWE post-training, separate evidence-grounded expert streams are progressively refined through agentic RL and Repair SFT on each expert’s successful rollouts.Prior approaches span trajectory supervision, SFT-centered pipelines, patch-similarity RL, execution-feedback RL, and selective expert assistance rather than fine-grained category-specific refinement.Source: §2 · p. 4

Reinforcement learning for long-horizon agents

• Agentic-miniRL adapts long-horizon RL principles to SWE trajectories with behavior-policy correction, RLOO advantages, a K1 reference penalty, and turn-aware loss reduction.The cited literature addresses tool-interaction credit and exploration, environment complexity, sparse rewards, learning stability, and policy staleness; repository-level studies emphasize direct environment feedback.Source: §2 · p. 4

Heterogeneous SWE tasks and category-aware specialization

• Fine-grained evidence-grounded labels expose category see-saw hidden by aggregate scores and route training and evaluation of specialized SWE experts.Prior work studies variation across broad domains or capabilities; this work focuses on heterogeneity within SWE and derives trainable categories from hierarchical task annotations.Source: §2 · p. 4

Model merging and multi-teacher on-policy distillation

• Label-routed MOPD integrates same-base SWE experts on student-generated long-horizon trajectories, combining routed on-policy distillation with reward extrapolation.This connects parameter-space merging and output-distribution distillation to intra-domain specialization, with observable category labels defining expert granularity and routing.Source: §2 · p. 5

3.1 Repository-level SWE and joint agentic RL

• A repository-level SWE instance combines a repository, base revision, issue statement, reproducible environment, and executable verifier; success requires a patch that passes repair tests without regressions.The agent does not observe the reference patch or hidden verifier outcomes.Source: §3.1 · p. 5

3.2 Empirical motivation: the category see-saw

• Repository-level SWE spans task families with different evidence sources, interaction patterns, and success criteria, so pooled joint RL can progress unevenly across categories.The paper treats these categories as observable operational groupings, not isolated capabilities or proof of conflicting gradients.Source: §3.2 · p. 6
• Pooled RL exhibits a category see-saw: gains in some Pro-618 categories coincide with regressions in others, and overall gains can coexist with a negative minimum-category gain.Figure 2 tracks category and overall gains across checkpoints; aggregate improvement alone does not ensure that every category improves.Source: §3.2 · p. 6, Figure 2 · p. 7

3.3 From category-aware diagnosis to specialization and integration

• A structured SWE label space maps observable instance properties into operational categories, enabling category-level diagnosis and consistent training-pool construction without treating categories as latent capabilities.The label axes include repository domain, maintenance intent, and modification scope; the main experiments use a frozen Domain L1 rule for Pro-A/B/C.Source: §3.3 · pp. 7–8
• The study targets a single deployable policy with stronger aggregate and category-level performance than Pooled RL and Balanced RL, using category-specific expert streams followed by label-routed MOPD.Categories determine training-pool construction and teacher routing, while the deployed agent receives ordinary task and tool observations.Source: §3.2 · p. 7, §3.3 · p. 8

4.1 Executable SWE task construction

• The executable task pool contains approximately 32K candidates built from repository pull requests and linked issues, with patches and executable tests used to validate instances.Instances retain fail-to-pass and pass-to-pass tests after gold-patch checks, with repeated checks filtering flaky or time-decayed verifiers.Source: §4.1 · p. 8

4.2 SWE Labeler: an evidence-grounded hierarchical labeling system

• SWE Labeler defines an evidence-grounded hierarchical taxonomy with interpretable, auditable labels for expert routing, data construction, and category-aware evaluation.It annotates Task Type and Repository Domain alongside three scale axes; the main configuration coarsens 18 routable Domain L1 families into three engineering-context routes, A, B, and C.Source: §4.2 · pp. 8, 9, 11

4.3 Agentic-miniRL

• Agentic-miniRL combines instance-wise RLOO, K1 reward regularization, behavior correction, and turn-aware token-loss reduction for long-horizon SWE trajectories.RLOO uses other rollouts for an instance-conditioned baseline without a critic; the implemented reduction sums token losses within assistant turns, then averages turns per trajectory and valid trajectories.Source: §4.3 · pp. 11–13

4.4 Refresh–Repair–Expand training of category experts

• RRE trains each category expert through alternating Agentic RL and Repair SFT, refreshing mastery after each phase and expanding the task frontier before the next RL phase.Repair SFT reuses the expert’s own verifier-approved successful trajectories; no external model supplies solution trajectories or action targets. Initial experts improved average training success while retaining instance-level regressions, motivating this loop.Source: §4.4 · pp. 13–14
• RRE refreshes mastery through environment verification, repairs with the expert’s own verified successes, and expands an executable task frontier using current mastery and historical success.Repair SFT allocates more successful trajectories to low-mastery instances; it is reward-filtered self-distillation rather than a source of external demonstrations.Source: §4.4 · p. 16

4.5 Label-routed multi-teacher on-policy distillation

• MOPD initializes a student from the base policy, balances the three initial-RL task pools, and routes each student-generated trajectory to its category expert for on-policy teacher queries.Teachers are queried only on prefixes the student visits, and the category map assigns each task to its corresponding expert.Source: §4.5 · pp. 16–17
• MOPD uses only its routed advantage as the student’s signal: environment returns, group baselines, and reference penalties are disabled, with a bounded vanilla policy-gradient update over assistant-turn losses.A finite advantage clip of 5 limits any single token’s influence; unrouted samples receive zero advantage.Source: §4.5 · p. 17
• ReLU-gated reward extrapolation retains positive teacher–reference log-probability gaps, adding a nonnegative reference-anchored direction to imitation without changing routing or aggregation.The audited configuration sets each teacher’s extrapolation coefficient to 1.25.Source: §4.5 · pp. 17–18

5.2 Base model and executable agent

• All principal policies start from Qwen3.6-27B and use an executable repository agent that submits patches scored by fail-to-pass and pass-to-pass tests.Training and Pro-618 evaluation use the R2E-Gym scaffold [Jain et al., 2025]; SWE-bench Multilingual uses SWE-agent [Yang et al., 2024].Source: §5.2 · p. 18, §5.3 · p. 18

5.3 Benchmarks and evaluation configurations

• Pro-618 is the primary benchmark for checkpoint selection, category analysis, and final comparisons; SWE-bench Multilingual evaluates fixed final models on all 300 instances.Pro-618 uses 221/201/196 instances in Pro-A/B/C; the multilingual mapping assigns 72/25/176 to A/B/C, with 27 unassigned but retained in Full.Source: §5.3 · pp. 18–19

5.4 Pro-618 construction and audit

• Pro-618 is an audited 618-instance SWE-bench Pro subset intended to reduce distortions from defective specifications, ground truths, graders, and prompt–test mismatches.The common routing rule divides it into Pro-A/B/C with 221/201/196 instances; the broader Pro task profile includes more cross-module changes than SWE-bench Verified.Source: §5.4 · pp. 19–20

5.5 Training data and category-conditioned curricula

• The 6,723-task joint-RL corpus combines 3,048 synthetic instances with 3,675 SWE-rebench V2 tasks; Pooled RL uses all tasks, while Balanced RL samples 516 per category.The shared category mapping assigns 2,769 tasks to A/B/C; RRE refreshes mastery, samples the expert’s own verified successes for Repair SFT, then selects further RL tasks. MOPD balances Stage 1 expert pools by repeating A/B records to match C, without adding tasks.Source: §5.5 · pp. 20–21

5.6 Training infrastructure and optimization

• RL phases share one optimization configuration across categories, while Repair SFT trains each expert on its own verified trajectories and MOPD distills from three frozen experts.MOPD uses label-routed on-policy distillation with imitation and extrapolation coefficients of 1 and 1.25, respectively, and a ReLU gate.Source: §5.6 · p. 21

5.7 Reward integrity and anti-hacking protocol

• The protocol distinguishes verifier exploitation from solution leakage and uses layered controls to protect executable reward integrity.Controls address task-construction leakage, repository state, verifier replay in a fresh sandbox, and trajectory-level detection of external answer retrieval; confirmed retrieval receives zero while remaining in the denominator.Source: §5.7 · p. 22

5.8 Baselines and comparison settings

• Pooled RL and Balanced RL are evaluated across scheduled checkpoints, experts are selected by target-category resolution at each stage, and MOPD is compared with the base, joint-RL policies, and routed experts.Training-instance mastery diagnostics are separate from benchmark checkpoint selection and identify which previously regressed instances received repair examples.Source: §5.8 · pp. 23–24

5.9 Metrics, uncertainty, and failure accounting

• Final comparisons report overall and category-wise resolution; minimum category lift summarizes the weakest category gain over Pooled RL, with Balanced RL also used as a comparator.Endpoint scores use three evaluation rounds with fixed task denominators; SWE-bench Multilingual Full-score differences additionally receive paired instance-bootstrap 95% intervals.Source: §5.9 · p. 24

6.1 Category dynamics under Pooled RL and Balanced RL

• Balanced RL uses 1,548 versus Pooled RL’s 6,723 distinct training instances and achieves close endpoint Full resolution (55.34% versus 55.50%), while raising average minimum category gain Gsim (0.19 versus 0.08 points).Average SSG is 1.18 versus 1.05 points for Balanced RL and Pooled RL, respectively; category gains are not consistently sustained together over training.Source: §6.1 · pp. 25–26

6.2 Category-expert evolution through RRE

• Initial category-specific RL raises target-category scores to 53.54%, 55.06%, and 52.55% for A/B/C, yet 839 of 2,769 matched training records decrease despite average category progress.The record-level changes were 1,108 increases, 839 decreases (30.3%), and 822 unchanged; these are changes between sampled estimates, not continuous trajectories or per-instance forgetting tests.Source: §6.2 · p. 26, Table 9 · p. 26, Table 10 · p. 27
• After Repair SFT, mean training success reaches 56.28%, 59.59%, and 52.75% for A/B/C, while many initially regressed instances recover to or above their base rates.Repair SFT reuses verifier-approved successes from the preceding RL phase and allocates more replay to low-mastery instances with available trajectories. Of the regressed records included in repair, 69.8%, 73.0%, and 61.3% for A/B/C recover to or above Base.Source: §6.2 · p. 26, Table 9 · p. 26
• Across four stages, the final RRE experts exceed Base on their target categories by 7.84, 4.48, and 5.61 points for A/B/C, with further gains after expanded RL and the second repair.The second repair endpoint improves on the first repair endpoint by 2.26, 2.32, and 2.04 points for A/B/C; within-stage raw rollout means also rise in both RL stages, though trajectories fluctuate and include local regressions.Source: §6.2 · pp. 27–28, Figure 6 · p. 27, Table 11 · p. 28

6.3 Integrating expert gains into a single policy

• 58.04% Full resolution on Pro-618 (+5.39 points over Base) is MOPD’s final mean; it exceeds Pooled RL by 2.54 points overall and Balanced RL by 2.70, with higher A/B/C means than both.The MOPD gains over Pooled RL are 3.32, 2.65, and 1.53 points on A/B/C; over Balanced RL they are 3.62, 2.32, and 2.04 points.Source: §6.3 · pp. 28–29, Table 12 · p. 29
• MOPD retains 80.8%, 111.1%, and 84.8% of expert gains on A/B/C, respectively, with the lowest proportional recovery on A.The student is 0.50 points above the B expert, but 1.51 and 0.85 points below the A and C experts; it still outperforms both joint-RL baselines in all three categories.Source: §6.3 · p. 29, Figure 8 · p. 30
• MOPD achieves the largest Pro-618 Full gain (5.39 points) and minimum category gain Gsim (4.76 points), while SSG is higher than both joint-RL baselines.Its minimum category gain exceeds Pooled RL and Balanced RL by 2.44 and 2.11 points; the higher gains across categories coexist with less uniform gains than Balanced RL (SSG 0.63 versus 0.04 points).Source: Table 12 · p. 29, Table 13 · p. 30, §6.3 · p. 30
• 59.00% Full resolution on SWE-bench Multilingual (+2.78 points over Base) extends MOPD’s aggregate advantage beyond Pro-618; it also exceeds both RL baselines in aggregate and routed categories A/B/C.MOPD’s gains over Pooled RL are 3.44 points overall and 3.70, 10.67, and 3.22 on A/B/C; over Balanced RL they are 2.00 overall and 2.78, 8.00, and 2.08 on A/B/C.Source: §6.3 · pp. 30–31, Table 14 · p. 31

7 Discussion

• Category decomposition is worth exploring when interpretable task groups show persistent learning-trajectory differences obscured by aggregate performance, rather than as an assumed advantage.Category gains move in opposing directions during parts of training, and balancing the training mixture does not clearly remove the pattern.Source: §7 · p. 31
• RRE treats specialization as iterative: it revisits an expert’s own successful trajectories and refreshes task selection as policy mastery changes.Initial-RL diagnostics show that average training improvement can coexist with declining success on individual instances.Source: §7 · p. 31
• MOPD improves overall resolution and every category over both joint-RL baselines, achieves a higher minimum category gain, but retains expert gains unevenly.Category C has the smallest gain over Base, whereas category A has the lowest expert-gain recovery.Source: §7 · pp. 31–32

8 Limitations

• Hard routing simplifies overlapping task structure, while expert training and teacher forwards add compute and the best K may vary across settings.Same-origin teachers may be too similar to add signal or too specialized and distant for stable integration.Source: §8 · p. 32
• Benchmark pass rates do not fully measure patch quality, and test-driven rewards can invite evaluator hacking or reflect brittle specifications [Baker et al., 2025, Zhao et al., 2026, Rajan, 2026].Sanitization, fresh-sandbox replay, and trajectory audits reduce leakage and tampering risks but do not eliminate pretraining memorization or every weak-verifier failure; Pro-618 results do not generalize to the unexecuted full Pro set.Source: §8 · p. 32
• The experiments evaluate the complete framework and stage-wise outcomes but do not isolate optimization components, replay choices, or routing granularity.Single-run category trajectories include evaluation variability, and broader sampling and routing comparisons remain open.Source: §8 · p. 32
• Coding agents can improve developer productivity but may produce insecure changes, automate vulnerability discovery, or pass tests without maintainable patches.Release decisions should consider provenance, licenses, contamination checks, unsafe-code handling, and capability-proportionate safeguards.Source: §8 · p. 32

• 58.04% mean resolution on Pro-618 and 59.00% on SWE-bench Multilingual; the final policy exceeds Pooled RL and Balanced RL overall and in all three categories on both benchmarks.Source: §9 · p. 32
• SWE Labeler defines evidence-grounded task categories, while Agentic-miniRL and RRE develop same-origin experts through executable-reward learning, mastery refresh, and replay of their own successful trajectories.Label-routed MOPD then consolidates the experts into one deployed policy.Source: §9 · p. 32
• Expert-gain recovery on Pro-618 remains uneven across categories, indicating room to improve single-model integration.The framework treats task categories as units for organizing post-training, with the practical goal of retaining expert gains in one SWE agent.Source: §9 · p. 32

A Evaluation benchmark construction and audit

• Pro-618 filters the 731 SWE-bench Pro instances using released instance-level evidence because audit defects can alter measured performance and model rankings [Kim, 2026, OpenAI, 2026].The excluded defects include specification gaps, incorrect reference solutions, and incomplete grading logic.Source: A Evaluation benchmark construction… · p. 38
• The first three exclusion groups follow the pinned public determinacy-audit artifact [Kim, 2026], while the final group follows the official OpenAI audit.The accompanying manifest records task membership and exclusion reasons; counts are reported for each group, with exclusions mutually exclusive at the final stage.Source: A Evaluation benchmark construction… · p. 38, Table 15 · p. 38
• After filtering, Domain L1 routing assigns 221 instances to Pro-A, 201 to Pro-B, and 196 to Pro-C.Source: A Evaluation benchmark construction… · p. 38

B.1 Label inventory

• The experiment artifact exports machine-readable definitions, active prompts, and assignment configuration, while each dataset manifest records the labeling model and source export.Source: §B.1 · p. 38
• Task Type has 26 L1 labels and 119 L2 labels in the current definition file.The L1 inventory spans categories including bug-fix, feature, refactor, security-fix, and user-docs.Source: §B.1 · p. 38
• Domain has 21 L1 labels and 108 L2 labels in the current definition file.The L1 inventory includes web_frontend, database_storage, devops_infra, security_auth, non_coding, and other.Source: §B.1 · p. 38
• Each L2 entry records its name, source attribution, source-derived and operational definitions, observable signals, and boundary notes.L1 labels without L2 children return unspecified rather than an invented label.Source: §B.1 · p. 39

B.2 Taxonomy grounding and structural audit

• The inventory draws on standards, curated catalogs, developer metadata, and framework documentation, but treats none as a ready-made universal taxonomy for SWE-agent training.Stack Overflow clustering is an empirical seed for domain organization, not a gold annotation set; global reasoning adapts cognitive-load inspiration into a SWE rule for tracing data, calls, or state across components.Source: §B.2 · pp. 39–40
• Every admitted label must add a coherent concept, have an attributable source and observable signals, define non-overlap boundaries, and permit residuals to abstain as unspecified.The inventory groups related API transformations and identity-lifecycle operations rather than creating labels for each narrow variant or broad catch-all.Source: §B.2 · p. 40
• All 239 decision records contain six schema fields, and all L1 families and orthogonal dimensions record source frameworks; the frozen 1,531-task audit measures coverage and consistency, not held-out accuracy.The records comprise 119 Task Type L2 labels, 108 Domain L2 labels, and 12 orthogonal levels; the source-framework audit covers 26 Task Type L1 families, 21 Domain L1 families, and three orthogonal dimensions.Source: §B.2 · p. 40
• The three orthogonal axes classify modification scope, cognitive complexity, and estimated time; fault localization, code navigation, test generation, and long-horizon planning are not annotated axes.Each axis has four levels, ranging from single_file to cross_repo, mechanical to design_decision, and trivial_15min to large_4h_plus.Source: §B.2 · p. 40
• Auxiliary Phase-1 metadata records code language, task-spec type, and dependency context to describe interpretive evidence, but these fields are not task categories or part of the expert mapping.Code language uses 21 normalized values, task-spec type is fuzzy or specific, and dependency context is codebase, env, codebase+env, or none.Source: §B.2 · p. 40
• The structural audit found one empty Domain L1 output and six repositories with multiple nonempty Domain L1 labels, including genuinely ambiguous products.The released artifact bundles the labeling model, active prompt, definition files, raw export, and grouping manifest, while the JSONL schema does not independently encode every artifact identifier.Source: §B.2 · p. 40

B.3 Annotation and validation

• Instance annotation uses issue descriptions, repository metadata, gold fixes, and test patches in two phases, assigning L1 labels before selecting L2 labels from those families.These offline annotation inputs are separate from the repair agent’s evaluation input, which excludes the gold fix and test patch.Source: §B.3 · p. 41
• Full-trajectory annotation can retain user requests, edits, tool actions, and test outcomes; outputs are checked against the inventory and hierarchy, and invalid outputs are excluded.Valid existing L1 labels may be preserved before L2 refinement; concrete L2 labels must belong to the resolved L1 family, while unspecified is used alone.Source: §B.3 · p. 41

B.4 Three-route construction and routing specification

• The resulting A/B/C assignment is reused for expert training, category-aware validation, and MOPD teacher routing.Source: Figure 10 · p. 42
• The three-route map uses only Repository Domain L1: each listed value maps to exactly one route, while all other values map to ∅.Table 18 specifies the Repository Domain L1 mapping for the three-route expert configuration.Source: §B.4 · p. 42, Table 18 · p. 42
• Task Type, L2 labels, and scale axes support stratification and analysis but are not additional routing predicates.Source: §B.4 · p. 42

C.1 Agentic-miniRL update order

• The rollout policy first generates multi-turn trajectories while recording inference log probabilities and response-segment masks.This establishes the data and masks used in the later update steps.Source: §C.1 · p. 42
• After environment execution, RLOO is applied to response-level returns as in Equation 11.Source: §C.1 · p. 42
• The centered return is expanded to the terminal token, and the K1 reward is subtracted.Source: §C.1 · p. 42
• Token-level return-to-go is then computed as in Equation 14.Source: §C.1 · p. 42
• Current and old trainer log probabilities are recomputed to form the behavior correction and sign-aware mask in Equation 15.Source: §C.1 · p. 42
• The batch update concludes by optimizing the complete turn-aware loss in Equation 16.Source: §C.1 · p. 42

D Additional training-instance analyses

• Table 19 compares base and post-RL success rates on matched training records, weighting each source record equally and counting records with higher, equal, or lower observed pass rates.Rates use observed successes divided by actual attempts; repeated records share instance-level estimates rather than serving as independent probes.Source: D Additional training-instance analyses · pp. 42–43, Table 19 · p. 43
• Repair recovery covers 751 confirmed SFT records whose initial-RL rates fell below Base, while gain retention covers 404 records whose rates rose under RL and then fell after SFT.Recovery is classified as to-Base, partial, or no improvement; gain-retention outcomes use that cohort’s denominator, not all 580 post-SFT decreases.Source: D Additional training-instance analyses · p. 43, Table 20 · p. 43
• The snapshots are conditional on a pool selected using base-model mastery estimates and cannot establish persistent forgetting or isolate SFT’s causal effect.Finite rollout sampling, regression to the mean, and the distinction between confirmed SFT exposure and a zero-success probe further qualify recovery and retention comparisons.Source: D Additional training-instance analyses · pp. 43–44
• Table 21 decomposes mean training-record gains over Base into positive and negative contributions, each normalized by all matched records, whose sum equals net gain.The contributions are not averaged over separate improving and declining subsets.Source: D Additional training-instance analyses · p. 43, Table 21 · p. 44