PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-MILE: Asynchronous Potential-Induced Milestone Credit Assignment for Long-Horizon Software Engineering Agents

Chaoqun Cui, Hao Zhou, Meiqi Chen, Fandong Meng, Wenji Mao

arXiv:2609.32631v1cs.AIcs.SE

TL;DR

Long-horizon SWE agents trained with terminal-only rewards lack action-level signals for distinguishing productive work from redundant exploration or regressions. SWE-MILE derives process credit from file exposure and test-state milestones, using asynchronous shadow probing for intermediate verification. Experiments report substantial performance improvements across issue resolution and whole-repository generation.

  • Problem

    Terminal verifier outcomes do not identify which actions in long SWE trajectories contribute to task success or undermine it.

    Source: §1 · pp. 1–2

  • Method

    SWE-MILE assigns step-level process credit from changes in file-navigation and test-verification potentials, with asynchronous shadow probing to obtain intermediate test states.

    Source: §1 · p. 2

  • Results

    SWE-MILE-trained policies achieve substantial performance improvements on both issue resolution and whole-repository generation tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Workflow runtime provides a practical source of fine-grained process supervision while retaining terminal outcome supervision and avoiding additional reward models or LLM evaluators.

    Source: §1 · p. 2, §4 · p. 10

    Each programming language typically requires a dedicated verification-potential collection pipeline, and the current potential does not distinguish individual test importance.

    Source: §D.2 · p. 21

Abstract

from arXiv · show

Long-horizon software engineering (SWE) agents trained with reinforcement learning with verifiable rewards (RLVR) typically receive only terminal outcome supervision, making it difficult to distinguish productive actions from redundant exploration or functional regressions. We propose SWE-MILE, an asynchronous potential-induced milestone credit assignment framework that derives fine-grained process supervision from workflow runtime, without auxiliary reward models or external evaluators. SWE-MILE quantifies task-relevant file exposure and test-state alignment as navigation and verification potentials, respectively. Differences in these potentials attribute milestone progress and regressions to individual actions, while discounted backward credit propagates supervision to preceding steps. To efficiently acquire intermediate verification states, SWE-MILE further introduces asynchronous shadow probing, which replays repository-changing actions in an isolated sandbox and runs verification in parallel with the agent's primary interaction, largely hiding verification latency. The resulting process credit augments terminal outcome advantages and provides informative learning signals. Experiments on two representative long-horizon SWE tasks demonstrate substantial improvements in agent performance, highlighting workflow runtime signals as a practical source of process supervision for long-horizon SWE agents.

1 INTRODUCTION

Terminal-only rewards leave long-horizon SWE agents without action-level signals for distinguishing progress from redundant exploration or regressions. SWE-MILE derives process credit from runtime milestones, and experiments report substantial improvements on issue resolution and whole-repository generation.

1 INTRODUCTION

  • Terminal verifier outcomes indicate rollout success but do not reveal which intermediate actions contribute to or undermine it.Long trajectories can include redundant exploration, temporary regressions, or useful partial fixes, making uniform outcome rewards a poor guide to individual contributions.Source: §1 · p. 1
  • Prior process-supervision approaches use interaction rules, auxiliary evaluators, or cross-rollout state aggregation, each with limitations for capturing verifiable repository progress.The cited concerns include shallow or hackable rules, proxy judgments with added inference workload, and sparse state matching.Source: §1 · p. 2
  • Workflow runtime exposes fine-grained milestones, including inspected key files, repaired target tests, and regressions in previously correct functionality.The central challenge is attributing these changes to specific actions.Source: §1 · p. 2
  • SWE-MILE derives action-level credit from task-relevant file exposure and verification-state alignment, without additional reward models or LLM evaluators.Potential differences assign rewards or penalties to progress and regression; asynchronous shadow probing checks intermediate verification states while primary interaction continues.Source: §1 · p. 2
  • SWE-MILE-trained policies achieve substantial performance improvements on both issue resolution and whole-repository generation tasks.Source: §1 · p. 2

2 METHOD

SWE-MILE retains terminal outcome rewards while deriving step-level credit from runtime milestones. It combines file-navigation and test-verification potentials, propagates credit backward, and obtains intermediate test states through asynchronous shadow probes.

2.1 PRELIMINARIES

  • The final verifier assigns a binary outcome that cannot distinguish individual actions’ contributions, motivating step-level rewards for action-caused progress or regression.A milestone is observable intermediate progress or regression, a potential is a scalar state function quantifying it, and a shadow probe runs the verifier in an isolated sandbox.Source: §2.1 · p. 3

2.2 OVERALL FRAMEWORK

  • SWE-MILE combines navigation and verification potentials, attributes changes through adjacent-step differences, and propagates credit backward while retaining terminal outcome supervision.A primary sandbox handles agent interaction while a shadow sandbox asynchronously replays repository-changing actions and probes intermediate test states.Source: §2.2 · p. 3, Figure 1 · p. 3

2.3 CODEFLOW INTERACTION HARNESS

  • Codeflow separates retrieval, inspection, editing, testing, and submission into five tools to facilitate attribution of agent behavior.Static shell analysis and visible output help identify retrieval and inspection done through Bash; repository-hash changes identify editing behavior.Source: §2.3 · pp. 3–4

2.4 MILESTONE POTENTIAL AND PROCESS CREDIT

  • Navigation potential measures exposure of relevant files, while verification potential measures test-state alignment; their differences assign credit to progress and regression.Repeated inspection of already exposed files adds no navigation reward, and breaking then repairing functionality is penalized before recovery credit.Source: §2.4 · pp. 4–5

2.5 POLICY OPTIMIZATION

  • The step-level PPO-style optimization preserves trajectory comparisons through outcome advantage while using process credit to distinguish steps within each trajectory.SWE-MILE avoids cross-trajectory step normalization and adds process credit as a correction to an RLOO-style outcome advantage.Source: §2.5 · p. 6

2.6 ASYNCHRONOUS SHADOW PROBING

  • Asynchronous shadow probes replay repository-changing actions in order and run verification alongside primary interaction, largely hiding their latency.Across 9,600 training rollouts, verification probing introduced about 2.07 seconds of average wall-clock latency.Source: §2.6 · p. 6, Figure 2 · p. 6, Table 1 · p. 6

3 EXPERIMENTS

The experiments evaluate SWE-MILE on issue resolution and whole-repository generation, comparing it with other methods, component ablations, and potential-weight settings. Across both tasks, the reported results favor SWE-MILE, while the ablations and sensitivity analysis show that its signals and their scales matter.

3.3 ABLATION STUDY

  • Removing all milestone credit causes the largest Avg@4 degradation; verification removal hurts more than navigation, while backward credit and RLOO advantage also contribute.Source: §3.3 · pp. 8–9, Table 4 · p. 9

3.4 HYPERPARAMETER ANALYSIS

  • Performance first rises and then falls as potential weights increase; excessively large verification weights cause sharper degradation than navigation weights.Source: Figure 3 · p. 9, §3.4 · p. 9

4 CONCLUSION

SWE-MILE derives action-level feedback from navigation and verification potentials, potential differences, and backward credit, while asynchronous shadow probing runs intermediate verification alongside agent interaction. The paper concludes that this provides a practical runtime-based source of process supervision without an additional reward model or LLM evaluator.

4 CONCLUSION

  • SWE-MILE derives action-level feedback from navigation and verification potentials, potential differences, backward credit, and asynchronous shadow probing.Source: §4 · pp. 9–10

AI USE STATEMENT

  • The authors used generative AI for implementation, environment setup, statistical analysis, and language polishing, but not for the research idea, method, or experimental plan.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

Appendix C Training Dynamics

This passage reviews research on SWE benchmarks, agent scaffolds, and reinforcement learning and process-reward methods for long-horizon agents.

Appendix C Training Dynamics

  • The section reviews SWE benchmarks and verifiable environments, agent scaffolds, and reinforcement learning and process-reward methods for long-horizon agents.Source: A RELATED WORK · p. 13

A.1 SWE BENCHMARKS AND SCALABLE VERIFIABLE ENVIRONMENTS

The related work traces SWE research from code generation toward repository-level tasks and scalable verifiable environments, alongside evolving agent interfaces and approaches to process credit assignment.

A.1 SWE BENCHMARKS AND SCALABLE VERIFIABLE ENVIRONMENTS

  • SWE benchmarks and environments have expanded from static evaluations toward scalable executable settings, including multilingual environments and whole-repository generation.Source: §A.1 · p. 14

A.2 SWE AGENT SCAFFOLDS AND ENVIRONMENT INTERACTION

  • SWE scaffolds range from structured repository tools to general-purpose Bash interfaces, while Agentless uses a predefined localization–repair–validation pipeline.Source: §A.2 · p. 14

A.3 CREDIT ASSIGNMENT FOR LONG-HORIZON SWE AGENT REINFORCEMENT LEARNING

  • Prior credit-assignment approaches include test-based rewards, externally evaluated process supervision, rule-based tool rewards, and graph- or search-based step credit.Source: §A.3 · pp. 14–15

B EXPERIMENTAL SETTINGS DETAILS

The experiments use consistent hyperparameters and comparison conditions, with dynamic sampling and a turn-limit reward adjustment shaping training.

B EXPERIMENTAL SETTINGS DETAILS

  • The experiments use shared hyperparameters unless otherwise specified, and compare methods using identical materialized data, inference budgets, and general optimization settings.The best checkpoint is selected from evaluations every five training steps.Source: B EXPERIMENTAL SETTINGS DETAILS · p. 15, §B.1 · p. 15

B.1 MAIN SETTING

  • Task groups without outcome-level comparison signals are filtered and resampled, with difficult tasks receiving more retry opportunities than easy tasks.For whole-repository generation, filtering also applies when pass counts match or pass rates are near full or zero.Source: §B.1 · p. 15
  • For issue-resolution trajectories that pass verification after hitting a turn or context limit, the terminal reward is discounted according to the first turn when verification potential reaches 1.The reward is Rlim = t*/|T|; outcome-only ablation training instead sets it to 0.6.Source: §B.1 · p. 15

B.2 BASELINE IMPLEMENTATION

Baseline implementations align comparison conditions and adapt state aggregation to SWE tasks; Codeflow records action evidence, while shadow probes improve verification efficiency for repository generation.

B.2 BASELINE IMPLEMENTATION

  • GiGPO and GraphGPO use compact SWE states comprising separate passing F2P and P2P test counts and distinct search and inspection exposure of relevant files.For whole-repository generation, file-exposure states are omitted; test states come from shadow probing.Source: §B.2 · p. 17
  • Baseline comparisons reuse SWE-MILE’s datasets and configurations, with GRPO and G-RA adjusted to align advantage estimation and reward conditions.G-RA is implemented from its paper because open-source code is unavailable; the SWE-TRACE implementation uses Qwen3.8-27B as rubric generator and trajectory judge.Source: §B.2 · pp. 15–17

B.3 CODEFLOW SCAFFOLD

  • Codeflow records tool events, file-exposure evidence, and repository changes to support milestone attribution, with structured full-tool and bash-only configurations.The available tool types and modes are listed in Table 7.Source: §B.3 · p. 17, Table 7 · p. 17

B.4 SHADOW PROBE FOR WHOLE-REPOSITORY GENERATION

  • For whole-repository generation, identical changed-file sets from consecutive eligible edits can be merged into one probe, while multiple sandboxes run probes in parallel.The verifier runs once on the final state in each merged group, and its net potential change is evenly assigned across that group’s actions; results are aggregated in action order before policy update. Table 8 reports runtime statistics.Source: §B.4 · p. 18, Table 8 · p. 18

C TRAINING DYNAMICS

  • The appendix examines model performance, interaction cost, and optimization dynamics as three perspectives on training behavior.It covers SWE-bench Pro evaluation, rollout interactions, and policy optimization.Source: C TRAINING DYNAMICS · p. 18

C.1 EVALUATION DYNAMICS ON SWE-BENCH PRO

Across training checkpoints, SWE-MILE’s success rate rises while interaction turns and token counts decline, unlike outcome-only training’s growing interaction burden.

C.1 EVALUATION DYNAMICS ON SWE-BENCH PRO

  • SWE-MILE shows an overall increase in success rate while interaction turns and token counts gradually decrease across checkpoints.Outcome-only training also improves success rate, but turns continue to increase and total trajectory length grows despite fewer model output tokens; removing either potential shortens trajectories less than the full method.Source: §C.1 · p. 18

C.2 TRAINING ROLLOUT DYNAMICS

Training rollouts grow as later sampling includes more difficult tasks, but SWE-MILE exhibits slower trajectory growth than outcome-only training. Verification potential is associated with more stable optimization dynamics, and asynchronous shadow probing’s applicability depends on replayable environment transitions.

C.2 TRAINING ROLLOUT DYNAMICS

  • Outcome-only shows the largest increase in interaction turns and trajectory length, while SWE-MILE exhibits slower trajectory growth during training.Trajectories grow under all methods as asymmetric dynamic sampling gives difficult tasks more retry opportunities; removing either potential generally raises interaction cost, especially turns when verification potential is removed.Source: §C.2 · pp. 18–19, Figure 5 · p. 19

C.3 POLICY OPTIMIZATION DYNAMICS

  • KL divergence and entropy rise noticeably later in outcome-only and w/o verification training, while SWE-MILE and w/o navigation remain relatively stable.The latter two retain verification potential; loss values should not be compared directly because the methods use different advantage signals.Source: §C.3 · p. 20, Figure 6 · p. 20

D.1 REPLAY FIDELITY OF SWE ENVIRONMENTS

  • Asynchronous shadow probing is particularly suited to sandboxed SWE environments, where repository-changing action sequences can typically be replayed with high fidelity.More stochastic environments may require state snapshotting, replay validation, or mechanisms for handling environmental divergence.Source: §D.1 · pp. 20–21

D.2 LIMITATIONS

The limitations section identifies dependence on language-specific verification pipelines and a test-importance blind spot in the current potential. The accompanying prompt material describes the system setup and task instructions for issue resolution and repository generation.

D.2 LIMITATIONS

  • Verification feedback requires a dedicated collection pipeline for each programming language, because test frameworks produce heterogeneous result formats.Source: §D.2 · p. 21
  • The current verification potential aggregates test outcomes without weighting individual test importance, so distinct functional progress can receive the same value.Source: §D.2 · p. 21

E PROMPTS

  • The prompt appendix notes that runtime-substituted placeholders are shown in blue.Source: E PROMPTS · p. 21

E.1 SYSTEM PROMPT

  • The system prompt specifies the agent role, one-tool-call interaction protocol, tool policy, path convention, and tool-observation semantics.Source: §E.1 · p. 21

E.2 ISSUE RESOLUTION PROMPT

  • Issue-resolution instructions provide the repository root and issue description, then direct the agent to inspect, update, test, and submit the repository.Source: §E.2 · p. 23

E.3 WHOLE-REPOSITORY GENERATION PROMPT

  • Whole-repository generation instructions ask the agent to build or complete files to meet a package specification rather than merely repair a defect.The passage connects this broader construction task with a longer horizon and more repository-changing actions.Source: §E.3 · p. 23
Loading 2609.32631v1…