PaperlayerView exact revision on arXiv

Source-linked AI summary

DCAS: Decoupling CLI Agent Scaffolding to Internalize Planning across Scaffolds

Kishanthan Thangarajah, Boyuan Chen, Ahmed E. Hassan

arXiv:2608.06113v1cs.SE

TL;DR

Fine-tuned CLI agents perform well in their training scaffold but degrade under others, raising questions about the role of scaffold-specific planning. DCAS enables cross-scaffold evaluation and planning-aware trajectory collection, showing that plan quality strongly affects performance and that planning learned under one scaffold transfers across others.

  • Problem

    Open CLI-agent training relies almost exclusively on OpenHands, leaving cross-scaffold performance and the source of its degradation insufficiently established.

  • Method

    DCAS routes API traffic between unmodified CLI scaffolds and substitute backend models, enabling cross-scaffold evaluation and planning-aware trajectory collection.

  • Results

    A plan-source intervention improved SWE-bench Verified Pass@1 from 42.8% to 57.8%, while planning-aware fine-tuning produced gains across multiple non-training scaffolds.

  • Takeaways & Limitations

    Planning structure is a learnable capability whose explicit and implicit senses can be separated in training data and transferred beyond the training scaffold.

  • Takeaways & Limitations

    Pass@1 does not directly measure trajectory efficiency, turn count, intermediate reasoning quality, or plan quality.

Abstract

from arXiv · show

CLI-based software-engineering agents have matured rapidly, yet the open ecosystem has converged on a single training environment: trajectory datasets used to fine-tune open models are collected almost exclusively under OpenHands. Models fine-tuned on this data score well under OpenHands but degrade substantially when deployed under any non-training scaffold. Untrained base models do not show this divergence, indicating the gap is fine-tuning-induced and tied to the conventions of the training scaffold. We argue that a load-bearing scaffold-specific behavior is planning structure, in two senses this paper distinguishes: explicit planning, a pre-execution plan produced as a first-class artifact, and implicit planning, the structural conventions that shape execution throughout the agent loop. Under this hypothesis, closing the gap requires moving planning from a fixed scaffold artifact to a learned model capability. We introduce Decoupling CLI Agent Scaffolding (DCAS), a backend-substitution interception layer that routes API traffic between any CLI scaffold and any backend model without modifying the scaffold, enabling cross-scaffold evaluation and planning-aware trajectory collection. Using DCAS, a controlled plan-source intervention confirms planning quality is a high-leverage component, with gains exceeding the cross-scaffold drops we observe. A model fine-tuned on a small set of DCAS-collected planning-aware trajectories under a single scaffold gains consistently across non-training scaffolds, and the two senses of planning are empirically separable in training data.

1 Introduction

CLI agents fine-tuned on OpenHands trajectories degrade substantially on non-training scaffolds, suggesting scaffold-specific planning conventions are installed during fine-tuning. DCAS enables controlled cross-scaffold evaluation and planning-aware trajectory collection to test whether planning can become portable through training.

  • Motivation: Open models at 32B scale resolve more than half of SWE-bench Verified instances, but OpenHands-trained models degrade substantially on non-training scaffolds.SWE-Gym, Nebius, SWE-Lego, and CoderForge collect training trajectories exclusively under OpenHands.
  • Motivation: Fine-tuning sharply widens cross-scaffold performance differences, indicating that scaffold conventions rather than underlying model capability install the responsible behavior.Untrained base models vary far less across evaluated scaffolds than models fine-tuned from them.
  • Planning hypothesis: Across 16,991 CLI agent trajectories, plan quality and plan adherence are primary drivers of resolution rate, motivating explicit and implicit planning as distinct concepts.Explicit planning is a pre-execution plan produced as a first-class artifact; implicit planning consists of structural conventions shaping execution throughout the agent loop.
  • DCAS: DCAS routes API traffic between any CLI scaffold and backend model without scaffold modification, enabling cross-scaffold evaluation and planning-aware trajectory collection.The same backend model can be tested under each target scaffold’s full convention surface, including proprietary-backend scaffolds.
  • Research questions: Controlled plan-source variation tests explicit planning’s effect, while plan-only and plan-and-execution training test internalization and generalization to unseen scaffolds.The introduction organizes these tests as RQ1, RQ2, and RQ3.

2 Background and Motivation

The paper identifies a fine-tuning-induced cross-scaffold performance gap rooted in scaffold-specific conventions, especially explicit and implicit planning structure. It therefore hypothesizes that planning must become a learned model capability rather than remain a fixed scaffold artifact.

  • Scaffold ecosystem: OpenHands dominates open trajectory collection, so strong open models are predominantly trained on OpenHands trajectories.SWE-Gym, Nebius, SWE-Lego, and CoderForge all collect under OpenHands.
  • Cross-scaffold degradation: 29 points: Nebius-SWE-Rebench-30B drops from 49.7% on OpenHands to 20.4% on mini-swe-agent.SWE-Lego-Qwen3-32B also falls from 52.6% on OpenHands to 44.2% on Claude Code, while tool-call incompatibility can cause sharper failures.
  • Fine-tuning effect: Untrained bases remain comparatively consistent across scaffolds, indicating that divergence is installed by fine-tuning rather than inherent to the models.Qwen3-32B scores 29.0/23.2/18.4/8.0 and Qwen3-30B-A3B-Instruct-2507 scores 25.2/26.8/27.6/14.0 across the four scaffolds.
  • Training distribution: 15.6%: Qwen3-32B-Nex-N1 still degrades on mini-swe-agent despite training on diverse multi-format trajectories.Broader training narrows the gap on Claude Code to 50.8% but does not eliminate dependence on training-scaffold conventions on OpenCode and mini-swe-agent.
  • Planning hypothesis: Planning structure is identified as a load-bearing scaffold-specific behavior, encompassing explicit plans and implicit turn-by-turn execution conventions.The conventions include sub-step granularity, exploration-to-action transitions, tool-call sequencing, and failure recovery.
  • Planning hypothesis: Activating a self-generated planning turn produces consistent gains on Claude Code and OpenCode for two representative fine-tuned models.The gains are modest and require no targeted training, motivating controlled experiments that test whether planning can move from a scaffold artifact to a learned capability.

3 DCAS: Design and Experimental Setup

DCAS decouples CLI scaffolds from backend models through an interception layer that logs trajectories without modifying the scaffold. The experiments isolate plan-source effects, collect planning-aware trajectories, and evaluate planning transfer across scaffold versions and implementations.

  • DCAS interception layer: DCAS translates API requests and responses between an unmodified CLI scaffold and any substituted backend model while logging interactions as trajectory files.The layer was built upon the open-source SWE-bench runner’s API-interception approach.
  • Evaluation setup: Evaluation uses SWE-bench Verified with Pass@1, and fine-tuned models are tested on CC 2.0.76, CC 2.1.73, OpenCode v0.0.55, and mini-swe-agent v2.2.8.The execution budget is capped at 100 turns, while planning receives up to 500 turns before execution.
  • Controlled planning experiments: RQ1 compares no-plan, self-plan, and external-plan conditions while holding the backend model, benchmark, and scaffold constant.External plans come from Claude Sonnet 4.5, Claude Opus 4.5, or Qwen3-Coder-480B-A35B-Instruct; self-plans are generated by the backend model.
  • Trajectory collection: RQ2 retains 576 correctly resolving two-phase trajectories collected with GLM-4.7 substituted inside Claude Code via DCAS.The source tasks come from SWE-Lego Real Data, a curated split of approximately 5,000 resolvable GitHub issues.
  • SFT dataset construction: The PlanOnly dataset contains 14,162 per-turn samples, whereas Plan+Exec contains 36,259 per-turn samples, both drawn from 576 instances.These variants separate training on planning conventions alone from training that includes planning and execution phases.
  • Fine-tuning setup: Qwen3-Coder-30B-A3B-Instruct is fine-tuned with full-parameter SFT using a 4.0e-6 learning rate and 65,536-token context.Training uses BF16 precision, Flash Attention 2, DeepSpeed ZeRO Stage 3, and disabled thinking tokens.

4 RQ1: How Much Does Plan Quality Matter?

RQ1 shows that explicit planning is a separable, high-leverage component of CLI-agent performance. Under a controlled Claude Code evaluation, improving plan quality raised Pass@1 from 42.8% without planning to 57.8% with a Claude Sonnet 4.5 plan, exceeding the observed cross-scaffold drops.

  • RQ1: The controlled evaluation held the backend model, benchmark, turn limit, and scaffold constant, isolating plan-related performance differences.All conditions used Qwen3-Coder-30B-A3B-Instruct with CC 2.0.76.
  • RQ1: 42.8% Pass@1 was achieved without planning, establishing the no-plan baseline under Claude Code.This baseline reflects deployment without learned planning conventions.
  • RQ1: 48.2% Pass@1, a gain of 5.4%, resulted from adding self-planning before execution.The result indicates that structured problem decomposition provides value even without an external planner.
  • RQ1: 49.2% Pass@1 (+6.4% over baseline) resulted from a Qwen3-Coder-480B-A35B-Instruct plan, showing gains from an open-weight planner.This supports scaling plan quality with planner capability before using proprietary frontier models.
  • RQ1: 57.8% Pass@1 from Claude Sonnet 4.5 represented a gain of up to 15% over the no-plan baseline, while Claude Opus 4.5 reached 56.0% under default selection.Sonnet achieved 57.8% under both routing variants; the improvement was attributable solely to plan quality under the controlled setup.
  • RQ1: The 15% planning gain exceeded the 8.4% SWE-Lego-Qwen3-32B drop from OpenHands to Claude Code, motivating planning as a training target.The experiment isolates only the explicit-planning contribution, while the broader cross-scaffold gap may also involve implicit planning conventions.

5 RQ2: Can Planning Be Internalized Through Fine-Tuning?

Fine-tuning on planning-aware DCAS trajectories internalizes scaffold planning structure, with PlanOnly learning implicit conventions and Plan+Exec learning both implicit conventions and productive explicit-plan use. Plan+Exec reaches 55.8% under self-plan, while PlanOnly reaches 53.8% without a plan but gains no further benefit from self-planning.

  • PlanOnly: PlanOnly reaches 53.8% without a plan (+11.0% over base), showing that planning-only training installs implicit scaffold conventions.The conventions include work decomposition, sub-step granularity, and tool-call structure.
  • PlanOnly: PlanOnly scores 53.2% with self-plan versus 53.8% without one, indicating it learned implicit planning but not productive use of an explicit planning turn.Its training data contained planning turns but did not show how to use plans during execution.
  • Plan+Exec: Plan+Exec reaches 52.8% without a plan (+10.0% over base) and 55.8% with self-plan (+13.0%), adding +3.0% over its no-plan score.Plan+Exec produced the strongest and most consistent results, installing implicit conventions and an explicit-planning benefit.
  • Decomposition: Both variants produce similar no-plan gains, but only Plan+Exec adds explicit-planning benefits because its data shows how to use plans during execution.The results empirically decompose implicit planning from productive explicit planning.

6 RQ3: Does the Learned Planning Generalize Beyond the Training Environment?

The Plan+Exec model’s learned planning transfers beyond its training environment, including to a newer scaffold release and unseen scaffolds. Its benefits remain positive across deployment environments, although their magnitude varies with scaffold alignment.

  • Evaluation setup: The same Plan+Exec model was evaluated across deployment environments with training, benchmark, and turn limits held constant, varying only the environment.The model was trained only on CC 2.0.76 trajectories and had not seen the evaluation environments during training.
  • Version generalization: 57.2% with self-generated plans on CC 2.1.73 exceeded 55.8% on CC 2.0.76, showing transfer to a newer release with updated tool schemas and turn management.The corresponding no-plan scores were 54.6% on CC 2.1.73 and 52.8% on CC 2.0.76.
  • Cross-scaffold generalization: Under OpenCode, the SFT model reached 49.4% with self-plan (+3.4% over base without a plan) and 47.6% without a plan (+1.6%).OpenCode was not used during training and was evaluated through the DCAS interception layer.
  • Cross-scaffold generalization: Self-planning improved the base model on Claude Code (+5.4%), OpenCode (+1.4%), and mini-swe-agent (+1.8%), while fine-tuning widened the no-plan versus self-plan gap.The results support planning as the primary capability acquired through DCAS fine-tuning, with benefits persisting across unseen scaffolds.
  • Cross-scaffold generalization: Under mini-swe-agent, the SFT model with self-generated plans achieved 42.4% versus 35.4% for the base model without a plan (+7.0%).Mini-swe-agent was an unseen deployment scaffold evaluated through DCAS.

7 Discussion

The discussion identifies planning structure—explicit and implicit—as a load-bearing scaffold-specific behavior behind deployment gaps, and argues that evaluation, planner calibration, and trajectory economics should reflect this finding. It recommends treating multi-scaffold evaluation as a default and transferring planning structure into learned model capability.

  • Central implication: Planning structure, in both explicit and implicit senses, underlies the cross-scaffold deployment gap and can be transferred from scaffold artifacts into learned model capabilities.This conclusion synthesizes the paper’s three research questions and frames the discussion beyond the specific experiments.
  • Evaluation methodology: Every fine-tuned model degraded on at least one non-training scaffold, while its within-row spread exceeded that of the corresponding base model.The discussion characterizes these cross-scaffold drops as a real failure mode rather than measurement noise.
  • Evaluation methodology: Published CLI-agent scores should report performance on at least one out-of-distribution scaffold and ideally provide a vector across several scaffolds.Single-scaffold reporting can obscure generalization deficits and concentrate optimization pressure on one training environment.
  • Planner-executor calibration: 57.8% versus 54.2–56.0%: Claude Sonnet 4.5 consistently outperformed Claude Opus 4.5 as planner with a 30B executor.The proposed explanation is planner-executor calibration: Opus’s plans may assume reasoning the executor cannot reliably perform, while Sonnet’s may be more explicit and stepwise.
  • Trajectory economics: The scaffold-portable signal is carried by planning structure rather than trajectory volume, making small planning-aware datasets potentially more informative per trajectory.Each retained trajectory requires a successful end-to-end run, while rejection sampling discards failures; the discussion therefore emphasizes collection efficiency.
  • Planning portability: Self-planning helped under all three evaluated scaffolds, with gains of +5.4% under Claude Code, +1.8% under mini-swe-agent, and +1.4% under OpenCode.The discussion attributes scaffold dependence to whether the base model has internalized compatible implicit planning conventions; Plan+Exec fine-tuning addresses this across scaffolds.

8 Related Work

DCAS builds on prior work in CLI coding-agent scaffolds, trajectory collection, planning architectures, and imitation learning for software engineering. Related studies establish benchmark and interface advances, expose scaffold-specific transfer limits, and motivate planning-aware cross-scaffold training.

  • CLI agent scaffolds: SWE-bench established repository-level issue resolution, while SWE-agent showed that a purpose-built agent-computer interface improves codebase navigation and editing.
  • CLI agent scaffolds: Agentless demonstrated that separating pre-execution structure from patch generation improves quality, and mini-swe-agent exceeds 74% Pass@1 with a strong frontier model.
  • Cross-scaffold transfer: Qwen3-Coder-Next corroborates that OpenHands-trained models transfer poorly to SWE-agent, while Kimi K2 reaches 65.8% via large-scale agentic RL and DCAS Plan+Exec reaches 55.8% at 30B scale with SFT alone.
  • Trajectory collection: Single-scaffold collection dominates open trajectory datasets, particularly under OpenHands, with SWE-smith a notable exception using SWE-agent instead.
  • Planning and multi-agent architectures: Reasoning and multi-agent work improves structured software engineering, with MASAI raising file recall from 61% to 75% through dedicated localization and CodeR achieving 28.33% on SWE-bench.
  • Trajectory-based learning: SWE-Gym, Kimi-Dev, SWE-RL, and DeepSWE show that supervised or reinforcement learning transfers coding-agent capabilities, while DCAS targets cross-scaffold transfer with planning-aware trajectories.

9 Threats to Validity

The findings are limited by incomplete isolation of planning, narrow evaluation and training coverage, and measurement constraints. Version controls, raw logs, diverse retained trajectories, and full-set evaluation mitigate some threats, but broader generalization and direct plan assessment remain open.

  • Causal isolation: Cross-scaffold planning effects are not fully isolated because planning covaries with tool schemas, parsers, turn management, and context-window handling.Within a scaffold, RQ1 manipulates plans while holding the model, benchmark, and scaffold fixed; future work must ablate remaining dimensions while holding planning fixed.
  • Causal isolation: The smaller Pass@1 spread for every untrained base model than for any fine-tuned model supports a training-induced divergence, but does not fully isolate planning.All conditions cap execution at 100 turns, ruling out turn budget as a differing factor.
  • Data and coverage: Rejection sampling biases training toward easier instances, although 576 diverse two-phase trajectories and evaluation on the full SWE-bench Verified set mitigate this concern.Training trajectories were collected exclusively under Claude Code, limiting cross-scaffold coverage.
  • External validity: Results may not generalize beyond SWE-bench Verified, Qwen3-Coder-30B-A3B-Instruct, Claude Code, OpenCode, and mini-swe-agent.The reported +15% plan-injection gain and +13% Plan+Exec fine-tuning gain may not transfer across models, task types, code bases, or other CLI scaffolds and action spaces.
  • Implementation: A 32,768-token ceiling caused API failures against Claude Code’s 32,000-token output limit for SERA-32B and Qwen3-32B-Nex-N1, resolved using YaRN RoPE scaling.The incompatibility was specific to those released artifacts rather than OpenHands generally; exact Claude Code versions and raw HTTP logs support inspection.
  • Measurement: Pass@1 omits trajectory efficiency, turn count, and intermediate reasoning quality, while self-plan gains may partly reflect improved execution rather than planning.Direct scoring of fine-tuned model plans is identified as future work.

10 Conclusion

The paper concludes that scaffold-specific planning conventions substantially explain cross-scaffold degradation and can be internalized as a learned model capability. Its findings show that plan quality is high-leverage, planning structure is learnable and separable, and learned behavior generalizes across deployment scaffolds.

  • Core hypothesis: OpenHands-trained CLI agent models degrade under other scaffolds, unlike untrained base models, implicating scaffold-specific planning structure.The paper distinguishes explicit planning, a first-class pre-execution plan, from implicit planning, the structural conventions shaping execution.
  • Empirical findings: 15 points: Claude Sonnet 4.5 plans raise Qwen3-Coder-30B-A3B-Instruct’s SWE-bench Verified Pass@1 from 42.8% to 57.8%.The plan-source intervention varies only the plan source, establishing plan quality as a separable, high-leverage determinant of task performance.
  • Empirical findings: Planning structure is learnable, and explicit and implicit planning come apart in training data.PlanOnly installs implicit planning conventions alone, producing 53.8% on no-plan evaluation.
  • Implications: The cross-scaffold gap is substantially a planning-convention mismatch, while tool-format and context-length issues remain real but largely orthogonal.The paper does not isolate the contribution of tool-format and context-length issues.
  • Implications: Planning can move from a fixed scaffold artifact to a learned model capability that generalizes across deployment scaffolds.This conclusion follows from the combined findings supporting the paper’s central hypothesis.

11 Disclaimer

The authors state that the views and findings are theirs, not Huawei’s, and disclose AI-assisted copy-editing subject to author review and IEEE/ACM publication policies.

  • The opinions, findings, conclusions, and recommendations do not reflect Huawei’s views.
  • AI tools were used for copy-editing, while the authors performed and thoroughly reviewed the experiments, analysis, writing, and results.The authors state that this practice complies with IEEE and ACM policies on AI use in publications.
Loading 2608.06113v1…