PaperlayerView exact revision on arXiv

Source-linked AI summary

MindForge: Teaching Small Language Models Whole-Life-Cycle Software Engineering via Source-Free Program Synthesis

Yihao Chen, Shi Chang, Khaled Chawa, Feng Lin, Boyuan Chen, Shaowei Wang, Ahmed E. Hassan

arXiv:2607.27146v1cs.SEcs.CLcs.LG

TL;DR

Constructing complete programs from scratch remains difficult, and scalable training environments for the full software-engineering life cycle are lacking. MindForge creates source-free environments and whole-life-cycle synthesis trajectories, raising Qwen3.6-27B’s ProgramBench average test pass rate from 37.98% to 49.51% while improving performance across seven unseen benchmarks.

  • Problem

    Constructing programs from scratch requires agents to perform the full development process, but existing coding-agent progress largely concerns modifying existing codebases.

  • Method

    MindForge converts open-source command-line programs into source-free environments and collects refined whole-life-cycle program-synthesis trajectories for training.

  • Results

    Qwen3.6-27B’s ProgramBench average test pass rate rises from 37.98% to 49.51%, with improvements across all seven unseen software-engineering benchmarks.

  • Takeaways & Limitations

    Whole-life-cycle software-engineering supervision transfers beyond from-scratch reconstruction to issue resolution, repository generation, translation, and feature implementation.

Abstract

from arXiv · show

Coding agents have made substantial progress on software engineering tasks that modify existing codebases, including bug fixing and feature implementation. However, constructing a complete program from scratch remains a major challenge: even the frontier models evaluated on ProgramBench fully resolve fewer than 1% of tasks. One obstacle is the lack of scalable training environments for this from-scratch setting, spanning the whole software engineering life cycle, as existing environment-construction frameworks focus only on a single phase in software development. To address this gap, we introduce MindForge, an automated pipeline that converts open-source command-line programs into source-free environments that expose only a compiled reference executable and its documentation. Using MindForge, we construct training environments from repositories disjoint from those in ProgramBench, and curate a high-quality data recipe consisting of program synthesis trajectories using GLM-5.2 as the teacher agent. Fine-tuning Qwen3.6-27B on these trajectories increases its ProgramBench average test pass rate from 37.98% to 49.51%, achieving performance comparable to substantially larger frontier models. Moreover, the fine-tuned model consistently improves over the base model across all seven unseen software engineering benchmarks, spanning long-horizon repository generation and translation, bug fixing, feature implementation, and cross-language issue resolution, with absolute gains of 31.00 points on RepoZero-C2Rust, 14.16 on DeepSWE, 10.70/4.56 on NL2Repo-Bench (with/without tests), 5.04 on SWE-bench Verified, 5.93 on SWE-bench Pro, 5.22 on SWE-bench Multilingual, and 4.94 on FeatBench.

1 Introduction

MindForge addresses the challenge of building complete programs from scratch by creating source-free environments and collecting refined whole-life-cycle software-engineering trajectories. Training Qwen3.6-27B on 1,001 such trajectories raises its ProgramBench average test pass rate from 37.98% to 49.51% and improves performance across seven unseen benchmarks.

  • Motivation: Building programs from scratch is substantially harder than modifying existing code because agents must handle the full program-development process.Existing coding-agent successes primarily target bug fixing, feature implementation, and code completion in established codebases.
  • MindForge: MindForge converts open-source command-line programs into source-free environments exposing only a compiled reference executable and public documentation.The pipeline targets whole-life-cycle software engineering for from-scratch program construction.
  • Data construction: 562 source-free program environments spanning six compiled languages yielded 1,001 whole-life-cycle trajectories from programs disjoint from ProgramBench, using GLM-5.2 as teacher.Agents had no access to the underlying source code during trajectory collection.
  • Results: 37.98% to 49.51%: fine-tuning Qwen3.6-27B increases its ProgramBench average test pass rate, surpassing DeepSeek V4 Pro at 47.80%.The result is comparable to substantially larger frontier models, including GLM-5.1 at 50.9% and Opus 4.7 at 51.38%.
  • Results: 31.00 points on RepoZero-C2Rust, 14.16 on DeepSWE, 10.70/4.56 on NL2Repo-Bench, 5.04 on SWE-bench Verified, 5.93 on SWE-bench Pro, 5.22 on SWE-bench Multilingual, and 4.94 on FeatBench are the reported absolute gains across seven unseen benchmarks.The improvements extend to eight evaluation settings not included in the training recipe.

2 MindForge: Program Synthesis Trajectories with Coding Agent at Scale

MindForge builds source-free executable environments from open-source command-line programs, then collects and refines long-horizon whole-life-cycle program-synthesis trajectories with GLM-5.2. The resulting corpus spans hundreds of programs and languages while covering activities from specification exploration through refinement.

  • Pipeline overview: MindForge has two phases: constructing reproducible source-free executable environments and collecting/refining long-horizon synthesis trajectories with a strong teacher agent.The trajectories cover specification exploration, implementation, and bug fixing across the software life cycle.
  • Source-free environments: Each accepted environment contains only a compiled reference executable and sanitized public documentation, so the trajectory agent never accesses source code.Environments are packaged as cleanroom Docker images aligned with ProgramBench instances.
  • Trajectory collection: The teacher agent uses GLM-5.2 to elicit specifications, design architectures, implement programs, and iterate until producing a passing build inside each cleanroom environment.Collected trajectories are subsequently refined to remove unresolved errors and provide clean supervision.
  • Dataset scale: 1,001 complete trajectories were collected across 562 unique programs spanning Go, Rust, C, C++, Swift, and TypeScript under a 256k context-window rollout budget.The programs were drawn from a larger construction pipeline that began with 2,235 candidate repositories.
  • Life-cycle coverage: The trajectories exhibit high coverage across specification exploration, design, implementation, bug localization, bug fixing, verification, and refinement.Coverage is measured as the fraction of trajectories in which each software-engineering activity appears.

3 Experimental Settings

Experiments fine-tune Qwen3.6-27B with full language-model updates and evaluate it across ProgramBench and seven unseen software-engineering benchmarks. Evaluation uses Mini-SWE-Agent, disabled Internet access, and each benchmark’s official native metric.

  • Model and training: Qwen3.6-27B is fine-tuned with all language-model weights updated, while unused vision components remain frozen, using bfloat16 for training and inference.The base student is Qwen/Qwen3.6-27B.
  • Model and training: Training uses MS-Swift with Megatron, sequence packing, micro-batch size 1, global batch size 96, and 8 epochs.Optimization uses AdamW with β1 = 0.9, β2 = 0.98, and weight decay 0.04.
  • Model and training: 4 × 10−5 is the peak learning rate, warmed up over 10% of training steps and cosine-decayed to 4 × 10−6, with gradient clipping at maximum norm 1.0.The random seed is 1105 for both training and data shuffling.
  • Evaluation: All eight benchmarks are evaluated with Mini-SWE-Agent, disabled Internet access, official evaluation protocols, and native resolve-rate or hidden-test pass-rate metrics.These settings prevent information leakage such as retrieving source code or reference solutions online.
  • Evaluation: ProgramBench contains 200 real-world open-source CLI programs across Rust (107), Go (46), C/C++ (45), Java (1), and Haskell (1).Each instance provides natural-language documentation and an execute-only binary as a behavioral oracle.
  • Evaluation: Cross-task evaluation covers seven unseen benchmarks spanning whole-program and repository construction, including NL2Repo-Bench (104 tasks) and RepoZero-C2Rust (200 tasks).These benchmarks test transfer beyond from-scratch program synthesis across distinct software-engineering task categories.

4 Results

MindForge-27B substantially improves ProgramBench performance over its base model and generalizes across all seven unseen software-engineering benchmarks. The gains are broad rather than outlier-driven, while fine-tuning also increases productive operational engagement and implementation follow-through.

  • ProgramBench Results: 37.98% to 49.51%: MindForge-27B raises ProgramBench average test pass rate by 11.53 points over its base model.This is a 30.4% relative improvement.
  • ProgramBench Results: 152 of 200 tasks (76.0%) score strictly higher for MindForge-27B, versus 43 (21.5%) lower and 5 (2.5%) tied.The models share the same architecture and parameter count, so the task-level pattern attributes the gain to the training recipe.
  • Cross-Benchmark Generalization: All seven out-of-distribution benchmarks show improvements, demonstrating generalization beyond from-scratch reconstruction.These benchmarks were not used during training, and Figure 2 reports results across eight evaluation settings.
  • Agent Behavior: 344.0 to 735.7 mean turns and 174.4 to 373.0 tool calls show that fine-tuning roughly doubles operational engagement across the same 200 ProgramBench instances.The reported pattern is accompanied by a lower per-command failure rate, indicating sustained and productive extra effort.
  • Agent Behavior: 50.1% / 48.8%: MindForge-27B nearly doubles the base model’s rates of following reasoning and failure-recovery actions with implementation edits.The base rates are 27.8% and 31.8%, respectively.

5 Related Work

MindForge builds on work constructing training environments for coding agents, distilling strong-agent trajectories into smaller models, and generating programs or repositories from scratch. Its main methodological distinction is the use of substantially longer, whole-life-cycle trajectories spanning specification discovery through a passing build.

  • Constructing Coding Agent Training Environments: SWE-Gym, SWE-Next, and R2E-Gym create executable or procedural environments for training coding agents on verified software-engineering tasks.SWE-Gym pairs GitHub issues with runtimes and unit tests; SWE-Next mines self-verifying pull-request pairs, while R2E-Gym builds procedural environments.
  • Distilling Long Trajectories into Small Models: Lingma-SWE-GPT, SWE-Fixer, SWE-Lego, and Devstral train small or open-weight models by imitating staged, filtered, real, or synthetic issue-resolution trajectories.The cited methods include mirroring developer processes, separating retriever and editor models, combining curated real and synthetic trajectories, and reinforcement learning after execution-trace mid-training.
  • Distilling Long Trajectories into Small Models: MindForge trajectories average 181.6 turns and reach 272K tokens, whereas the compared corpora are short-horizon and mostly fall within 32K context at the 95th percentile.MindForge spans the full software-engineering life cycle, from specification discovery to a passing build.
  • Program and Repository Generation from Scratch: RPG and RepoZero represent program or repository generation from scratch through explicit planning or reproduction against a withheld reference implementation.RPG uses a two-stage repository planning graph, while RepoZero provides API specifications and requires re-implementation matching a withheld reference.

6 Conclusion · A Environment Construction Agents Details

MindForge converts open-source command-line programs into source-free software engineering environments and distills whole-life-cycle trajectories into a small model. Its 27B-parameter model improves substantially on ProgramBench and generalizes across unseen software engineering benchmarks, while the pipeline uses standardized sandboxed agents coordinated by an external host.

  • 6 Conclusion: MindForge converts open-source command-line programs into source-free software engineering environments.The pipeline exposes these environments for whole-life-cycle software engineering training.
  • 6 Conclusion: It collects whole-life-cycle software engineering trajectories from a strong teacher agent and refines them into high-quality distillation supervision.The resulting data recipe is designed for supervision of smaller models.
  • 6 Conclusion: 49.51% is the refined 27B-parameter model’s ProgramBench score, up from 37.98%.This performance matches substantially larger frontier systems.
  • 6 Conclusion: The gains generalize to unseen benchmarks covering issue resolution, repository generation, translation, and feature implementation.Behavioral analysis confirms that the improvements reflect genuine transfer of software engineering behavior.
  • A Environment Construction Agents Details: Every MindForge pipeline agent runs on the mini-swe-agent harness and is driven by Qwen/Qwen3.5-397B-A17B.The same model drives every agent in the pipeline.
  • A Environment Construction Agents Details: Each agent operates inside its own disposable container sandbox, while the MindForge orchestrator runs outside the sandbox as the host.The host prepares each sandbox on a Kubernetes cluster and supplies inputs for each agent.

A.1 Explorer Agent for Initial Screening … B.1 Infrastructure-Noise Recovery

MindForge uses strict source-free screening, reproducible build-and-coverage verification, and infrastructure-noise recovery to construct reliable CLI environments and training trajectories.

  • A.1 Explorer Agent for Initial Screening: The explorer agent inspects only source and documentation, then emits a structured JSON accept/reject judgment without building, testing, patching, or network access.Its schema records executable, dependency, licensing, runtime, behavior, documentation, oracle-leak, determinism, and coverage information.
  • A.1 Explorer Agent for Initial Screening: A candidate is kept only after validated acceptance: any failed blocking gate forces rejection, while review signals cannot independently reject it.Sandbox and host validators separately check the judgment, and validated rejects are distinguished from schema-invalid outputs.
  • A.2 Build Agent: The build agent captures a self-contained script that reproducibly creates clean and coverage-capable executables from a pinned source checkout.The host re-runs the script in a fresh sandbox, so interactive exploratory installations are not trusted.
  • A.2 Build Agent: The host replays both executables on declared equivalence cases and requires matching behavior, while separately verifying nonzero first-party source coverage.Checks compare exit code, stdout, and stderr; coverage targets must be repository-relative files exercised by specified cases.
  • B Trajectory Refinement Details: The trajectory-refinement pipeline preserves validated construction outputs by replaying recorded tool-call prefixes rather than invoking teacher-agent inference after interruptions.Recovery resumes generation only for the unfinished suffix.
  • B.1 Infrastructure-Noise Recovery: After transient infrastructure noise, the host rewinds to the last healthy step and reconstructs state before continuing the remaining trajectory.This recovery procedure is illustrated as replay-based state reconstruction without teacher-agent inference.
  • A.3 Coverage Images: Coverage images package an instrumented executable with its runtime configuration, coverage tool, build commands, and replay settings for reproducible trace generation.These artifacts are verified during build replay.

B.2 Reasoning Rewrite

MindForge repairs malformed tool-use trajectories by locally rewriting only the affected reasoning against cleaned context, while preserving the teacher agent’s original action and all structured execution artifacts. Rewrites enter training data only after safety checks confirm trajectory consistency and unchanged preserved fields.

  • Local repair: GLM-5.2 rewrites the affected follow-up reasoning against the cleaned trajectory after malformed assistant and scaffold-error artifacts are removed.The repair reconnects the retained prefix to a locally coherent rationale.
  • Local repair: The cleanup preserves the teacher agent’s original action exactly while reconnecting the retained context to coherent reasoning.Figure 4 describes removal of the malformed turn and scaffold error, with the original action preserved exactly.
  • Safety constraints: Only the affected reasoning text may change; follow-up actions, arguments, tool calls, and recorded program outputs remain byte-for-byte unchanged.A rewrite is admitted only when safety checks confirm consistency with the visible trajectory and unchanged preserved structured fields; rejected proposals are excluded or regenerated.

C Evaluation Details · C.1 Mining various SE activities from trajectories

MindForge’s trajectory analysis uses deterministic event-level rules to identify broad software-engineering activities across 1,001 trajectories, including conditional analyses for failure-driven localization and fixing. These signals are observable proxies rather than latent intent, and rule-based matching plus fixed temporal windows can produce false positives and negatives.

  • C.1 Mining various SE activities from trajectories: 1,001 post-rewrite trajectories are parsed deterministically, with each trajectory receiving at most one presence flag per activity.Activities are not mutually exclusive, so one trajectory may contribute to multiple activity rows.
  • C.1 Mining various SE activities from trajectories: Observed failures require build or verification return codes 1–127, while expected timeout code 124 is excluded; reliable success requires final return code 0.Heredoc bodies are removed before command classification, and temporal windows count tool events rather than assistant turns.
  • C.1 Mining various SE activities from trajectories: 992, 872, 998, and 838 trajectories match exploration, design, implementation, and verification, respectively, using all 1,001 trajectories as the denominator.These activity counts use unconditional coverage across the full corpus.
  • C.1 Mining various SE activities from trajectories: 736 trajectories contain an observed failure; among them, 595 include localization and 627 include a subsequent edit under the stated rules.Localization and fixing are conditioned on opportunities created by observed failures.
  • C.1 Mining various SE activities from trajectories: The rules detect observable development-stage signals but do not recover latent intent or establish that the signals caused downstream generalization gains.Implicit design may be missed, negative tests may resemble bug-triggering failures, and post-success edits are only operational refinement proxies.
  • C.1 Mining various SE activities from trajectories: Rule-based matching can yield false negatives and false positives when implicit activities lack signatures or incidental events satisfy activity rules.The fixed 10- and 15-event windows trade recall against precision and cannot capture every valid development sequence.

C.2 Repository Overlap with Generalization Benchmarks

The 562 repositories used for ProgramBench training overlap with five repository identities underlying generalization benchmarks, covering 17 evaluation instances, while no overlap is found for four benchmarks. ProgramBench uses source-free program synthesis, and evaluation prompts, tests, and reference solutions are withheld during training.

  • Repository overlap: Five repository identities overlap, covering 17 evaluation instances: three DeepSWE instances and 14 SWE-bench Multilingual instances.The comparison covers 562 repositories used to construct ProgramBench training environments.
  • Repository overlap: No repository-identity overlap is found for RepoZero C2Rust, SWE-bench Verified, SWE-bench Pro, or FeatBench.NL2Repo’s public metadata exposes target package names rather than GitHub repositories, limiting the overlap comparison.
  • Benchmark distinction: ProgramBench trains from-scratch implementation using only a reference binary, while DeepSWE and SWE-bench Multilingual evaluate engineering tasks in existing codebases.Evaluation prompts, tests, and reference solutions are not used during training.

D Statistical Significance Analysis · D.1 Statistical Testing and Run Accounting · E Examples

MindForge’s improvements are evaluated with paired task-level statistical tests and repeated-run accounting across ProgramBench and out-of-distribution benchmarks. The reported ProgramBench gain is statistically significant, with additional task-level outcome counts and benchmark-specific testing procedures supporting the analysis.

  • D Statistical Significance Analysis: Every improvement across ProgramBench and eight out-of-distribution settings remains statistically significant after Holm correction for nine comparisons (pHolm < 0.05).ProgramBench’s paired-bootstrap 95% confidence interval for improvement is [8.34, 14.74] percentage points.
  • D Statistical Significance Analysis: ProgramBench yields p = 2.38 × 10−14 from a paired Wilcoxon signed-rank test and pHolm = 1.90 × 10−13 after correction.The paired-bootstrap 95% confidence interval for the improvement is [8.34, 14.74] percentage points.
  • D.1 Statistical Testing and Run Accounting: The analysis pairs each benchmark task between Qwen3.6-27B and MindForge-27B as its experimental unit.ProgramBench, DeepSWE, both NL2Repo-Bench settings, and RepoZero-C2Rust were evaluated once because of high inference costs.
  • D.1 Statistical Testing and Run Accounting: SWE-bench Pro, SWE-bench Verified, SWE-bench Multilingual, and FeatBench each use three runs per model, averaged per task before comparison.The remaining listed long-horizon benchmarks were evaluated once.
  • D.1 Statistical Testing and Run Accounting: Single-run binary outcomes use exact McNemar tests, while fractional task-level scores and three-run means use paired Wilcoxon signed-rank tests.The procedure applies McNemar testing to DeepSWE and RepoZero-C2Rust, and Wilcoxon testing to ProgramBench, NL2Repo-Bench, and four three-run benchmarks.
  • D.1 Statistical Testing and Run Accounting: DeepSWE improves on 18 tasks and regresses on 2, while RepoZero-C2Rust improves on 67 and regresses on 5.Exact McNemar p-values are 4.02 × 10−4 for DeepSWE and 6.39 × 10−15 for RepoZero-C2Rust.
  • D.1 Statistical Testing and Run Accounting: ProgramBench has 152 improved task scores, 43 decreases, and 5 ties, with paired Wilcoxon p = 2.38 × 10−14.These unadjusted values are reported transparently; conclusions use jointly Holm-adjusted values in Table 9.

E.1 Examples of editing after failure recovery · E.2 Coverage report

The examples show that successful failure recovery depends on converting an accurate diagnosis into an immediate code edit, while fine-tuning increases how thoroughly agents exercise reference-tool behavior before reproduction. MindForge-27B edits after diagnosing a help-format mismatch, whereas Qwen3.6-27B repeatedly rereads unchanged output despite localizing the defect.

  • E.1 Examples of editing after failure recovery: At turns 442–443 of 482, MindForge-27B diffs its --help output against the reference and identifies a format mismatch.
  • E.1 Examples of editing after failure recovery: The next action edits the source by enabling the clap flag that produces the reference’s two-line help layout.The mismatch arose because the original used two lines per option, while clap used a single-line format unless next_line_help was enabled.
  • E.1 Examples of editing after failure recovery: Five actions later, MindForge-27B’s help output matches the reference exactly.
  • E.1 Examples of editing after failure recovery: Six turns after detecting Qwen3.6-27B’s usage mismatch, the agent localizes the defect to its print_usage function.
  • E.1 Examples of editing after failure recovery: Across 29 consecutive non-edit actions, Qwen3.6-27B reruns ./executable -h 16 times against the same unchanged binary before eventually fixing the issue.The repeated commands included two byte-identical invocations producing byte-identical output, despite the reference behavior already being captured in the first diff.
  • E.1 Examples of editing after failure recovery: The key difference between the agents is whether a correct diagnosis becomes a code change, not simply whether editing is delayed.Re-examining the reference can be legitimate investigation; the problematic span is characterized by redundant rereading of already captured output.
  • E.2 Coverage report: Fine-tuning increases how much reference-tool behavior agents exercise before attempting reproduction, measured across 200 official ProgramBench instances.The study builds an instrumented coverage image for each instance by rebuilding every reference tool from its pinned upstream commit with native coverage instrumentation.
Loading 2607.27146v1…