PaperlayerView exact revision on arXiv

Source-linked AI summary

ProgramGym: Scaling Training Environments for Long-Horizon Program Reproduction Agents

Chengyi Yang, Renxu Bao, Jue Hou, Zongpei Teng, Ziheng Wang, Yujie Lin, Chengsong Huang, Junnan Zhu, Delai Qiu, Yining Wang, Shengping Liu, Fei Long, Jinsong Su

arXiv:2610.13395v1cs.SEcs.AIcs.CL

TL;DR

Program reproduction from scratch requires agents to discover behavior and design systems, but scalable verifiable training environments and long-horizon workflows remain scarce. ProgramGym constructs such environments and pairs them with a role-isolated agent workflow. On ProgramBench, both environment training and Program-Agent improve performance, with Program-Agent models exceeding some flagship models.

  • Problem

    Program reproduction from scratch requires agents to discover requirements and make design decisions, but scalable verifiable training environments and workflows for long-horizon development remain scarce.

    Source: §1 · p. 2

  • Method

    ProgramGym builds repository-based environments through feedback-driven build and black-box test loops, and uses role-isolated analysis, implementation, and review with revision management.

    Source: §1 · pp. 2–3

  • Results

    Program-Env training and Program-Agent both improve program reproduction; with Program-Agent, Qwen3.8-27B reaches 55.71% Tests Passed and Qwen3.8-27B-SFT reaches 56.25%, exceeding flagship Claude Opus 4.7.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    ProgramGym is intended to support research on coding agents that discover requirements and design systems from scratch.

    Source: §7 · p. 10

    Whole-program reproduction remains unsolved: flagship models under mini-SWE-agent remain at 0.00% Resolved, even when Tests Passed exceeds 50%.

    Source: §6.2 · p. 8

Abstract

from arXiv · show

Coding agents are increasingly used to build programs from scratch as software engineers do rather than to resolve issues within a repository or to generate a repository from a given requirements document. However, scalable verifiable training environments that force agents to discover what is required and to decide how the system should be designed remain scarce. Moreover, existing agent workflows also cannot sustain long-horizon development in which the agent explores the reference program before implementation. To address these challenges, we introduce ProgramGym, a framework consisting of Program-Env and Program-Agent for scalable training environments of long-horizon program reproduction agents. Specifically, Program-Env constructs verifiable training environments from repositories through a feedback-driven loop that repairs runtime sandbox builds and the quality of black-box tests. Using this novel pipeline, we obtain 1,648 training environments spanning C, C++, Go, and Rust. Program-Agent combines a role-isolated workflow of analysis, implementation, and review with revision management for compilation checks and failure recovery. Experiments on ProgramBench show that both Program-Env and Program-Agent improve performance on program reproduction. Fine-tuning on trajectories with these constructed environments raises the performance of Qwen3.8-27B from 34.87% to 42.84%. Program-Agent further raises the base and SFT models to 55.71% and 56.25%, both exceeding some flagship models. We will release the ProgramGym to support further research on coding agents that build programs from scratch.

1 INTRODUCTION

ProgramGym addresses the lack of scalable, verifiable training environments and workflows for long-horizon program reproduction from scratch. It combines feedback-driven environment construction with a role-isolated agent workflow, and reports improved ProgramBench performance.

1 INTRODUCTION

  • Program reproduction requires agents to discover behavior and make design choices, unlike tasks that begin with a codebase or a prescribed requirements document.The original ProgramBench evaluation found that none of its models fully solved a task, motivating training environments with reliable behavioral supervision.Source: §1 · p. 2
  • Program-Env uses feedback-driven loops to repair executable and sandbox builds and iteratively construct black-box tests from repository behavior.It provides reference executables, runtime sandboxes, documentation, and behavioral test suites as training supervision.Source: §1 · p. 2
  • On ProgramBench, with Program-Agent, Qwen3.8-27B achieved 55.71% Tests Passed and Qwen3.8-27B-SFT achieved 56.25%; both exceeded flagship Claude Opus 4.7.Source: §1 · p. 2
  • Program-Agent separates analysis, implementation, and review, using persistent records, build checks, and validated checkpoints to support long-horizon development.The experiments report improvements from both Program-Env training and Program-Agent.Source: §1 · pp. 2–3

2 RELATED WORK

Related work spans issue resolution, code generation from requirements, and program reproduction from scratch. ProgramGym targets scalable, verifiable environments for the last setting.

2 RELATED WORK

  • Issue-resolution benchmarks start from an existing codebase, while requirements-document tasks prescribe functionality and leave less room for requirement discovery or architectural design.Program reproduction instead asks agents to reconstruct observable behavior from a reference program and documentation without prescribing implementation structure.Source: §2 · p. 3
  • ProgramGym turns repositories into verifiable training environments, addressing the scarcity of realistic executable environments for program reproduction from scratch.The related-work discussion contrasts this with issue-resolution frameworks that start from an existing codebase or a well-specified task description.Source: §2 · p. 3
  • MindForge does not filter trajectories by recovered program behavior, so functionally incorrect reproductions can still count as successful supervision.The cited discussion distinguishes ProgramGym's verification focus from prior training-environment approaches.Source: §2 · p. 3

3 PRELIMINARY

Program-Env formalizes program-reproduction environments and constructs them through repository collection, executable and sandbox generation, and feedback-driven black-box test construction.

3 PRELIMINARY

  • Program reproduction asks agents to recover a program’s functionality from its documentation and reference while making software design decisions; finite tests verify only behaviors they cover.Scaling this task requires reliable builds across varied repositories and test suites that distinguish faithful reproductions from incorrect ones.Source: §3 · pp. 3–4

4.1 REPOSITORY COLLECTION

  • Program-Env collects repositories through language- and star-stratified GitHub searches, screening metadata, licenses, duplicates, benchmark overlap, and build evidence.The process yields 30,459 candidate repositories.Source: §4.1 · pp. 4–5

4.2 EXECUTABLE AND SANDBOX GENERATION

  • For each repository with a standalone executable target, the pipeline iteratively validates online dependency preparation and offline compilation, returning diagnostics when builds or loadability checks fail.The validated executable becomes the reference program, and the prepared image supplies the runtime sandbox.Source: §4.2 · p. 5

4.3 BLACK-BOX TESTS CONSTRUCTION

  • Black-box tests are built in parallel by functional module, with feedback used to expand coverage and strengthen assertions.The validation process checks that tests pass on the reference executable, behave consistently, reject dummy implementations, and use sufficiently strong assertions.Source: §4.3 · p. 5

5 PROGRAM-AGENT

Program-Agent supports long-horizon reproduction through isolated analysis, implementation, and review stages, paired with revision management for compilation validation and recovery.

5 PROGRAM-AGENT

  • Program-Agent combines role-isolated iterative development with revision management to address lost behavioral evidence and compilation failures in long-horizon work.Handoff carries evidence and reports across sessions, while the workspace and revisions preserve evolving and independently buildable code.Source: §5 · p. 5, Figure 3 · p. 6

5.1 ROLE-ISOLATED ITERATIVE WORKFLOW

  • Analysis explores documentation and the reference program; implementation builds from frozen evidence, and review checks candidate behavior against the reference.These stages use separate sessions, with evidence and stage reports passed through handoff artifacts rather than the full conversation history.Source: §5.1 · p. 6

5.2 REVISION MANAGEMENT

  • Each implementation round is gated by independent compilation, and validated source snapshots are archived for review and recovery.If a retry fails, the implementation rolls back to the last validated revision.Source: §5.2 · pp. 6–7

6 EXPERIMENTS AND ANALYSIS

The experiments evaluate ProgramGym’s training environments and agent workflow on ProgramBench, finding gains from both training on Program-Env trajectories and using Program-Agent. They also characterize the environments, iteration-budget effects, and changes in agent behavior.

6.1 EXPERIMENTAL SETTINGS

  • The evaluation uses 1,648 environments across four languages, and compares Claude Code with Program-Agent under shared container limits and up to five iterations.The environments span C, C++, Go, and Rust; each task runs with 8 CPU cores, 16 GB RAM, 30 GB storage, and an 8-hour limit.Source: §6.1 · p. 7

6.2 EFFECTIVENESS OF PROGRAMGYM (RQ1)

  • Qwen3.8-27B-SFT reaches 42.84% Tests Passed versus 34.87% for Qwen3.8-27B, while Program-Agent reaches 55.71% and 56.25% for the base and SFT models, respectively.Program-Agent runs exceed Claude Opus 4.7 (50.90%) and Gemini 3.5 Flash (53.60%) under mini-SWE-agent; Table 2 covers models and agent scaffolds.Source: §6.2 · p. 8, Table 2 · p. 8
  • Whole-program reproduction remains unsolved: flagship models have 0.00% Resolved, and Qwen3.8-27B-SFT’s Almost Resolved rate falls from 3.50% to 1.50% under Claude Code.Tests Passed can exceed 50% while Almost Resolved and Resolved remain near zero.Source: §6.2 · p. 8

6.3 TRAINING ENVIRONMENTS ANALYSIS (RQ2)

  • Program-Env’s 1,648 repositories span eight categories and four languages, with mean coverage of 72.99% and 665,512 validated test cases.Coverage has a long lower tail where behavior is only partially reachable through the black-box interface; test suites average 404 cases per repository. Figure 5 presents distributions of code lines, coverage, and test cases.Source: §6.3 · pp. 8–9, Figure 5 · p. 9

6.4 TEST-TIME SCALING (RQ3)

  • Tests Passed for Qwen3.8-27B-SFT with Program-Agent rises from 43.49% at K=1 to 54.13% at K=3 and 56.25% at K=5.Most of the gain arrives by K=3, and five iterations are comparable to DS-V4-Flash-0731 at 57.68%; Figure 6 shows results across iteration budgets.Source: §6.4 · p. 9, Figure 6 · p. 9

6.5 AGENT BEHAVIOR ANALYSIS (RQ4)

  • Both models mainly execute and probe with Claude Code, while Program-Agent increases read and git actions, most after Program-Env training.Figure 7 distributes action types across agent turns, weighting each task equally per turn and dividing its weight among tool calls.Source: §6.5 · p. 9, Figure 7 · p. 9

7 CONCLUSION

ProgramGym pairs repository-derived training environments with a role-isolated agent workflow for long-horizon program reproduction. The conclusion reports improved ProgramBench results from fine-tuning and from Program-Agent, and describes release and reproducibility materials.

7 CONCLUSION

  • Program-Env builds reference executables, runtime sandboxes, and black-box tests, producing 1,648 environments in C, C++, Go, and Rust.Program-Agent combines analysis, implementation, and review roles with revision management for compilation checks and failure recovery.Source: §7 · pp. 9–10
  • Fine-tuning raises Qwen3.8-27B from 34.87% to 42.84% on ProgramBench, and Program-Agent raises the base and SFT models to 55.71% and 56.25%.The conclusion reports that both Program-Agent results exceed some flagship models, and that extra iterations continue to recover program behavior.Source: §7 · p. 10
  • The authors plan to release ProgramGym; the reproducibility statement specifies environments, construction code, training artifacts, and materials for reconstructing the pipeline.The paper also states that generative AI tools assisted experiment implementation and editing, with the authors reviewing edits and taking responsibility for the final content.Source: §7 · p. 10, REPRODUCIBILITY STATEMENT · p. 10, AI USE STATEMENT · p. 10

A FREQUENTLY ASKED QUESTIONS (FAQS)

Program-Env builds and validates repository-based training environments through staged collection, build, and test-generation processes. Its feedback loops use diagnostics to repair build scripts and refine tests, while filtering and construction costs constrain the current environment set.

A FREQUENTLY ASKED QUESTIONS (FAQS)

  • 30,459 repositories were collected, but filtering and an average construction cost of about 82.99M tokens limited the set to 1,648 tasks.Repositories may be excluded for Linux build failures, hardware or display requirements, or compilation or instrumentation exceeding 3,600 seconds; the pipeline is automated for further construction.Source: A FREQUENTLY ASKED QUESTIONS (FAQS) · p. 14, §B.1 · p. 15

B.1 REPOSITORY COLLECTION DETAILS

  • The four-stage collection pipeline searches, validates metadata, locks each repository to a commit, and detects build configuration.It targets repositories primarily using C, C++, Go, or Rust; detected build markers serve only as a proxy and do not establish successful compilation.Source: §B.1 · p. 15

B.2 BUILD PROCEDURE DETAILS

  • Program-Env prepares project-specific toolchains and dependencies in download.sh, then uses compile.sh to build the executable offline.Offline compilation checks that the prepared environment contains the resources required to build without network access.Source: §B.2 · p. 16

B.2.1 BUILD-STAGE VALIDATION

  • Build validation combines strict script linting with online preparation and offline execution checks for a usable, correctly linked executable.The linter rejects masked failures and placeholders; execution checks scripts, artifact presence and format, and dynamic linkage.Source: §B.2.1 · p. 16

B.2.2 BUILD-STAGE FEEDBACK

  • When a build fails, structured diagnostics and suggested fixes guide script revisions, with construction stopping after four attempts.In the illustrated repair, the agent adds the missing libopenexr-dev package and then passes preparation, compilation, and linkage checks.Source: §B.2.2 · pp. 17, 19, Figure 10 · p. 19

B.3 DISPATCHER DETAILS

  • The dispatcher decomposes repositories, instruments coverage, constructs a test-generation sandbox, and extracts documentation; independent components run concurrently.Instrumentation depends on the runtime sandbox, and sandbox construction depends on the instrumented program.Source: §B.3 · p. 20, Figure 11 · p. 20

B.3.1 TASK DECOMPOSITION

Program-Env divides repository-scale test generation into focused modules, instruments programs with a rule-first fallback, and supports parallel work in a shared test-generation sandbox. It then validates and refines generated tests to produce a reliable black-box suite.

B.3.1 TASK DECOMPOSITION

  • A repository too large for one agent is decomposed into cohesive, non-overlapping modules with complexity-based budgets and parallel subagents.The score uses production-code lines and runtime dependencies; easy, medium, and hard repositories receive module budgets of 4, 6, and 8, respectively.Source: §B.3.1 · pp. 20–21

B.3.2 COVERAGE INSTRUMENTATION

  • Language-specific rules first generate coverage-instrumentation candidates, with an agent repairing them only when independent validation fails.The fallback addresses repository-specific build behavior that rules may not capture and the risk of conflicting build flags.Source: §B.3.2 · p. 21

B.3.3 TEST GENERATION SANDBOX CONSTRUCTION

  • A shared repository-level test-generation sandbox avoids separate program copies, while isolated copy-on-write run state prevents subagents from writing to the same coverage counters.Space overhead drops from O(k) to O(1) in the number of subagents.Source: §B.3.3 · p. 21, §1 · pp. 21–22

B.3.4 DOCS EXTRACTION

  • Docs Extraction selects repository documents and otherwise hard-to-synthesize test assets, while excluding source-code directories from the solving-side documentation.A manifest records selected files, and export copies only those listed files.Source: §B.3.4 · p. 22

B.4 TESTS CONSTRUCTION DETAILS

  • Test construction iteratively generates, validates, and refines tests for modules using environment feedback.Validation reports rejected tests and warnings for revision in the next iteration.Source: §B.4 · p. 22, §B.4.1 · p. 23

B.4.1 VALIDATION FOR TEST CONSTRUCTION

  • Test validation checks faithfulness, stability, and effectiveness through reference runs, repeated runs, dummy-program execution, and static weak-assertion linting.The linter rejects defective tests and flags risks for review against task semantics.Source: §B.4.1 · p. 23

B.4.2 FEEDBACK FOR TEST GENERATION

Program-Env turns execution, coverage, and program-analysis data into actionable feedback for test repair and new test targeting. In case studies, this feedback improves test quality and expands coverage; termination rules bound iterative generation.

B.4.2 FEEDBACK FOR TEST GENERATION

  • Each round repairs flagged tests before writing new ones and turns execution logs, validation reports, and coverage into guidance for repair, uncovered targets, and reachable public entries.Repair comes first because identified local fixes are cheaper than new tests and can prevent defects from accumulating.Source: §B.4.2 · p. 25
  • Core coverage counts only core files and weights valid units: executable lines for C/C++ and Rust, and statement-weighted basic blocks for Go.Unattributable units are excluded, and reports with no valid units or failed collection are discarded rather than treated as zero coverage.Source: §B.4.2 · pp. 25–26
  • Targets prioritize partially covered files, falling back to uncovered files only when no partially covered files remain; zero-coverage files are deferred without execution evidence.The system limits each round’s target list to a manageable subset.Source: §B.4.2 · p. 26
  • Static analysis maps uncovered lines to enclosing functions and traverses call graphs in reverse to identify public entries that black-box tests can use to reach them.The pipeline queries Joern or CodeQL only for uncovered targets, beginning after the first coverage measurement.Source: §B.4.2 · pp. 26–27
  • Because subagent context does not persist between rounds, each agent writes a summary of completed, skipped, and remaining targets for the next round.The next round reads this summary before acting.Source: §B.4.2 · p. 27
  • Feedback raised Violet’s linter pass rate from 0% to 98.3% and reduced its false-pass rate from 1.41% to 0%; Kubesess coverage rose from 4.88% to 85.77%.On COLMAP, dynamic coverage guidance raised module coverage from 29.92% to 42.86%, while a call-graph-revealed route raised it from 36.41% to 45.47%.Source: §B.4.2 · p. 28

B.4.3 LOOP TERMINATION

  • Subagents stop at a coverage threshold or complexity-scaled round limit, with early stopping when three-round gains in coverage and linter pass rate both fall below set thresholds.The earliest early stop is round 5; the thresholds are less than 0.25% coverage gain and less than 20% linter pass-rate gain.Source: §B.4.3 · p. 28, Table 12 · p. 28

B.5 ADDITIONAL DATASET STATISTICS

The appendix characterizes dataset diversity and construction quality, then details how Program-Agent orchestrates isolated analysis, implementation, and review stages with build validation and recovery.

B.5 ADDITIONAL DATASET STATISTICS

  • The 1,648 repositories span long-tailed community activity and projects from hundreds to over one million code lines, with eight functional categories.Stars and commits average 981.77 and 499.19; production code averages 11,048.92 lines and 102.63 files per repository.Source: §B.5 · p. 30, Table 13 · p. 30, Figure 14 · p. 31
  • Constructing the 1,648 environments used 136.77B tokens, while the final suites contain 665,512 tests and reach 72.99% mean coverage.The pipeline wrote 98.50% of tests from scratch; coverage remains high even for the largest repositories, where test count and token use grow with size.Source: §B.5 · pp. 31–32, Figure 17 · p. 32, Figure 18 · p. 32

C.1 PHASE ORCHESTRATION

  • Program-Agent runs one initial analysis, K implementation rounds, and K−1 reviews, transferring persistent stage information through Handoff artifacts.Each stage runs in a fresh model session, and review artifacts guide only the next implementation round.Source: §C.1 · p. 33

C.2 WORKSPACE AND ACCESS ISOLATION

  • Analysis and review use disposable workspaces, while implementation alone edits the shared source workspace and cannot access the reference executable.Completed Handoff artifacts are made read-only, and review can execute the reference and independently built candidate without modifying source.Source: §C.2 · pp. 33–35

C.3 BUILD VALIDATION AND FAILURE RECOVERY

  • Machine checks require validated stage reports and source state, while each implementation revision must independently build before it becomes the last-stable version.Failed attempts receive bounded diagnostics in fresh sessions; failed implementation/build attempts are reset, and recovery restores the last independently built source and executable.Source: §C.3 · pp. 35–36

D.1 TRAINING DETAILS

The training details describe trajectory collection and supervised fine-tuning, then specify sequence handling and the loss mask used for training.

D.1 TRAINING DETAILS

  • Qwen3.8-27B is fine-tuned for two epochs on 64,505 trajectories collected from task-level rollouts across all 1,648 environments.Context resets or switches can split a task rollout into multiple segments, so the trajectory count does not represent independent environment rollouts.Source: §D.1 · p. 36, Table 17 · p. 36
  • Training packs sequences into blocks of up to 262,144 tokens, discards examples exceeding the maximum length, and computes loss on eligible assistant tokens.Tool calls unavailable in an environment are masked from the loss; training uses BF16 with full activation recomputation.Source: §D.1 · p. 37

D.2 TOKEN COST ANALYSIS

Program-Agent improves test pass rates while using fewer output tokens than Claude Code, and additional implementation rounds raise performance further.

D.2 TOKEN COST ANALYSIS

  • Qwen3.8-27B rises from 34.87% to 55.71% Tests Passed with Program-Agent, while Qwen3.8-27B-SFT rises from 42.84% to 56.25%.Both Program-Agent runs exceed flagship Claude Opus 4.7 under mini-SWE-agent and use fewer output tokens than Claude Code.Source: §D.2 · p. 37, Figure 19 · p. 37, Figure 20 · p. 37
  • Program-Agent’s Tests Passed rate increases from 43.49% at K=1 to 54.13% at K=3 and 56.25% at K=5, while each budget uses fewer tokens than Qwen3.8-27B with Claude Code.Source: §D.2 · p. 38

D.3 EXTRA ANALYSIS OF AGENT BEHAVIOR

The analysis examines how Program-Agent changes pass-rate distributions and implementation size, including differences across model configurations and reference languages.

D.3 EXTRA ANALYSIS OF AGENT BEHAVIOR

  • Program-Agent raises Qwen3.8-27B tasks passing at least 50% of tests from 40.5% to 62.0% and at least 75% from 18.0% to 34.0%.For Qwen3.8-27B-SFT, the corresponding shares rise from 43.5% to 62.0% and from 9.5% to 29.5%; the two Program-Agent curves stay close over most thresholds.Source: §D.3 · p. 38, Figure 21 · p. 38
  • Among solutions passing at least 75% of tests, median generated code is 1,881 lines versus 2,596 reference lines, and the median model/reference ratio is 0.64.Across 182 test runs covering 93 tasks, 67.6% of runs fall below the parity line; solutions above parity are concentrated in smaller projects.Source: §D.3 · p. 38, Figure 22 · p. 39
  • Program-Agent raises median LoC ratios from 0.58 to 0.75 for Qwen3.8-27B and from 0.49 to 0.67 for Qwen3.8-27B-SFT.Its upper quartiles reach 1.53 and 1.36, compared with 1.01 and 0.75 for the matching Claude Code configurations.Source: §D.3 · pp. 38–39, Figure 23 · p. 39
  • Median model/reference LoC ratios vary by language: 0.39 for C/C++, 0.55 for Go, and 0.72 for Rust.Generated implementations meet or exceed reference size in 13.3% of C/C++ cases and 35.5% of Rust cases.Source: §D.3 · p. 39, Figure 24 · p. 39
Loading 2610.13395v1…