PaperlayerView exact revision on arXiv

Source-linked AI summary

Envs-FORGE: Frontier-Optimized Reward-Grounded Environment Synthesis for Agent RL

Xiaojun Wu, Cehao Yang, Honghao Liu, Xueyuan Lin, Zhichao Shi, Hao Zhou, Xuhui Jiang, Chengjin Xu, Jia Li, Jian Guo

arXiv:2608.14312v1cs.CL

TL;DR

Terminal-agent RL needs executable environments with reliable rewards and policy-matched difficulty. Envs-FORGE selects a verifier-reward-grounded synthesis action per seed and uses it for synchronized, gold-verified rewriting. It achieves the highest Pass@1 among evaluated methods on tb-core, tb-2.0, and SWE-bench Verified, improving over Base by 9.2, 6.4, and 3.7 percentage points.

  • Problem

    Terminal-agent RL requires executable environments with reliable rewards and difficulty matched to the current policy.

  • Method

    Envs-FORGE estimates verifier-derived seed difficulty, selects a projection–direction action per seed with a MILP, and conditions synchronized gold-verified environment rewriting.

  • Results

    Envs-FORGE achieves the highest Pass@1 on tb-core, tb-2.0, and SWE-bench Verified, improving over Base by 9.2, 6.4, and 3.7 percentage points.

  • Takeaways & Limitations

    Environment synthesis for agent RL should choose how each seed moves relative to the current policy before rewriting it.

  • Takeaways & Limitations

    The study evaluates only per-seed MILP mode at a fixed 100-environment export size; portfolio-level quotas and broader scaling remain outside the comparison.

Abstract

from arXiv · show

Reinforcement learning (RL) for terminal agents needs executable training environments with reliable rewards and useful difficulty. Fixed recipes such as few-shot, Self-Instruct, and Evol-Instruct apply the same prompting policy to every seed, even when the current policy would benefit from a harder, easier, or simply different task. We present Envs-FORGE, a prompting policy that converts verifier rewards into per-seed environment-synthesis actions. Envs-FORGE estimates seed pass rates, scores six projection--direction actions around a target learning frontier, and solves a per-seed mixed-integer linear program (MILP) to choose the action that conditions generation. The selected action drives synchronized rewriting of the instruction, fixtures, oracle solution, tests, and Docker environment; only gold-verified bundles enter RL training. The indexed MILP form also supports optional soft skill coverage for portfolio planning. On Qwen 3.5 35B, Envs-FORGE improves Pass@1 over Base by 9.2 percentage points on tb-core (40.0% to 49.2%) and 6.4 points on tb-2.0 (23.0% to 29.4%), exceeding the strongest fixed-recipe baseline by 2.4 and 2.1 points. It reaches 77.1% on SWE-bench Verified versus 73.4% for Base, and improves tb-core by 6.8--9.2 points across the evaluated 4B--35B models. All synthesis methods export 100 verified environments and use 2.27M--2.88M synthesis tokens, placing the comparison at the same downstream training-set size and the same operational scale. The source code is available at https://github.com/DataArcTech/DataArc-SynData-Toolkit/.

1 Introduction

Envs-FORGE selects environment-synthesis actions per seed from verifier rewards, targeting difficulty near the current policy’s learning frontier. It combines projection–direction selection, synchronized verified environment construction, and resource-matched evaluation, improving Pass@1 over Base and fixed-recipe baselines.

  • Motivation: RL terminal agents need executable environments with reliable test rewards and difficulty matched to the current policy.The paper situates this need in interactive terminal-agent systems and containerized terminal-task evaluation.
  • Method: Envs-FORGE estimates each seed’s verifier-reward pass rate, scores six projection–direction actions, and selects one with a per-seed MILP.The six actions combine increase, reduce, or diversify projections with in-depth or in-breadth evolution directions.
  • Method: The method treats fixed synthesis recipes as restricted action masks and supports optional soft skill coverage through an indexed mixed-integer formulation.Its construction also includes synchronized artifact rewriting and gold verification.
  • Results: 49.2% versus 40.0% raises Pass@1 over Base on tb-core, while 29.4% versus 23.0% raises Pass@1 on tb-2.0 for Qwen 3.5 35B.Envs-FORGE exceeds the strongest fixed-recipe baseline by 2.4 and 2.1 percentage points, respectively.
  • Results: 100 verified environments are exported by each synthesis method, with total synthesis-token use ranging from 2.27M to 2.88M.The comparison reports this resource accounting alongside downstream evaluation.

2 Related Work

Prior work provides executable terminal-agent benchmarks and diverse synthetic-environment generation methods, while difficulty-aware approaches focus on allocating synthesis effort. Envs-FORGE instead selects which transformation to apply to each executable seed based on its estimated pass rate.

  • Terminal agents and benchmarks: SWE-agent and OpenHands use tool-using execution loops, while Terminal-Bench and SWE-bench Verified evaluate containerized terminal tasks and repository repair.These benchmarks provide executable evaluation targets rather than a policy-conditioned rule for synthesizing new training environments.
  • Synthetic instruction and environment generation: Self-Instruct, WizardLM, and AgentInstruct expand instruction data through self-generated examples, complexity evolution, or agentic flows.CLI-Gym, SkillSynth, TermiGen, and TerminalTraj modify the terminal-specific synthesis substrate through runtime, skill-graph, container, or trajectory operations.
  • Difficulty-aware selection: Difficulty-aware methods allocate more synthesis effort to difficult data, whereas Envs-FORGE chooses which transformation to apply to an executable seed.DART-Math assigns more response-generation trials to difficult queries, while targeted tabular synthesis trains generators on observations identified as hard.
  • Difficulty-aware selection: Fixed prompt recipes apply the same rewrite strategy to every seed, while Envs-FORGE estimates pass rate and selects among increase, reduce, and diversify projections.The selection aims to land near the learning frontier.

3 Method

Envs-FORGE separates frontier-aware action selection from synchronized environment synthesis, using verifier rewards and a per-seed MILP to choose how each seed should change. The selected action conditions joint rewriting and gold verification, so only consistent, runnable environments enter training.

  • Frontier-aware selection: Envs-FORGE estimates each seed’s pass rate from verifier rewards, scores six projection–direction candidates, and solves one per-seed MILP to select an action.The estimator remains fixed during optimization, while transfer priors rank candidates before generation.
  • Frontier-aware selection: The increase, reduce, and diversify projections respectively make easy tasks harder, create bridge tasks for overmatched seeds, or vary requirements while preserving approximate difficulty and skill family.Evolution directions are in_depth and in_breadth.
  • MILP formulation: The MILP selects one action per seed, prioritizes frontier value, favors compact skill realizations, and optionally penalizes unmet skill-coverage targets with bounded slack.Reported experiments use a slack cap of ξ̄ = 0.2; eligibility, overlap, and prompt-length constraints restrict feasibility before materialization.
  • Joint materialization: The selected action conditions synchronized rewriting of the instruction, fixtures, oracle solution, tests, and Docker environment under a preserved skill subgraph and consistency contract.The prompt forbids instruction-only edits and hidden test requirements.
  • Verification: Only environments passing schema, safety, length, Docker, test, overlap, and gold-verification checks enter training, with the oracle required to obtain reward 1 under generated tests.This makes synchronized, runnable, and gradeable artifacts part of the training-data definition.

4 Experiments

Envs-FORGE is evaluated against fixed prompting recipes using exactly 100 gold-verified environments at comparable synthesis scale. It achieves the strongest downstream Pass@1 across benchmarks and model sizes while preserving the fixed training-set endpoint.

  • Main comparison: Envs-FORGE reaches 49.2% on tb-core and 29.4% on tb-2.0, gaining 9.2 and 6.4 points over Base and 2.4 and 2.1 over the strongest fixed baselines.It surpasses few-shot, Self-Instruct, and Evol-Instruct in the main Qwen 3.5 35B comparison.
  • Benchmark coverage: Envs-FORGE leads all three benchmarks with 49.2% on tb-core, 29.4% on tb-2.0, and 77.1% on SWE-bench Verified.Relative to Base, gains are +9.2, +6.4, and +3.7 points; margins over the strongest fixed recipe are +2.4, +2.1, and +1.3 points.
  • Model coverage: Envs-FORGE improves tb-core over Base by 6.8 points at 4B, 7.2 at 9B, 8.1 at 27B, and 9.2 at 35B.The model ablation reports gains across all four evaluated Qwen 3.5 sizes.
  • Synthesis cost: Each synthesis method exports exactly 100 gold-verified environments, with total synthesis use spanning 2.272M–2.881M tokens and 194–210 materialized task directories.These aggregates measure reaching the fixed acceptance endpoint rather than enlarging the downstream training set.
  • Experimental design: The evaluation compares complete prompting policies at a fixed 100-task export and comparable synthesis scale, with additional studies isolating portfolio mode, solver-off selection, transfer-prior sensitivity, and Evol-Instruct variants.The main comparison holds artifact generation and verification fixed across prompting policies.

5 Conclusion

Envs-FORGE is a frontier-aware prompting policy that selects per-seed synthesis actions from verifier-derived difficulty estimates and conditions synchronized, gold-verified environment rewriting. It achieves the highest Pass@1 among evaluated methods on tb-core, tb-2.0, and SWE-bench Verified.

  • Synchronized, gold-verified environment rewriting is conditioned on the selected synthesis action.
  • Envs-FORGE selects a per-seed synthesis action using verifier-derived difficulty estimates.The selected action conditions environment synthesis.
  • 9.2, 6.4, and 3.7 percentage points are the improvements over Base on tb-core, tb-2.0, and SWE-bench Verified, respectively.Envs-FORGE obtains the highest Pass@1 among evaluated methods on all three benchmarks.

Limitations

The study evaluates only Envs-FORGE’s per-seed MILP setting with active coverage slack capped at ξ̄ = 0.2, while other portfolio and component configurations remain outside the comparison.

  • Evaluation scope: The reported synthesis run uses per-seed MILP with active coverage slack capped at ξ̄ = 0.2.This is the evaluated Envs-FORGE configuration.
  • Evaluation scope: Portfolio-level skill quotas with different export targets or slack budgets are supported but not evaluated.These settings are outside the present comparison.
  • Evaluation scope: The study compares complete prompting policies at a fixed 100-environment export size, leaving finer component studies for future analysis.Potential component studies include solver-off selection and transfer-priority configurations.

Ethical Considerations

The work uses benchmark tasks and existing records without collecting human-subject data, while recognizing that stronger terminal-agent environments may improve reliability but increase capability and persistence risks.

  • No human-subject data are collected; experiments use benchmark tasks and existing task and experiment records.
  • More effective environment synthesis may reduce repeated operational failures and improve software-engineering agent reliability.
  • The same synthesis may increase agent capability and persistence in command-line settings.

Information About Use of AI Assistants … A.4 Preprocessing, Training, and Evaluation Protocol

The appendix documents Envs-FORGE’s action semantics, solver-to-generation boundary, artifact verification, and reproducible training and evaluation protocol. It also states that AI-assisted tools were used only for language refinement, while the authors retained responsibility for the manuscript’s technical content and results.

  • Information About Use of AI Assistants: AI-assisted tools, including GPT-5 and DeepSeek-V4, were used only for proofreading, grammatical correction, and linguistic polishing.The authors remained responsible for final content, technical claims, citations, experimental results, and verification.
  • Appendix Overview: Appendix A covers optimization and reproducibility details, Appendix B MILP assembly and solver behavior, Appendix C prompt clauses, and Appendix D compute resources.The appendix also expands action semantics, solver boundaries, environment verification, and qualitative frontier cases.
  • A.1 Action Semantics and Baseline Restrictions: Envs-FORGE selects both projection and evolution direction per seed from the full feasible region using policy-relative frontier scores.Projection controls difficulty movement, while direction chooses same-skill-chain depth or neighboring-skill breadth; optional portfolio mode adds shared skill-coverage targets.
  • A.2 Candidate Construction and Solver Boundary: Each seed produces six candidate action variants and one local MILP instance after pass-rate estimation, transfer-heuristic application, frontier-utility computation, and skill-node derivation.Candidate prompt length, split membership, and overlap or validity indicators are treated as constants before solving.
  • A.2 Candidate Construction and Solver Boundary: The artifact edit mask is derived deterministically from the selected action, while generation and executable verification enforce consistency across instructions, fixtures, solutions, tests, and containers.The MILP selects the discrete action; the artifact mask and materialization remain outside the solver.
  • A.3 Artifact-Consistency Contract: Accepted environments must synchronize instructions, fixtures, oracle solutions, executable tests and reward logic, and container contents without exposing oracle solutions or hidden tests.Added requirements must appear in instructions and verifier tests, while removed requirements must disappear from the oracle and tests.
  • A.3 Artifact-Consistency Contract: A bundle is accepted only after static checks and isolated executable checks pass and the verifier emits reward 1.The gold reward certifies internal task consistency and is distinct from the downstream policy reward used to estimate learning difficulty.
  • A.4 Preprocessing, Training, and Evaluation Protocol: Each method contributes exactly 100 accepted bundles after normalization and filtering, with failed candidates, repairs, and intermediate materializations excluded from downstream RL.Preflight validation rechecks splits and overlap filters, builds representative containers, and runs the synthesis oracle-plus-test path before training workers launch.

A.5 Qualitative Frontier Cases

Five verified case studies show Envs-FORGE selecting hardening, reduction, or breadth diversification according to seed difficulty while synchronously updating each environment artifact under a single contract. The traces preserve the intended reasoning skill, achieve frontier-oriented projected rates, and obtain reward 1 after static validation.

  • Case II: verifier-facing hardening: 1.0 to 0.750: depth increase hardened a saturated multi-format merger by specifying exact keys, missing-field handling, quoting, headers, and row order.The contract retained CSV, JSONL, and JSON parsing, making superficially plausible but incorrect merges insufficient.
  • Case III: deterministic bridge: 0.0: reduction transformed a systemd deployment task into deterministic log reasoning while preserving restart-loop detection, severity counting, malformed-record validation, and error ranking.The reduction was selected because the seed was unsolved in the recorded rollouts.
  • Case IV: security-state isolation: 0.0: reduction isolated security-state reasoning from a token-service web stack by fixing configuration, event records, and reference time while retaining token validation, repair, labeling, scoring, and ranking.The reduced environment removed Java, web-server, database, and concurrency overhead.
  • Case V: frontier-preserving diversification: 0.533 to 0.533: breadth diversification preserved the PGN seed’s pass rate while changing the illegal move from 15. Kf9 to its legal repair 15. Kf1, with frontier score 0.9862.The action retained chess parsing, header preservation, legal-move validation, and structured repair because the seed was already near the frontier.
  • Cross-case interpretation: Across cases, Envs-FORGE hardens saturated or easy seeds, reduces infrastructure that masks target reasoning, and diversifies seeds already near the frontier.Instruction, fixtures, oracle, tests, and environment are updated under one artifact contract, and static validation yields reward 1; downstream performance is reported separately.

B MILP Solution Procedure

Envs-FORGE instantiates per-seed synthesis-action selection as a PySCIPOpt model solved by SCIP, choosing among six projection–direction candidates under optional skill-coverage constraints. A recorded 100-seed run achieved an optimal solution and exported one selected action per seed.

  • Model assembly: Per seed, the MILP selects exactly one of six candidates, each defining a complete pre-materialization contract for environment synthesis.The implementation uses PySCIPOpt as the modeling interface and SCIP as the solver.
  • Model assembly: Required and forbidden skills are linked directly to selected actions, while optional skills use bounded activation with a small penalty, avoiding big-M constants.Uniform candidate–skill indexing keeps the decoded solver payload auditable.
  • Branch-and-cut search: SCIP solves the linear mixed-integer model through presolve, LP relaxations, branching, cutting planes, propagation, and primal heuristics.The exact search is appropriately described as branch-and-cut.
  • Recorded environment-synthesis solve: 100 seeds, six candidates per seed, and 600 primary action variables produced 100 requested environments with status=optimal and objective value 49.9104.The run used active coverage slack cap ¯ξ = 0.2, and the target count equaled the seed count.
  • Fallback and correctness boundary: Unavailable PySCIPOpt or non-optimal SCIP status triggers backend-explicit fallback to enumeration for at most 24 candidates or deterministic coverage-aware greedy selection for larger instances.Solver optimality certifies the discrete selection problem, while generated-environment consistency is verified separately after synchronized materialization.

C Prompt and Task-Instruction Excerpts

The excerpts define a shared executable-environment contract for fixed prompting baselines and show how Envs-FORGE replaces predetermined strategy injection with solver-conditioned, skill- and artifact-constrained synthesis. The synthesized bundles preserve internal consistency through synchronized artifacts, static validation, and oracle reward 1.

  • C.1 Shared Baseline Synthesis Contract: Fixed baselines share one artifact-generation template and differ only in injected strategy and evolution-direction fields.Few-shot remains structurally close, Self-Instruct creates a related same-domain task, and Evol-Instruct selects depth or breadth before policy-relative rewards.
  • C.1 Shared Baseline Synthesis Contract: The shared contract requires synchronized instructions, fixtures, oracle solutions, tests, environments, and helper files in a valid executable bundle.It also requires stated verifier expectations, pinned runtimes, valid build contexts, and reward 0 or 1 even when tests fail.
  • C.1 Shared Baseline Synthesis Contract: Repair prompts preserve each baseline policy while correcting parsing, build, static-validation, or oracle-verification failures.Repairs return a complete JSON object and re-evaluate the build–solve–test path toward oracle reward 1.
  • C.2 Solver-Conditioned Envs-FORGE Prompt: Envs-FORGE conditions generation on an optimized action, passing seed estimates, reward bands, projections, frontier scores, target difficulty, edit masks, and skill subgraphs before seed files.Its objective is to project tasks toward the current policy’s learning frontier while preserving the target skill subgraph and materializing one synchronized executable environment.
  • C.2 Solver-Conditioned Envs-FORGE Prompt: 6 action variants combine increase_complexity, reduce_complexity, or diversify_frontier with in_depth or in_breadth.The projections respectively make easy seeds harder, turn overly hard seeds into bridge tasks, or diversify seeds near the frontier while preserving relevant skills.
  • C.2 Solver-Conditioned Envs-FORGE Prompt: Envs-FORGE’s artifact masks make required edits explicit across instruction, data, solution, tests, environment, and helpers, while skill constraints govern preserved, added, and removed nodes.Every required artifact must be synchronized, and edits must remain within operator families selected by the mask.
  • C.2 Solver-Conditioned Envs-FORGE Prompt: Unlike fixed baselines, Envs-FORGE injects a policy-relative solver decision plus explicit skill and artifact constraints under the same executable-environment and gold-verification standard.Across five synthesized instruction examples, fixtures, oracle solutions, tests, and environments were synchronized; static validation passed and every oracle received reward 1.

D Compute Resources

The training setup uses fixed rollout, batching, sequence-length, and interaction limits, with a 1 × 10−6 actor learning rate and one training epoch. Current runs require two H800 80 GB GPUs.

  • Training Configuration: Training samples 8 rollouts per prompt at temperature 1.0 and top-p = 0.9, with batch, mini-batch, and micro-batch sizes all set to 1.Maximum prompt and response lengths are 8192 tokens.
  • Training Configuration: Each trajectory allows at most 50 agent-environment interaction steps and has a 900-second timeout.The actor learning rate is 1 × 10−6, and training runs for one epoch with automatic checkpoint resume enabled.
  • Compute Resources: Current runs require two H800 80 GB GPUs.This hardware requirement accompanies the rollout and optimization configuration described above.
Loading 2608.14312v1…