PaperlayerView exact revision on arXiv

Source-linked AI summary

NexForge: Scaling Agent Capabilities through Requirement-Driven Task Synthesis for LLMs

Jiarong Zhao, Zhikai Lei, Zhiheng Xi, Rui Zheng, Hang Yan, Jie Zhou, Qin Chen, Liang He

arXiv:2607.14186v6cs.SEcs.AIcs.LG

TL;DR

Scaling executable agent training data is labor-intensive and tied to predefined substrates, limiting efficient coverage across domains. NexForge instead synthesizes executable tasks and expert trajectories from high-level capability requirements, improving performance across terminal and office benchmarks, including 58.4% on Terminal-Bench 2.0 at 43.2K tasks.

  • Problem

    Executable agent training data remains scarce and expensive to scale because task design, materials, environments, and trajectories require labor-intensive, domain-specific preparation.

  • Method

    NexForge uses high-level capability requirements to drive demand discovery, task compilation, executable workspace instantiation, and expert trajectory generation.

  • Results

    58.4% on Terminal-Bench 2.0 at 43.2K tasks, while the same framework improves performance across terminal and office capabilities.

  • Takeaways & Limitations

    NexForge transfers across capability domains and scales training-data production without domain-specific infrastructure.

Abstract

from arXiv · show

Scaling executable agent training data for LLM post-training is bottlenecked by substrate-bound methods that tie task generation to predefined tools, repositories, or skill graphs: expanding coverage requires manual substrate engineering, each new domain demands a bespoke pipeline, and the resulting task distributions often reflect substrate biases rather than real-world demand. We introduce NexForge, a requirement-driven framework that takes high-level capability requirements as input and synthesizes diverse, executable agent tasks and expert trajectories for SFT. NexForge first investigates real-world demand to construct scenarios and task profiles, then performs distribution-aware compilation to generate task directives. For each directive, NexForge automatically retrieves or constructs the required files, dependencies, and runtime configurations, and finally collects expert rollouts to produce training trajectories. Without domain-specific infrastructure, NexForge produces 3.6K terminal and 2K office tasks, improving Qwen3.5-35B-A3B Base from 22.5\% to 52.0\% on Terminal-Bench 2.0 and from 813 to 1338 Elo on GDPval; scaling further to 43.2K terminal tasks yields 58.4\%, on par with Claude Opus 4.6 equipped with Claude Code. Scaled further, NexForge-synthesized data contributes to the training of Nex-N2, a family of publicly available agent models that lift Qwen3.5-397B-A17B to 75.3\% on Terminal-Bench 2.1 and to 1585 Elo on GDPval---achieving state-of-the-art open-source performance and surpassing several frontier proprietary systems. Nex-N2 models are available at https://nex.sii.edu.cn/.

1 INTRODUCTION

NexForge addresses the labor-intensive scarcity of executable agent training data by decoupling task generation from predefined substrates and driving synthesis from high-level capability requirements. It produces diverse executable tasks and expert trajectories across terminal and office domains without domain-specific infrastructure.

  • Motivation: Agent post-training data is scarce because it requires task descriptions, grounded materials, executable environments, and long-horizon interaction trajectories.Producing these components at scale remains labor-intensive and expensive.
  • Requirement-Driven Formulation: NexForge begins with capability requirements, identifies representative task types and frequencies, and then determines the materials and runtimes needed for execution.This separates what an agent should practice from how that practice is made executable.
  • NexForge: NexForge investigates real-world demand, compiles distribution-aware task directives, instantiates executable workspaces, and collects expert trajectories for SFT.The same end-to-end framework supports terminal and office capabilities without domain-specific infrastructure.
  • Nex-N2: NexForge also powers the publicly available Nex-N2 family with scaled training signals spanning terminal operations, office productivity, and long-horizon agentic reasoning.These synthesized tasks provide diverse executable training signals across domains.
  • Limitations of Existing Methods: Substrate-bound synthesis is limited by input-bounded scale, high transfer cost, and substrate-biased task distributions.Existing methods tie generation to predefined repositories, tools, skills, or execution traces, making each domain require bespoke infrastructure.

2 RELATED WORKS

Prior work synthesizes instructions and executable agent data through decomposition, simulated environments, execution traces, tool ecosystems, repositories, and skills. These approaches establish executable environments’ importance but remain substrate-bound, motivating NexForge’s requirement-driven starting point.

  • Instruction and task-distribution synthesis: Instruction synthesis methods decompose capability spaces using bootstrapped seeds, iterative complexity augmentation, hierarchical taxonomies, and explicit task control.Examples include Self-Instruct, Evol-Instruct, GLAN, and TRouter.
  • Agent task and environment synthesis: Executable agent-data research spans simulated environments, tool ecosystems, execution traces, subtask composition, domain-specific repositories, and skills.The passage identifies these as major directions in recent agent task and environment synthesis.
  • Agent task and environment synthesis: These methods demonstrate the importance of executable environments but share substrate-bound limitations.NexForge addresses all three limitations by starting from high-level capability requirements.

3 METHOD

NexForge converts high-level capability requirements into diverse executable tasks and training trajectories through demand discovery, distribution-aware task compilation, and environment-based trajectory generation. Its design separates intended work distributions from task materials, enabling scalable synthesis without manually authored answers or task-specific verifiers.

  • Framework overview: The three-stage pipeline performs demand discovery, task compilation, and trajectory generation, determining capabilities before constructing executable materials and runtime environments.Demand discovery builds task demand profiles and scenario reservoirs; compilation creates directives; trajectory generation collects teacher interactions.
  • Framework inputs and outputs: NexForge takes a capability requirement, task count, and optional batch constraints, then outputs executable task packages containing instructions, materials, files, dependencies, and runtime configurations.The packages are designed for trajectory collection and do not require manually authored reference answers or task-specific verifiers.
  • Demand discovery: Real-world evidence from workflows, documentation, role descriptions, examples, and artifacts is organized into weighted task-demand profiles and diverse scenario reservoirs.Evidence-derived weights control option sampling, while scenario filtering removes duplicates and semantically redundant contexts.
  • Task compilation: The composer combines each scenario with profile options, sequentially filtering fields for compatibility and sampling according to profile weights and batch constraints to produce internally consistent directives.Directives specify intended work before repositories or documents are selected, constraining downstream material construction.
  • Environment instantiation and trajectory generation: Each directive becomes an executable workspace through material mining, blueprint planning, and workspace generation and validation, after which teacher interactions are collected under fixed budgets and cleaned into standardized trajectories.Collected trajectories record responses, tool calls, observations, failures, and recovery behaviors.

4 DATASETS

NexForge produces distinct terminal and office corpora from capability requirements, with controlled scaling variants, balanced bilingual difficulty distributions, and task-specific grounding. The resulting datasets vary substantially in task types and resources while avoiding concentration in a few repeated templates.

  • Corpus construction: NexForge produces Terminal-3.6K and Office-2K, plus Terminal-2K, Terminal-43.2K, and Office-22K for data-scaling studies.Terminal-2K contains the first 2,000 tasks of Terminal-3.6K, while the additional corpora use the same synthesis pipeline.
  • Capability coverage: Terminal tasks cover command-line work, while office tasks cover spreadsheets, documents, aggregation, summarization, drafting, planning, and evidence-based recommendations.Both specifications use a 50/50 Chinese–English split and balanced difficulty distributions.
  • Trajectory generation: 3 teacher rollouts per task are collected for Terminal-2K, Terminal-3.6K, and Office-2K, whereas Terminal-43.2K and Office-22K use 1 rollout per task.All synthesis agents and the composer are driven by GPT-5.5; trajectory conversion and task-independent cleaning are applied afterward.
  • Dataset composition: Terminal-3.6K realizes 24 task types and is primarily grounded in public repository adaptation, while Office-2K realizes 15 task types and more often uses public documents, spreadsheets, and reports.The specifications therefore induce clearly different data distributions and grounding characteristics.
  • Dataset diversity: 20.6% and 15.1% are the shares of the most frequent task signatures in Terminal-3.6K and Office-2K, respectively, indicating limited repetition of a small template set.The reported concentration figures support diversity beyond repeatedly instantiating a few task templates.

5 EXPERIMENTS

NexForge improves terminal and office-agent performance across model and data scales, transfers across domains without changing its pipeline, and attributes gains to complementary controls for diversity, grounding, and directive–scenario coherence.

  • Terminal performance: 29.5 points: Qwen3.5-35B-A3B rises from 22.5%±0.9 to 52.0%±3.6 on Terminal-Bench 2.0 with Terminal-3.6K.Qwen3-32B also improves from 5.6%±0.9 to 32.3%±2.4, indicating gains across substantially different initial capabilities.
  • Domain transfer: 2,000 office tasks and 5,940 trajectories improve Qwen3.5-35B-A3B Base from 813 to 1338 GDPval Elo.The office corpus covers document processing, spreadsheet analysis, planning, communication, and evidence-based recommendations while retaining the unchanged pipeline.
  • Data volume scaling: 46 Elo: Office-22K reaches 1384 versus 1338 for Office-2K, showing consistent gains as office data scale increases.Elo values are contextual because they depend on the comparison pool.
  • Ablations and compilation controls: exp(H)≈6: Terminal-2K balances task-type diversity and frequency, whereas w/o profile collapses to one dominant type and w/o scenario expands coverage to exp(H)≈16 with an overly flat distribution.The results identify task-form control as preventing mode collapse and scenario grounding as shaping a realistic distribution; the full setting achieves the highest mean accuracy and longest trajectories.
  • Ablations and compilation controls: 81.0% versus 7.0%: DATC achieves a higher directive–scenario match rate than direct sampling without DATC.DATC retains only compatible options on average—4.4 of 24 task types, 5.2 of 8 deliverables, 2.7 of 4 source strategies, and 4.5 of 20 runtimes—leaving 2.0% of combinations compatible.

6 CONCLUSION AND FUTURE WORK … C.2 TRAJECTORY SCALE AND TOKEN MIX

NexForge scales requirement-driven agent-task synthesis across domains by compiling coherent directives, instantiating executable workspaces, and collecting diverse trajectories. The supplementary analyses detail this pipeline, its corpus configuration, and the differing tool, build, read, and document characteristics of terminal and office data.

  • 6 CONCLUSION AND FUTURE WORK: NexForge decouples task-form control and scenario grounding from predefined substrates, enabling transfer across capability domains and larger-scale synthesis through specification changes.Terminal-3.6K improves Qwen3-32B by 26.7 points and Qwen3.5-35B-A3B by 29.5 points.
  • APPENDIX: The supplementary material formalizes task compilation, traces an end-to-end synthesized task, and reports synthesis configuration plus trajectory-scale and token/operation statistics.It organizes the supporting analyses around Algorithm 1, a complete terminal-task case study, synthesis settings, and corpus composition.
  • A SCENARIO-CONDITIONED COMPILATION PROCEDURE: Algorithm 1 composes each directive from a requirement, demand profile, and scenario by sequentially filtering compatible task-form fields while preserving profile-guided corpus distributions.It selects task type, deliverable, source strategy, and runtime, then samples language and difficulty under batch constraints and records candidate sets with rationales.
  • B END-TO-END CASE STUDY: The end-to-end case study fixes the work contract before material gathering, then mines artifacts, builds a blueprint, and generates a validated executable package.The pMARS example specifies a cross-compilation task, retrieves source and Redcode materials, resolves CPU-only unprivileged constraints, and checks completeness, consistency, restrictions, and solution leakage.
  • B END-TO-END CASE STUDY: Requirement-driven synthesis produces a long-horizon cross-compilation toolchain task where a substrate-bound pMARS pipeline would more naturally produce implementation or test-repair work.The intended work contract determines the task’s structure and difficulty rather than the repository’s most convenient affordances.
  • C AGENT POST-TRAINING DATA: SYNTHESIS CONFIGURATION AND CORPUS STATISTICS: The corpus-statistics section reports reviewed task-form profiles, keyword pools, trajectory scale, and the token/operation composition of collected agent data.These statistics characterize both the candidate space used for synthesis and the resulting trajectories.
  • C.1 REVIEWED PROFILE AND KEYWORD POOLS: Terminal-2K uses 24 task types, 20 runtimes, 8 deliverables, 4 source strategies, and 3,678 keywords, while the office run uses 15, 12, 23, 4, and 773 respectively.The terminal profile covers a broader operational surface through more task types and runtime environments.
  • C.2 TRAJECTORY SCALE AND TOKEN MIX: Terminal trajectories are more tool- and build-intensive, whereas office trajectories are shorter and more read- and document-oriented.Table 7 distinguishes original rollouts from cleaned records and reports median tokens and mean deduplicated tool calls; Table 8 breaks down tokens and operations by message roles and tool classes.

C.3 CAPABILITY REQUIREMENT SPECIFICATIONS

NexForge uses two independently written, high-level natural-language capability requirements as pipeline inputs. The specifications target broad, real-world terminal-agent tasks and high-value professional work in real organizations.

  • Specification design: The two corpora are generated from independently written capability requirements that describe target capabilities in high-level natural language.Each specification serves as the pipeline input I in Section 4.1.
  • Terminal capability requirement: Terminal tasks span multi-domain technical work in real command-line environments, including software engineering, system administration, security, scientific computing, and data science.The stated scope also includes data processing and querying, machine learning, model training, mathematical problem solving, optimization, code comprehension, file operations, game or strategy solving, multimedia processing, and personal-assistant automation.
  • Office capability requirement: Office tasks target high-value professional work situated in real organizations across industries including real estate, government, manufacturing, healthcare, finance, retail, information and media, logistics, and operations support.The requirement covers digital knowledge work in high-output industries and large organizations.

D AGENT POST-TRAINING: FINE-TUNING CONFIGURATION · E DETAILED EVALUATION RESULTS

The paper documents reproducible full-parameter SFT configurations and reports controlled evaluation procedures for Terminal-Bench 2.0 and GDPval, alongside external Terminal-Bench references.

  • D AGENT POST-TRAINING: FINE-TUNING CONFIGURATION: Teacher rollouts use DeepSeek V4 Pro, retaining trajectories that survive cleaning, while all runs use 128 H100 GPUs across 16 nodes.Malformed and degenerate trajectories are removed; the infrastructure uses 8 NVIDIA H100 80 GB GPUs per node.
  • D AGENT POST-TRAINING: FINE-TUNING CONFIGURATION: All experimental runs use full-parameter SFT with shared hyperparameters from Table 10 and model-specific parallelism configurations from Table 9.The configuration is explicitly documented for reproducibility.
  • D AGENT POST-TRAINING: FINE-TUNING CONFIGURATION: Qwen3.5-35B-A3B uses TP=2, EP=8, CP=8, PP=1, whereas the Qwen3-32B comparison uses TP=2, CP=4, PP=8.The runs also use different qwen3 chat templates.
  • D AGENT POST-TRAINING: FINE-TUNING CONFIGURATION: 1.61B packed train tokens are used for Terminal-2K, 0.97B for Terminal-2K w/o profile, and 1.72B for Terminal-2K w/o scenario.These approximate values come from logged mean packed sequence length multiplied by logged packed quantities.
  • E DETAILED EVALUATION RESULTS: Terminal-Bench 2.0 controlled pass@1 results use the same NexAU scaffold, task-suite revision, tools, inference settings, and 89-task denominator.Unfinished, timeout, nonzero-exit, and verifier-error trials count as failures.
  • E DETAILED EVALUATION RESULTS: GDPval Elo intervals are fit from pairwise outcomes across all 220 tasks, anchoring GPT-5.1 at 1000 points under GDPval-AA v1 methodology.An LLM judge returns win, loss, or tie outcomes, and all rows use one in-house NexAU-based joint comparison pool.
  • E DETAILED EVALUATION RESULTS: External Terminal-Bench reference values are recorded from the public leaderboard and published terminal/agent-data studies for contextual comparison.The cited sources include Terminal-Bench Team, Fan et al., Pi et al., Peng et al., Wu et al., Cheng et al., and Raoof et al.

E.1 PER-TASK OUTCOMES AND PASS@K

NexForge improves reliability across individual Terminal-Bench tasks, increasing consistently solved tasks and lifting pass@k across multiple base models. These gains persist as the number of sampled attempts grows.

  • Per-task outcomes: 29 tasks are never solved after Terminal-3.6K training, down from 54, while tasks solved in all four runs rise from 7 to 29.Across 89 tasks and four independent runs, the shift indicates more robust task solving rather than occasional lucky successes.
  • Pass@k: Qwen3.5-35B-A3B reaches approximately 68% at pass@4 from 52.0% at pass@1, remaining roughly 30 points above its base throughout.NexForge improves the entire task-level pass@k curve.
  • Pass@k: Qwen3-32B rises from 32.3% at pass@1 to approximately 55% at pass@4, while its base reaches only about 14% at four attempts.The NexForge-trained model therefore improves across sampled-attempt counts.

F TERMINAL ABLATION DETAILS · G TASK-PACKAGE AUDITS FOR AGENT TRAINING DATA QUALITY

The terminal ablation isolates synthesis controls under fixed training conditions, while package audits evaluate static integrity, scenario-conditioned filtering, and realized task-form distributions. Removing profiles or scenarios changes distributional coverage, with w/o scenario producing a flatter distribution.

  • F TERMINAL ABLATION DETAILS: The ablation fixes the student, rollout budget, SFT recipe, and Terminal-Bench 2.0 target while varying synthesis controls.This isolates the effects of the synthesis-control variants.
  • F TERMINAL ABLATION DETAILS: Terminal-2K w/o profile retains the diverse scenario reservoir but removes selected task-form dimensions from directives received by downstream stages.Downstream stages receive the scenario without the removed task-form fields.
  • F TERMINAL ABLATION DETAILS: Terminal-2K w/o scenario retains sampled task-form fields but replaces the concrete scenario with a scenario-free construction.The supplied passage is truncated after describing this replacement.
  • F TERMINAL ABLATION DETAILS: w/o profile collapses toward software-engineering tasks and nearly eliminates some terminal task families, reflecting the model’s default scenario-to-task mapping.Table 15 is identified as showing the strongest distributional effect.
  • G TASK-PACKAGE AUDITS FOR AGENT TRAINING DATA QUALITY: The package audit covers static integrity, scenario-conditioned filtering, and realized distributions over task-form dimensions.Tables 16–18 and Figure 8 respectively address artifacts and sources, candidate-pool narrowing, and realized coverage or concentration.
  • G TASK-PACKAGE AUDITS FOR AGENT TRAINING DATA QUALITY: Table 16 checks whether each synthesized package contains expected artifacts and is grounded in real sources.This is the static synthesis audit.
  • G TASK-PACKAGE AUDITS FOR AGENT TRAINING DATA QUALITY: Table 17 measures how aggressively scenario-conditioned filtering narrows the candidate pool before sampling.Table 18 and Figure 8 instead report realized coverage and concentration for each task-form dimension.

G.1 TASK QUALITY AUDIT · H SCENARIO RESERVOIR AUDIT

The audits find that synthesized task packages are coherent, executable, and well calibrated, while the accepted scenario reservoir is diverse and free of exact or embedding near-duplicates. Branch-level analysis further shows differences in target conditioning and separation across generation strategies.

  • G.1 TASK QUALITY AUDIT: Most inspected task packages score at least 4 on all five quality dimensions, and none scores below 3.The rubric covers description clarity, material completeness, workspace executability, difficulty calibration, and absence of internal-artifact leakage.
  • G.1 TASK QUALITY AUDIT: The manual audit evaluates stratified packages from each corpus and ablation variant for clarity, completeness, executability, calibration, and artifact leakage.These are the five dimensions labeled D1 through D5.
  • G.1 TASK QUALITY AUDIT: The audit includes scenario-conditioned compatibility filtering, realized task-form and material distributions, and task-type and source-strategy distributions.These analyses are presented in Tables 17–18 and Figure 8 for Terminal-2K and Office-2K.
  • H SCENARIO RESERVOIR AUDIT: 5,600 accepted scenarios span four generation branches, with 0 exact text duplicates and 0 embedding near-duplicates above the cosine 0.85 threshold.The global maximum nearest-neighbor cosine is 0.837, and statistics are computed before scenario mining and task-form composition.
  • H SCENARIO RESERVOIR AUDIT: Self-instruct is the most target-conditioned branch, combining the highest target silhouette with the lowest cross-target nearest-neighbor rate.The comparison uses the same embedding model as the online novelty filter.
  • H SCENARIO RESERVOIR AUDIT: Keyword research and keyword self-instruct have similar local compactness, while keyword research separates the two target intents slightly more strongly.Knowledge graph expansion has the weakest target separation, consistent with exploring reusable entities.

I TASK-DESCRIPTION EMBEDDING · J SYNTHESIS PROMPT TEMPLATES

NexForge’s task descriptions form distinct Terminal and Office embedding clusters without collapsing within targets, while its synthesis prompts compile requirements into researched, executable tasks and teacher trajectories. The pipeline proceeds through profile planning, scenario mining, blueprint ideation, materialization, and NexAU-based rollout execution.

  • J SYNTHESIS PROMPT TEMPLATES: The four-stage synthesis pipeline uses diverse, mine, ideate, and gen prompts, followed by the NexAU runtime prompt for teacher rollouts.The implementation writes stage prompts in Chinese, with faithful English renderings provided.
  • J.1 PLANNING: TASK PROFILE (DIVERSE): J.1 Planning converts free-form specifications into a reviewed task profile whose candidate lists are comprehensive, discriminative, mutually exclusive, and sampling-weighted.The profile organizes task types, runtime environments, deliverables, information carriers, and material file types for later generation.
  • J.2 MINING: SCENARIO RESEARCH (MINE): J.2 Mining researches each directive’s scene, constraints, compatible task forms, environments, information carriers, and delivery boundaries to produce a blueprint-design research record.It progressively retrieves and curates usable resources, distinguishing solver-facing inputs from hidden evaluation materials, dependencies, and background.
  • J.3 IDEATION: TASK BLUEPRINT (IDEATE): J.3 Ideation combines the query, scene research, task profile, and directive into a high-difficulty executable ideation.json blueprint.Blueprint design extracts the real-world scene, role, objects, constraints, target state, and material forms rather than merely rewriting the guide.
  • J.4 GENERATION: TASK MATERIALIZATION (GEN): J.4 Materialization writes the solver-facing task package from the blueprint, prioritizing prepared real sources while aligning files, Docker configuration, paths, and validation.Generated tasks preserve the blueprint’s intent and delivery boundary, deepen one primary goal, and run CPU-only in ordinary unprivileged containers.
  • J.4 GENERATION: TASK MATERIALIZATION (GEN): Materialization enforces real-file packaging and excludes symlinks, virtual filesystems, snapshots, histories, caches, and oversized vendor or build outputs.The package workflow reads the prepared-source manifest, lands assets, writes the Dockerfile, checks consistency, and records validation results.
  • J.5 TEACHER ROLLOUT RUNTIME (DISTILLATION): J.5 Teacher rollouts use the NexAU runtime with per-run tool descriptions, optional sub-agent delegation, and continuation summaries for resuming unfinished work.The runtime prompt frames the agent’s objective as understanding requests, selecting tools, delegating when needed, and executing actions efficiently.
Loading 2607.14186v6…