Source-linked AI summary
Scaling Verifiable Environments for Long-horizon Work Agents
Jiazheng Zhang, Long Ma, Yunxian Yang, Zhiheng Xi, Zhikai Lei, Yajie Yang, Chenyang Liao, Enyu Zhou, Yang Nan, Yuchen Tian, Senjie Jin, Yibo Wang, Wei He, Boyang Liu, Jixuan Huang, Xin Guo, Zhezheng Hao, Xinbing Liang, Zhihao Zhang, Changzhi Zhou, Wiggin Zhou, Tao Gui, Qi Zhang, Xuanjing Huang, Clarenceai, Aiden Adams
TL;DR
Professional work requires agents to handle complex, authentic workspaces with trustworthy verification, but expert-curated environments are costly and synthesis can sacrifice realism or grounded evaluation. WORKFORGE derives tasks and verifiers from facts in real-world workspaces. Fine-tuning on its trajectories improves benchmark scores, and the paper reports benefits with increasing data volume and trajectory length.
Problem
Professional workflows require agents to navigate interdependent authentic artifacts, but expert-curated environments are costly and synthesized workspaces or ungrounded evaluation can limit realism and verification reliability.
Source: §1 · pp. 1–2Method
Results
28.1 points on GDPval and 16.3 points on APEX are gained by fine-tuning Qwen3.5-35B-A3B-Base on WORKFORGE trajectories.
Source: §1 · pp. 2–3Takeaways & Limitations
Controlled studies report benefits with increasing data volume and trajectory length.
Source: §1 · p. 2WORKFORGE depends on expert workflows and accessible realistic resources, and verifier reliability depends on complete, accurate workspace profiling.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Work agents operate over digital artifacts to execute professional knowledge-intensive work, requiring training environments that support long-horizon interaction and trustworthy verification. However, hand-crafted environments incur prohibitive engineering overhead that prevents environment scaling, whereas synthesis methods sacrifice workspace complexity, realism, or grounded verifiability. To bridge this gap, we introduce WorkForge, a scalable synthesis framework for constructing verifiable work-agent environments from real-world resources. Starting from expert workflows, WorkForge first identifies the resources, decisions, and deliverables required by each workflow. It then retrieves relevant real-world files and organizes them into a workspace. WorkForge inspects the workspace to extract concrete, checkable facts about its content. These factual anchors fix which task types the workspace can support and how their outcomes can be verified. Therefore, WorkForge derives each task's instructions, solution plan, and complementary programmatic and semantic verifiers directly from these factual anchors, keeping verification traceable to observable workspace evidence. Furthermore, we construct 16.7K verifiable environments across 40 professional domains, with workspaces collectively covering 60 file types. Post-training Qwen3.5-35B-A3B-Base improves GDPVal from 45.5 to 73.6 and APEX Score from 5.0 to 21.3, while enabling Qwen3.5-27B to achieve highly competitive performance and outperform strong competitors. Our analyses confirm the efficacy of the proposed method and reveal consistent scaling behaviors across both data volume and interaction horizons.
1 INTRODUCTION
Professional work requires agents to navigate authentic, interdependent workspace artifacts with trustworthy verification, but expert-curated environments are costly and automated synthesis can sacrifice realism or grounded evaluation. WORKFORGE builds environments from real-world resources and workspace facts; the resulting training data improves benchmark performance.
1 INTRODUCTION
- Professional workflows require agents to resolve file dependencies and make decisions in authentic artifacts, while training environments must provide long-horizon interaction and trustworthy supervision.Expert-curated environments offer high fidelity but impose prohibitive overhead; automated synthesis can produce toy workspaces with limited file diversity and cross-file dependencies, while ungrounded evaluation criteria risk unreliable judgments.Source: §1 · pp. 1–2
- WORKFORGE retrieves real-world resources, profiles workspace facts, and uses those anchors to derive task specifications and complementary programmatic and semantic verifiers.Autonomous solvers execute the tasks to produce long-horizon post-training trajectories.Source: §1 · p. 2, Figure 2 · p. 3
- 28.1 points on GDPval and 16.3 points on APEX are gained by fine-tuning Qwen3.5-35B-A3B-Base on WORKFORGE trajectories.WORKFORGE instantiates 16.7K environments across 40 professional domains.Source: §1 · pp. 2–3
- WORKFORGE instantiates 16.7K agentic environments across 40 professional domains, with complex filesystems and long-horizon solver trajectories.The paper reports gains across five diverse agent benchmarks.Source: §1 · p. 3
2 WORKFORGE
WORKFORGE turns expert workflows into realistic, verifiable work-agent environments by grounding task construction and interaction trajectories in instantiated workspace evidence.
2.1 WORKFLOW SPECIFICATION
- Expert Scene Blueprints encode workflow context, deliverables, required sources and dependencies, and domain-specific decisions, guiding resource acquisition and evidence-grounded task generation.Each blueprint connects expert intent to task generation using evidence from the instantiated workspace.Source: §2.1 · p. 3
2.2 ENVIRONMENT SYNTHESIS
- Environment synthesis maps each blueprint to resource-backed workspaces and profiles them with professional context, artifact characteristics, and observable facts.Resource pools contain repository files, and acquisition agents search for and assess relevant sources.Source: §2.2 · pp. 3–4
2.3 TASK GENERATION
- Task packages pair workspace-grounded instructions and hidden plans with atomic rubrics: programmatic checks inspect artifacts, while semantic judges score open-ended requirements.Semantic scores are normalized to a 0–10 scale.Source: §2.3 · p. 4
2.4 TRAJECTORY GENERATION
- Solver agents receive only the workspace and visible instruction, interact until submission or an execution limit, and produce deliverables that graders score and retain as trajectories.A trajectory records ordered messages and tool interactions alongside output artifacts and model configuration.Source: §2.4 · p. 5
3 EXPERIMENTS
The experiments compare WORKFORGE with baselines, test training-data and trajectory choices, and analyze scaling and agent behavior across benchmarks.
3.1 EXPERIMENTAL SETUP
- The evaluation covers five work-agent benchmarks, compares frontier and open-source baselines, and fine-tunes Qwen3.5-35B-A3B-Base and Qwen3.5-27B.Both models are trained for five epochs with a maximum sequence length of 256K tokens.Source: §3.1 · pp. 5–6
3.2 MAIN RESULTS
- 73.6 GDPval Score makes WORKFORGE-A3B the strongest prior specialized sub-100B baseline, 38.6 points above ClawGym-30A3; its ALE Pass@1 is 12.4% versus 2.9%.Table 2 reports performance across five benchmarks; the reported strict-pass comparison is on ALE.Source: §3.2 · p. 6, Table 2 · p. 6
- WORKFORGE-27B improves Qwen3.5-27B across all five benchmarks and reaches 82.4 GDPval, above Qwen3.5-397B-A17B’s 79.7.The comparison is despite nearly 15× fewer total parameters; the result narrows the gap to Claude Opus 4.8 and GPT-5.6 Sol.Source: §3.2 · p. 6
3.3 ABLATION STUDIES
- Training-harness alignment is associated with better performance: Claude Code-trained models lead on ALE under Claude Code evaluation, while OpenClaw-trained models lead on WildClaw under OpenClaw.The OpenClaw-trained model also leads on APEX, with its advantage narrowing to 2.7 points under the official ReAct Toolbelt.Source: §3.3 · p. 7, Figure 5 · p. 7
- The hybrid trajectory-selection strategy outperforms programmatic-only and LLM-only selection when all use 1K trajectories and the same fine-tuning configuration.The hybrid variant thresholds the equally weighted mean of the two verification scores at 0.7.Source: §3.3 · p. 7, Table 3 · p. 7
3.4 SCALING ANALYSIS
- 28.7 ALE and 29.9 WildClaw are the scores with 16K trajectories, exceeding the 1K setting by 13.0 and 8.4 points, respectively.Performance improves monotonically across the tested data volumes.Source: Figure 4 · p. 7, §3.4 · p. 8
- 17.5 ALE and 27.8 WildClaw are the scores for 41–60-turn training trajectories, exceeding 0–20-turn training by 4.2 and 5.0 points, respectively.On WildClaw, the longer-trajectory model averages 24.0 productive tool calls versus 11.7, and performs 38.3 distinct actions versus 17.0.Source: Figure 4 · p. 7, §3.4 · p. 8
3.5 BEHAVIOR ANALYSIS
- WORKFORGE-27B shifts toward exploration, increases targeted workspace evidence retrieval, and improves scores by 4.5–13 points across five GDPval-derived categories.The fine-tuned model visits 6.52 external web domains per task versus 5.3× fewer for the base model, and evidence retrieval rises from 5.9% to 16.2% of tool calls.Source: §3.5 · p. 8, Figure 6 · p. 8
4 RELATED WORK
Related work synthesizes agent environments and long-horizon training data; WORKFORGE distinguishes itself by grounding tasks and verification in realistic workspaces and their evidence.
Agent Environment Synthesis
- Prior environment synthesis builds tool interfaces and backends, incorporates real-service resources, or constructs tasks and workspaces; WORKFORGE profiles realistic workspaces before specifying tasks.The paper says preserving repository and dataset organization grounds tasks in heterogeneous materials, existing file relationships, and observable evidence.Source: §4 · p. 9
Long-Horizon Agent Data Synthesis
- Long-horizon data synthesis explores tool interactions, task composition, and task discovery, while other work studies filtering, long-context training, or artifact assessment.The cited approaches include Nemotron-Terminal and OpenThoughts-Agent (Raoof et al., 2026).Source: §4 · p. 9
- WORKFORGE generates sustained-exploration trajectories in content-rich workspaces, using shared workspace evidence for task requirements and complementary programmatic and semantic evaluation.Source: §4 · p. 9
5 CONCLUSION, LIMITATIONS AND FUTURE WORK
WORKFORGE derives tasks and verification from observable facts in real-world workspaces, producing 16.7K environments across 40 domains. The conclusion reports benchmark gains and scaling trends, while noting limits on resource coverage and verifier reliability.
5 CONCLUSION, LIMITATIONS AND FUTURE WORK
- WORKFORGE instantiated 16.7K environments across 40 domains, with Qwen3.5-35B-A3B-Base gaining 28.1 points on GDPval and 16.3 points on APEX.The pipeline derives task specifications, solution plans, and programmatic and semantic verifiers from workspace factual anchors.Source: §5 · p. 9
- Comparable grounding is limited in proprietary or sensitive domains because WORKFORGE depends on expert workflows and accessible realistic resources.Reliability also depends on profiling completeness and accuracy; current verifiers emphasize final deliverables and receive limited expert calibration.Source: §5 · p. 9
AI USE STATEMENT
- Generative AI assisted manuscript polishing and generated synthetic task packages and training trajectories, which the authors reviewed.The authors state that the study is for research purposes and does not pertain to malicious applications or several listed ethical and societal issues.Source: AI USE STATEMENT · p. 9, ETHICS STATEMENT · p. 9
A.1 DOMAIN TAXONOMY
- The taxonomy spans 40 domains, from office productivity and business operations to scientific, healthcare, security, and infrastructure-oriented tasks.Figure 7 shows all 40 domains and the ten most represented.Source: §A.1 · p. 14, Figure 7 · p. 14
A.2 DOMAIN DISTRIBUTION
- The four most represented domains account for 31.9% of tasks, the top ten for 56.9%, and the remaining 30 domains for 43.1%.The distribution combines recurring workplace domains with specialized applications.Source: §A.2 · p. 14, Table 4 · p. 15
B.1 PIPELINE PSEUDOCODE
- Algorithm 1 presents the end-to-end construction procedure using notation introduced in Section 2.Its listed inputs include expert-authored workflow specifications and a resource-acquisition interface; its output includes grounded task packages and a scored rollout corpus.Source: §B.1 · pp. 14–15
C.1 SUPERVISED FINE-TUNING SETUP
The setup fine-tunes Qwen3.5 models on WORKFORGE trajectories, with specified training, data-formatting, and inference configurations.
C.1 SUPERVISED FINE-TUNING SETUP
- Full-parameter supervised fine-tuning uses 16,665 WORKFORGE-16k trajectories generated by GLM 5.3 with Claude Code and OpenClaw harnesses.Qwen3.5-35B-A3B-Base and Qwen3.5-27B are trained for five epochs using the same optimization recipe.Source: §C.1 · pp. 15–16, Table 5 · p. 16
C.2 TRAINING INFRASTRUCTURE AND DATA FORMATTING
- Training preserves assistant-turn reasoning through cumulative conversation prefixes, applies per-message assistant loss masks, and packs variable-length examples without padding.Dynamic batching caps each GPU batch at 262,144 tokens; the infrastructure uses FSDP with Ulysses sequence parallelism of size four.Source: §C.2 · p. 16
C.3 INFERENCE CONFIGURATION
- Inference uses a shared 262,144-token context limit and 32,768-token response limit, extending shorter native context windows with YaRN.The passage identifies four Qwen3-based baselines using the shared YaRN settings; Table 6 reports the decoding configuration.Source: §C.3 · p. 16, Table 6 · p. 16, Table 7 · p. 17
C.4 BENCHMARK PROTOCOLS
The evaluation covers five benchmarks with benchmark-specific metrics, rollout protocols, harnesses, and execution budgets.
C.4 BENCHMARK PROTOCOLS
- The evaluation runs cover WildClawBench, Claw-Eval, GDPval, APEX-Agents, and Agents’ Last Exam.Table 8 summarizes benchmark metrics and sampling protocols.Source: §C.4 · p. 17, Table 8 · p. 17
- Pass@1 is the fraction of tasks passed in one rollout, while Claw-Eval’s Passˆ3 counts tasks passed in all three rollouts.Passˆ3 measures consistency across repeated runs; unlike Pass@3, it requires success in all three rather than at least one rollout.Source: §C.4 · p. 17
- WildClawBench and Claw-Eval use OpenClaw, GDPval and ALE use Claude Code, and APEX-Agents uses its official ReAct Toolbelt agent.Table 9 lists harnesses, per-task wall-clock limits, and interaction budgets.Source: §C.4 · p. 17, Table 9 · p. 17
C.5 GDPVAL EVALUATION SUBSET
The appendix specifies the GDPval evaluation subset and rubric-level analysis, then describes data-volume and trajectory-length ablations.
C.5 GDPVAL EVALUATION SUBSET
- GDPval evaluation uses the same 201-task subset of the official 220-task set, excluding 19 multimodal tasks, and reports official rubric scores rather than win-or-tie rates.Tasks are aligned by case ID corresponding to the official task ID.Source: §C.5 · p. 17, Table 10 · p. 18
C.6 GDPVAL RUBRIC-LEVEL ANALYSIS
- The rubric analysis groups GDPval criteria into five deliverable-quality categories and computes the pass rate within each category.A criterion is represented as satisfied or not satisfied; Table 11 reports category criteria counts and shares alongside pass rates.Source: §C.6 · pp. 17–18, Table 11 · p. 18
D ADDITIONAL ABLATION ANALYSES
- The training-data ablations use Qwen3.5-35B-A3B-Base with the supervised fine-tuning recipe specified for the main models.Table 5 gives the training configuration.Source: D ADDITIONAL ABLATION ANALYSES · p. 18
D.1 DATA VOLUME SCALING
- Data-volume scaling compares independently sampled sets of 1K, 3.6K, 8K, and 16K trajectories, with 16K using all 16,665 trajectories.Figure 4(a) reports results on ALE and WildClaw.Source: §D.1 · p. 18
D.2 TRAJECTORY-LENGTH SCALING
- Trajectory-length scaling samples 1,000 Claude Code trajectories from each of three disjoint turn-count intervals to evaluate ALE and WildClaw.Table 13 summarizes WildClaw tool use; its caption reports increased exploration intensity, breadth, and productive execution with longer-turn supervision.Source: §D.2 · pp. 18–19, Table 12 · p. 19, Table 13 · p. 19
D.3 VERIFICATION-FILTER ABLATION
The ablation creates three 1,000-trajectory training sets from the same completed rollouts, using semantic, programmatic, or hybrid filtering. Downstream results are reported on ALE and APEX.
D.3 VERIFICATION-FILTER ABLATION
- Three 1,000-trajectory sets are selected from the same pool of completed rollouts using different filtering rules.The selection rules are identified as Table 14; its caption alone does not specify further rule details.Source: §D.3 · p. 19, Table 14 · p. 19
- The LLM-only variant selects by Ssem, the programmatic-only variant by Sprog, and the hybrid variant by their equally weighted mean.Source: §D.3 · p. 19
- Each variant applies a 0.7 threshold, and downstream results are reported on ALE and APEX in Table 3.Source: §D.3 · p. 19
E DATA EXAMPLES
The example builds a cross-region factor-premium analysis workspace from a local mirror of Kenneth French data, then derives task checks from its contents. A rollout illustrates the data-handling decisions, common-window construction, and deliverables involved.
E.1 CROSS-REGION FACTOR-PREMIUM ANALYSIS
- WorkForge frames a Finance & Payments task around assessing factor premia and preparing an investment-committee briefing from an August 2025 vintage of the Kenneth French data library.The workspace source is a public Kaggle dataset that locally mirrors the library.Source: §E.1 · p. 19
Task Instruction
- The strategist is asked to compare factor performance across US, developed, and emerging markets, recommend allocations, address data caveats, and deliver a briefing, workbook, and tidy returns CSV.The instruction also asks the solver to explore workspace structure, coverage, and pitfalls.Source: Task Instruction · pp. 19–20
E.1.2 WORKSPACE AND SOURCE MATERIALS
- The workspace contains 297 CSV files across 16 directories and five branches, including factor series, portfolios, breakpoints, and industry returns.Its maximum nesting depth is six; momentum data also occur outside the US factor directory.Source: §E.1.2 · p. 20
- Files can combine descriptive preambles and multiple frequency blocks, while missing-value sentinels, differing CRSP/Bloomberg provenance, and Mom/WML naming require careful parsing and reconciliation.The US three-factor file stacks monthly and annual observations; regional comparisons also require identifying overlapping valid dates.Source: §E.1.2 · p. 20
E.1.3 WORKSPACE EVIDENCE AND VERIFICATION CRITERIA
- Workspace inspection yields 12 usable fact records, including the distinction between Developed data starting in 199007 and Emerging five-factor data starting in 198907.The verification criteria include separating annual from monthly returns, treating -99.99 and -999 as missing, and discussing source differences.Source: §E.1.3 · p. 21
- The task uses 23 criteria—eight programmatic and 15 semantic—to check deliverable validity, reported facts, analysis, data handling, and recommendation consistency.A programmatic check explicitly requires distinguishing the Emerging 198907 start from the Developed 199007 start.Source: §E.1.3 · p. 21
E.1.4 TRAJECTORY EXCERPTS
- The rollout identifies Emerging CMA, starting in 1992-07, as the binding constraint for a fair all-factor, all-region panel, producing a common window of 1992-07 to 2025-08 (398 months).The solver also checks file ends, schemas, provenance, date coverage, and factor availability; the analytics script supplies tables for workbook and slide builders.Source: §E.1.4 · pp. 21–22