GraphForge: Training Working Agents with Graph-Anchored Workspace Synthesis

1 INTRODUCTION

• Existing pipelines either generate files with models or build tasks on real files without task-specific rubrics, leaving result quality difficult to check systematically.Working-agent training needs tasks built on many real files with verifiable results.Source: §1 · p. 1
• GraphForge uses occupation-grounded seeds to guide task direction, then derives task statements and file-anchored rubrics from an evidence graph over real workspaces.An initial rollout tests executability, and a revision agent repairs the task and rubrics against the original files before trajectories are collected.Source: §1 · p. 2, Figure 1 · p. 2
• 1445.7 (+65.7) on GDPVal, 63.7 (+7.7) on Workspace-Bench-Lite, and 24.0 (+13.7) on SpreadsheetBench II follow fine-tuning Qwen3.6-27B on 2,169 GraphForge trajectories.Rejection fine-tuning on the SFT model’s own rollouts yields further improvements on all three benchmarks.Source: §1 · p. 2
• The paper presents GraphForge as grounding both task requirements and verification in real files, and releases the synthesized data and trained models.The contribution summary also reports gains for two trained base models and rubric-guided selection signal beyond training on the model’s own rollouts.Source: §1 · p. 2

2 RELATED WORK

• Task-synthesis research spans general tool use, computer use, software engineering, and terminal operation, where outcomes can be verified programmatically.The cited examples include Dong et al. (2026), Xie et al. (2026), Yang et al. (2025), and Fan et al. (2026).Source: §2 · p. 2
• NexForge releases trained models but not its synthesized tasks or trajectories.Source: §2 · p. 3
• GraphForge addresses EnvCraft and NexForge’s shared gap by compiling evidence graphs over crawled real files into anchored rubrics for trajectory selection and evaluation.EnvCraft generates files and checks workspace state with scripts; NexForge uses real files but lacks task-specific rubrics or verifiers.Source: §2 · p. 3
• The paper evaluates on GDPVal, Workspace-Bench, and SpreadsheetBench II, which cover occupational work tasks, cross-file reasoning, and spreadsheet workflows, respectively.The cited benchmark descriptions are GDPVal (Patwardhan et al., 2025), Workspace-Bench (Tang et al., 2026), and SpreadsheetBench II (Zhu et al., 2026).Source: §2 · p. 3

3.1 O*NET-GROUNDED TASK-FORM SEEDS

• Seeds use O*NET occupations and digitally executable work tasks, then coverage-based selection balances occupations, task types, execution patterns, and input families.After filtering, 246 occupations across 16 sectors and 43 sub-sectors remain, covering 891 task types and 3,419 occupation-task-type pairs.Source: §3.1 · pp. 3–4

3.2 REAL-FILE WORKSPACE CONSTRUCTION

• Each case-neutral seed is instantiated by retrieving a coherent set of real files, which are parsed and filtered to form its workspace.Files may serve core, supporting, confuser, or ambient roles; those roles are hidden from the working agent.Source: §3.2 · p. 4

3.3 EVIDENCE GRAPHS AND VERIFIABLE RUBRICS

• An evidence graph links source files and cross-file dependencies, then grounds task requirements and rubric criteria in the evidence needed to verify them.Criteria specify deliverables, requirements, expected values or computations, weights, evidence anchors, and verification procedures; the judge sees anchors, while the working agent does not.Source: §3.3 · p. 4

3.4 EXECUTION-CONDITIONED ONE-STEP REVISION

• A teacher rollout tests each initial specification, and a revision agent checks executability, evidence support, criterion anchors, and artifact-verification instructions against the original files.Only components needing changes are returned; the initial execution can be reused when only rubric bindings change.Source: §3.4 · pp. 4–5

3.5 ARTIFACT-LEVEL ADMISSION AND TRAJECTORY CLEANING

• Trajectories are admitted after deliverables pass deterministic checks and an evidence-anchored judge scores criteria; trajectories with degenerate tool use are discarded.Checks include filenames, readable formats, spreadsheet sheets and formulas, and task-specific structural constraints.Source: §3.5 · p. 5

4.2 TRAINING AND EVALUATION SETUP

• The study trains two Qwen3.6 variants on admitted teacher trajectories, then evaluates pass@1 performance on three working-agent benchmarks.RFT samples four rollouts per query from the SFT model, retains high-scoring valid trajectories, and compares arms under matched query pools and optimization budgets.Source: §4 · p. 5, §4.2 · p. 6

4 MAIN RESULTS

• The corpus comprises admitted teacher trajectories, and the benchmark comparison reports GDPVal Elo alongside workspace micro scores and spreadsheet execution accuracy.For GDPVal, each model–scaffold pair is a separate node on an Elo scale anchored at GLM-5.3 (OpenHands) = 1667.Source: §4 · p. 5, Table 2 · p. 7, §4.2 · p. 7

4.1 TRAINING DATA

• The 2,169-trajectory SFT corpus spans 466 O*NET task types, 15 occupational sectors, and all 16 execution patterns.It realizes 164 of 256 sector–pattern combinations; data analysis and reporting account for 14.2%, research and source synthesis for 13.2%, and PDF appears in 96.7% of trajectories.Source: §4.1 · p. 5, Figure 2 · p. 6
• 50.0 assistant steps and 162.0k tokens are the average trajectory lengths in the 2,169-example corpus.The medians are 48 steps and 158.0k tokens; 28 sequences (1.3%) reach the 262,144-token training ceiling.Source: Figure 3 · p. 6, §4.1 · p. 6

4.3 OVERALL COMPARISON

• The 27B SFT model reaches 1445.7 Elo on GDPVal with OpenHands and 1427.4 with Codex, improving over its base model by 65.7 and 63.4 points, respectively.On Workspace-Bench-Lite, SFT improves the two base models by up to 6.6 and 7.7 points; on SpreadsheetBench II, gains reach 16.5 and 13.7 points.Source: Table 2 · p. 7, §4.3 · p. 7
• SFT improves over the base model under every evaluated scaffold, although training trajectories were rolled out with Codex.Evaluation covers OpenHands and Codex on GDPVal and Claude Code and Codex on the other two benchmarks.Source: §4.3 · p. 7

4.4 RUBRIC-GUIDED REJECTION FINE-TUNING

• Anchored RFT leads on Workspace-Bench-Lite and SpreadsheetBench II, but GDPVal does not statistically resolve the two judge variants.On GDPVal, anchored RFT changes are +7.2 and +11.0 Elo, while unanchored selection has higher point estimates (+34.1 and +25.3); none of these differences is statistically resolved at this scale.Source: Table 3 · p. 8, §4.4 · p. 8

4.5 ABLATION STUDIES

• Only 13 of 44 GDPVal occupations appear in the training taxonomy, yet SFT win rates are no lower on uncovered than covered tasks.The win rate is 0.739 on 155 uncovered tasks and 0.692 on 65 covered tasks; no identical training and GDPVal files or substantive shared text were found.Source: §4.5 · pp. 8–9, Table 4 · p. 9

5 CONCLUSION

• 1445.7 (+65.7) on GDPVal under OpenHands, 63.7 (+7.7) on Workspace-Bench-Lite, and 24.0 (+13.7) on SpreadsheetBench II under Claude Code follow Qwen3.6-27B training on 2,169 trajectories.The gains hold across OpenHands, Codex, and Claude Code scaffolds.Source: §5 · p. 9
• Occupational seeds fix task direction, while evidence graphs over real workspaces supply tasks and verification rubrics anchored to source files.The graph-backed requirements and criteria are tied to the files needed to verify them.Source: §5 · p. 9
• The same data also improves Qwen3.6-35B-A3B, suggesting GraphForge trajectories generalize across base models.Source: §5 · p. 9
• Rubric-guided rejection fine-tuning yields additional gains over SFT and supports the value of evidence-anchored selection.Source: §5 · p. 9

6 DISCUSSION OF LIMITATIONS

• The corpus contains 2,169 trajectories, and the study does not test how GraphForge’s benefits scale with larger data budgets.Source: §6 · p. 9
• Both the agent judge and synthesis pipeline use GLM-5.2; exploring whether stronger frontier models improve data quality and judging reliability remains future work.Source: §6 · p. 9
• Experiments cover two base models from one family, leaving transfer to other model families unstudied.Source: §6 · p. 9
• Planned work includes more task families and file types, plus study of evidence-anchored verification with longer-horizon agent scaffolds.Source: §6 · p. 10
• Generative AI assisted with English polishing and coding tasks such as debugging, but not method design, experiments, or scientific claims; authors reviewed the assisted content.Source: AI USE STATEMENT · p. 11
• Training workspaces use publicly available documents for research, with source links released instead of file contents; no additional personally identifiable information is collected, curated, or inferred.The authors report no foreseeable harmful applications and no conflicts of interest.Source: ETHICS STATEMENT · p. 11
• The paper reports method, training, and hyperparameter details, releases data and checkpoints, and states that experiments use fixed random seeds with hardware and software setup reported.Source: REPRODUCIBILITY STATEMENT · p. 11

A TRAINING CONFIGURATIONS

• The 27B and 35B SFT models use the same corpus and optimization recipe.Source: A TRAINING CONFIGURATIONS · p. 14
• The anchored, unanchored, and random RFT arms share 462 query IDs, candidate pools, behavior filtering, and optimization budget.Source: Table 5 · p. 14
• RFT arms use identical query IDs and hyperparameters, differing only in how one trajectory is selected from each four-candidate pool; sequences exceeding 262,144 tokens are excluded rather than truncated.Training preserves complete assistant reasoning and tool-interaction histories.Source: A TRAINING CONFIGURATIONS · p. 14

B CONTAMINATION AUDIT

• No SHA-256 hash is shared between the 2,169 training sequences and files referenced by the 220 GDPVal tasks.The training corpus contains 49,750 file instances and 39,201 unique hashes; GDPVal references 261 file instances and 260 unique hashes.Source: B CONTAMINATION AUDIT · p. 14
• Among the top 20 manually reviewed text-similarity pairs, none shares task requirements, entities, business facts, or deliverable content.Five pairs share only generic numeric sequences and fifteen only PowerPoint master placeholder text.Source: B CONTAMINATION AUDIT · p. 14
• Thirteen of 44 GDPVal occupations also occur in the training taxonomy, covering 65 tasks; the other 155 tasks are from uncovered occupations.The overlap follows the shared O*NET taxonomy, not shared files or tasks.Source: B CONTAMINATION AUDIT · p. 14
• The SFT model wins at least as often on uncovered tasks as covered ones; the win-rate difference is 0.047 with a task-bootstrap 95% confidence interval of [−0.079, 0.175].Together with the file- and text-level audits, the authors find no sign that improvement relies on proximity to benchmark content.Source: B CONTAMINATION AUDIT · p. 14

• Table 6 reports 95% bootstrap confidence intervals for the GDPVal Elo scores in Table 2.Source: C GDPVAL ELO CONFIDENCE INTERVALS · p. 15
• The main-table pool excludes RFT arms but retains Claude Opus 4.8 (OpenHands) as an unreported bridge node to keep the comparison graph connected.Source: C GDPVAL ELO CONFIDENCE INTERVALS · p. 15
• Intervals come from 10,000 resamples of edge-level W/T/L outcomes and refitting the full Bradley–Terry graph, with GLM-5.3 (OpenHands) fixed at 1667.Source: C GDPVAL ELO CONFIDENCE INTERVALS · p. 15

• All RFT difference intervals include zero.Source: D RFT ELO CONFIDENCE INTERVALS · p. 15, Table 7 · p. 15
• Each scaffold compares three RFT arms directly with SFT, using the frozen main-table SFT point estimate as a fixed reporting anchor.Source: D RFT ELO CONFIDENCE INTERVALS · p. 15
• With ties counted as half wins, each edge's Bradley–Terry score is 400 log10(p/(1−p)) relative to SFT.Source: D RFT ELO CONFIDENCE INTERVALS · p. 15
• Intervals jointly resample task UUIDs across the three edges for 10,000 draws, reflecting task-sampling uncertainty conditional on the fixed SFT anchor.Source: D RFT ELO CONFIDENCE INTERVALS · p. 15
• Table 7 reports conditional 95% bootstrap confidence intervals for RFT direct-comparison Elo scores.Source: Table 7 · p. 15, D RFT ELO CONFIDENCE INTERVALS · p. 15