PaperlayerView exact revision on arXiv

Source-linked AI summary

WorkFactory: Turning Professional Intents into Executable Work for Scalable Agent Training

Ziyang Zeng, Tingyang Chen, Jia Deng, Qiyao Wang, Shuo Tang, Yike Yang, Tao Ren, Wayne Xin Zhao, Feng Chang, Jujie He, Chuan Hao, Ran Tao, Bryan Dai

arXiv:2610.37161v1cs.AI

TL;DR

Professional-agent supervision needs diverse tasks with sufficient, consistent information while leaving the intended work for teachers to complete. WorkFactory plans information dependencies, constructs and probes task environments, and uses collected trajectories to train WorkPilot models that improve across four professional-work benchmarks.

  • Problem

    Professional-work tasks can lack consistent premises distributed across artifacts, while supplying derived results would remove work teachers are meant to perform.

    Source: §1 · p. 1

  • Method

    WorkFactory plans backward from intended outputs with an information support graph, probes and repairs support failures, and collects teacher trajectories from retained tasks.

    Source: §1 · p. 2, §3.1 · p. 3, §3.4 · p. 6

  • Results

    All three WorkPilot models improve over their matched bases across four professional-work benchmarks, with scaling analyses showing model-dependent diminishing returns.

    Source: §1 · pp. 2–3, §5 · p. 11

  • Takeaways & Limitations

    The results support coherent task construction as a way to scale working-agent supervision with expert trajectories.

    Source: §1 · p. 3, §5 · p. 11

    Training and evaluation use a text-only task-solving interface, and the training study focuses on supervised fine-tuning rather than reinforcement learning.

    Source: Limitations · p. 11

Abstract

from arXiv · show

Scaling supervised training for working agents requires diverse professional tasks that yield reliable expert demonstrations. However, missing or inconsistent task information can obstruct teacher execution and weaken the resulting supervision. We propose WorkFactory, a scalable framework for constructing coherent and executable professional tasks. Starting from professional intents, information-level planning builds an information support graph to guide task instantiation, explicitly representing the dependencies and locations of information required for the intended outputs. Probe executions then identify support failures and guide repairs while preserving the intended work. We construct 112,686 tasks spanning 612 occupations and use the collected teacher trajectories to fine-tune Qwen models at 4B, 27B, and 35B-A3B scales, yielding the WorkPilot family. All three models improve over their respective base models across four professional-work benchmarks, with WorkPilot-35B-A3B improving from 879 to 1565 Elo on GDPval-AA v2.1. Scaling analyses reveal model-dependent diminishing returns, with execution length first increasing and then decreasing. Controlled component analyses show that planning and repair reduce probe-diagnosed task environment defects and improve downstream performance. The data and models are available at https://github.com/IQuestLab/WorkFactory.

1. Introduction

WorkFactory targets a bottleneck in professional-agent supervision: tasks must expose sufficient, consistent premises while leaving the intended work for teachers to perform. It builds tasks around information dependencies and repairs support failures, and the resulting WorkPilot models improve across four professional-work benchmarks.

Motivation

  • Professional tasks can require premises distributed across artifacts and intermediate calculations, yet supplying derived results would remove substantive work from the teacher.Missing conditions can leave teachers to make unsupported assumptions or incomplete work, weakening training supervision.Source: §1 · pp. 1–2

Results

  • All three WorkPilot models improve over their matched bases across four professional-work benchmarks; WorkPilot-27B reaches locally estimated 1612 Elo on GDPval-AA v2.1.Figure 1 illustrates WorkPilot-35B-A3B’s gains over its Qwen3.6-35B-A3B base model.Source: Figure 1 · p. 2, §1 · p. 2

Approach

  • WorkFactory plans backward from intended outputs with an information support graph, then probes and repairs support failures without changing the intended work.Only environments that pass the process are assigned to an independent teacher for trajectory collection.Source: §1 · p. 2

Study scope

  • Scaling working-agent data up to 110K tasks reveals model-dependent diminishing performance returns, an initial rise then fall in execution length, and changing tool-use reliability; planning and execution-based repair reduce probe-diagnosed defects and improve downstream performance.Source: §1 · p. 3

2. Related Work

Prior work synthesizes agent tasks through software repositories, compositional expansion, environment feedback, and stateful interaction. Professional-task frameworks differ in how they construct workspaces and use evidence or execution feedback to revise tasks.

Agent Task and Environment Synthesis

  • Agent-task synthesis spans executable software-engineering settings, longer-horizon multi-tool tasks, environment exploration, and stateful interaction environments.These approaches motivate coherent workspaces combining heterogeneous information and domain-specific deliverables for working agents.Source: §2 · p. 3

Professional Task Construction for Working Agents

  • NexForge and Synthetic Computers use capability requirements or artifact dependencies to guide professional task construction.The passage distinguishes these approaches from WorkFactory’s output-driven coordination of information dependencies and locations.Source: §2 · p. 3
  • GraphForge revises evidence-graph specifications with execution feedback, while WorkGenesis uses rubric-based failure attribution; WorkFactory probes solver-visible environments and constrains repairs to preserve intended work.GraphForge builds tasks and rubrics from evidence over assembled real-file workspaces; WorkGenesis revises requests, materials, and rubrics.Source: §2 · p. 3

3. WorkFactory

WorkFactory constructs professional tasks from diverse intents, plans information dependencies and locations backward from intended outputs, and probes realized environments before teacher collection. Retained tasks form a broad corpus for trajectory-based supervision.

3.1. Overview

  • The workflow builds a diverse intent library, constructs environments from information-support plans, probes and repairs materialized tasks, then assigns passing tasks to independent teachers.Figure 2 illustrates the process with a sourcing example.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2. Multi-source Professional Intent Synthesis

  • WorkFactory derives professional intents from occupational hierarchies, documented skills, and deliverables, then reviews them for coherence, feasibility, appropriateness, and workplace plausibility.Source: §3.2 · p. 4

3.3. Graph-guided Environment Construction

  • The information support graph links information units through support relations and binds them to carriers, so required premises are available while calculations and judgments remain for the solver.The graph guides an artifact-level construction plan that organizes the request, artifact specifications, and construction dependencies.Source: §3.3 · pp. 4–5

3.4. Execution-based Environment Repair

  • Probe solvers use only visible requests and resources to identify blockers; repairs restore missing support while preserving task structure and intended work, and unresolved tasks are discarded.Repairs undergo construction checks and a work-preservation review, followed by a fresh probe; retained tasks proceed to trajectory collection.Source: §3.4 · p. 6

3.5. Task Corpus and Training Data

  • 112,686 tasks span 612 fine-grained occupations and 23 groups; Figures 3–4 show selected occupations and the distribution of target deliverable file types.The corpus was checked for evaluation-task overlap and byte-identical files, with none identified.Source: Figure 3 · p. 5, Figure 4 · p. 6, §3.5 · p. 7

4. Experiments

WorkFactory supervision improves all three tested WorkPilot models across four professional-work benchmarks. The experiments also examine data scaling, task-environment repairs, teacher choice, and estimated inference costs.

4.1. Experimental Setup

  • The study compares WorkPilot models with matched base models and representative frontier and open-weight systems across four professional-work benchmarks.Locally evaluated model–task pairs use three independent rollouts; GDPval follows the GDPval-AA protocol, with official v2.1 Elo ratings reported where available.Source: §4.1 · p. 7

4.2. Effectiveness of WorkFactory Data

  • All three WorkPilot models improve over matched bases on four benchmarks; WorkPilot-35B-A3B rises from 879 to 1565 GDPval-AA Elo.The reported WorkPilot-35B-A3B scores also rise from 28.91 to 62.35 on JobBench, from 47.12 to 56.53 on Workspace-Bench, and from 7.14 to 21.18 on OmegaUse-OfficeVal.Source: §4.2 · pp. 7–8, Table 2 · p. 8

4.3. Scaling Properties

  • 447 Elo points come from the first 5K tasks, versus 13 from 75K to 110K, as Qwen3.6-35B-A3B scales on GDPval-AA.By 30K tasks, the 35B-A3B and 27B models have captured 88% and 94% of their respective gains through 110K; the 4B model gains 546 Elo by 30K.Source: §4.3 · pp. 8–9, Figure 5 · p. 9
  • Mean execution length for Qwen3.6-35B-A3B rises from 24.87 turns at baseline to 162.88 at 2K tasks, then falls to 85.90 at 110K while Elo rises.The 27B and 4B models also show an increase from their bases to 30K followed by a decrease at 110K; the 35B-A3B share reaching the 250-turn limit falls from 21.9% at 2K to 2.7% at 110K.Source: §4.3 · p. 9, Figure 5 · p. 9

4.4. Component Analysis

  • Information-level planning reduces hard-defect incidence in six of eight categories, while execution-based repair reduces incidence in all eight relative to planning alone.With repair, cross-file inconsistency falls from 25.74% to 4.84%, execution-feasibility defects from 6.64% to 0.48%, and leakage or shortcuts from 3.36% to 0.84%.Source: §4.4 · pp. 9–10, Table 3 · p. 10
  • Information-level planning followed by execution-based repair raises GDPval-AA Elo from 1245 to 1326 across the three synthesis conditions.Planning alone raises Elo to 1293; adding repair raises it to 1326, an 81-point cumulative gain over direct artifact-level planning.Source: §4.4 · p. 10, Table 3 · p. 10

4.5. Effect of Teacher Choice

  • GLM-5.3 teacher trajectories yield 1483 Elo, versus 1248 for DeepSeek-V4-Flash-0731 trajectories, on the same task set; both exceed the 879-Elo base.The comparison reuses the same tasks and fine-tunes the same base model under the shared configuration.Source: §4.5 · p. 10

4.6. Cost–Performance Trade-off

  • All three WorkPilot models lie on the non-dominated cost–performance frontier among the models shown.Estimated inference costs per GDPval-AA task are approximately $0.012 for WorkPilot-4B, $0.031 for WorkPilot-35B-A3B, and $0.396 for WorkPilot-27B.Source: §4.6 · p. 11, Figure 6 · p. 11

5. Conclusion

WorkFactory constructs executable professional tasks through information-level planning and execution-based repair. Its teacher trajectories train WorkPilot models that improve over their bases across four professional-work benchmarks.

5. Conclusion

  • WorkFactory uses information-level planning and execution-based repair to construct tasks, and its 112,686-task teacher trajectories train three WorkPilot models that improve over their bases across four benchmarks.The conclusion also reports model-dependent diminishing returns, changing execution length and tool-use reliability, and component-study links between fewer probe-diagnosed defects and higher downstream utility.Source: §5 · p. 11

Limitations

The study trains and evaluates the Qwen backbones through a text-only task-solving interface because the GLM-5.3 teacher accepts only text input.

Limitations

  • Because GLM-5.3 accepts only text input, the study does not examine learning from visual evidence directly.The authors identify multimodal professional-task trajectory collection and student training as future work.Source: Limitations · p. 11

Ethical Considerations

WorkFactory targets professional work and cautions that its outputs need human review in consequential settings. It synthesizes work intents from occupational roles, documented skills, and real deliverables, then reviews candidate intents for feasibility and plausibility.

Ethical Considerations

  • Outputs may be incorrect, biased, or misused in consequential settings, so they require appropriate human review rather than replacing qualified professionals.Dataset and model release should also respect privacy, copyright, and licensing requirements.Source: Ethical Considerations · p. 12

A. Professional Intent Synthesis

  • Occupational hierarchies seed role-specific situations and objectives grounded in responsibilities, business settings, and motivating events.Source: A. Professional Intent Synthesis · p. 17
  • Documented skills are analyzed for purpose, applicability, prerequisites, and operations; compatible skills can be composed into coherent multi-step workflows.For example, quotation extraction can provide inputs for cash-flow modeling and then an NPV-based sourcing recommendation.Source: A. Professional Intent Synthesis · p. 17
  • Real professional deliverables cue intents to produce, update, review, or compare artifacts based on their likely use, audience, content, and production process.Source: A. Professional Intent Synthesis · p. 17
  • An LLM reviewer excludes candidate intents judged incoherent, infeasible, professionally inappropriate, or implausible as workplace activities.Source: A. Professional Intent Synthesis · p. 17

B. Information-level Planning and Task Instantiation

WorkFactory plans tasks around the information needed for intended outputs, where that information is available, and how it is supported. It materializes artifacts, probes the resulting environment, and repairs blocker-level failures while checking that the intended work remains intact.

B. Information-level Planning and Task Instantiation

  • The task representation distinguishes planned outputs and construction requirements from information already available to the solver; all completion-critical information must be accessible or externally obtainable.Source: B. Information-level Planning and Task… · p. 17

B.1. Information Nodes and Relations

  • Information nodes record content, entity and scope, provision status, and carrier bindings; relations specify how source nodes support targets through dependencies such as calculation, evidence, or constraints.Source: §B.1 · p. 17

B.2. Carriers and Access

  • Carrier and location bindings identify where information is planned or found and its access conditions; repeated information is linked by shared identity and scope to support consistency across files.Source: §B.2 · pp. 17–18

B.3. Artifact Construction and Checks

  • An artifact plan specifies the request, deliverables, artifact roles and formats, provenance, materialization routes, and acyclic construction dependencies; workers reuse, adapt, or synthesize materials.Programmatic file checks and LLM review assess the materialized environment, with up to three retries before failed tasks are discarded.Source: §B.3 · p. 18

C.1. Probe Reporting and Defect Taxonomy

  • Probe reports ground diagnoses in inspected sources and tool observations, separate environmental defects from solver mistakes and open choices, and label severity; only blocker findings trigger repair.The retrospective defect categories are distinct from severity labels, and blocker and correctness findings count as hard defects in the incidence analysis.Source: §C.1 · pp. 18–19

C.2. Work-Preservation and Repair Constraints

  • Repair review compares the revised task with its original intent and plan, preserving its core objective, essential operations, and substantive deliverable requirements while allowing minor scope changes needed for completion.Revisions undergo construction and work-preservation checks, are re-probed in a clean sandbox, and unresolved blockers after up to three rounds lead to discard.Source: §C.2 · p. 19
  • The initial plan is a reference for intended work, not a requirement to retain every instance detail; information relations specify support and constraints without prescribing a unique judgment or execution strategy.Source: §C.2 · p. 20

C.3. Repair Dynamics on the 5K Analysis Set

The repair analysis reports how often tasks receive refinement and how execution behavior changes afterward. The accompanying example traces a missing-information repair, while corpus and construction-cost statistics characterize the resulting task collection and its production.

C.3. Repair Dynamics on the 5K Analysis Set

  • 20.24% of tasks undergo one repair, 7.70% undergo two, and 2.48% reach three rounds; 24 tasks (0.48% of the full set) remain blocked and are excluded from training.After repair, mean reasoning tokens decrease by 1,738.58, tool-call argument tokens increase by 5,265.34, and body-text tokens decrease by 88.70, for a net increase of approximately 3,438 assistant-output tokens.Source: §C.3 · p. 20

C.4. End-to-End Supplier-Selection Example

  • In the supplier-selection example, a probe identifies missing freight terms that prevent comparable lifecycle-cost calculations; repair adds a USD 0.15-per-carton charge without supplying spending calculations, NPVs, or the recommendation.Source: §C.4 · p. 21

D.1. Corpus Statistics

  • 112,686 tasks span 612 fine-grained occupations, average 4.15 provided reference files per task, and have teacher trajectories averaging 57.4 assistant turns.Source: §D.1 · p. 21

D.2. Construction Cost

  • 11.656M model tokens are used per attempted task on average (11.448M input; 0.208M output), with information-level planning the largest stage at 4.737M tokens per task.Source: Table 8 · p. 22, §D.2 · p. 22

D.3. Trajectory Generation and Training

The paper collects teacher trajectories under specified execution settings, filters and selectively judges rollouts, then converts retained trajectories into supervised training sequences. The sequence objective masks failed tool-call spans while retaining their interaction context.

D.3. Trajectory Generation and Training

  • DeepSeek-V4-Flash-0731 agents perform intent synthesis, planning, construction, and repair, while GLM-5.3 serves as the independent teacher for trajectory collection.Source: §D.3 · p. 22
  • Probes and teachers start from clean sandboxes with solver-visible inputs; probes have a 1M-token context and teachers a 256K-token context, with teacher tasks split across Stirrup and Claude Code harnesses.Source: §D.3 · p. 22
  • Programmatic validity checks remove rollouts with repeated reasoning or tool calls, missing deliverables, or deliverables below a minimum content threshold; the checks do not judge task-specific correctness.Source: §D.3 · p. 22
  • The longest 5% of initial rollouts receive pairwise deliverable rejection sampling against a fixed baseline, with up to four teacher attempts; tasks without a teacher win are excluded.Tasks outside that selected subset retain their initial episodes after lightweight filtering.Source: §D.3 · pp. 22–23
  • Selected rollouts become one or more sequences, with sequences over 262,144 tokens removed; all three WorkPilot models are trained on selected GLM-5.3 trajectories using the same configuration.Only failed tool-call token spans are removed from supervision, while the calls and results remain in history for subsequent recovery.Source: §D.3 · pp. 22–23

D.4. Teacher-to-Student Behavior Analysis

The comparison uses matched WorkFactory tasks to examine teacher and student tool behavior under GLM-5.3 and DeepSeek-V4-Flash-0731 supervision. Students trained on either teacher substantially outperform the 879-Elo base model, while GLM-trained trajectories use more turns and tool calls.

D.4. Teacher-to-Student Behavior Analysis

  • The GLM-5.3 condition reuses the Qwen3.6-35B-A3B checkpoint trained on 30K tasks; the DeepSeek condition regenerates those task trajectories and trains the same base model.Both use the shared supervised fine-tuning configuration.Source: §D.4 · p. 24
  • Students trained on either teacher substantially outperform the 879-Elo base model, so WorkFactory gains are not exclusive to GLM-5.3 supervision.On matched tasks, GLM-5.3 uses more assistant turns and tool calls than DeepSeek-V4-Flash-0731 despite comparable reasoning-token counts; the GLM-trained student also uses more turns and calls.Source: §D.4 · p. 24
  • Figure 8 compares task-level mean assistant-turn distributions, tool-call composition, and task-macro tool-failure rates for teachers and students.Its top row shows teacher trajectories on the matched 30K-task subset; its bottom row shows student rollouts on the same 220 GDPval tasks.Source: Figure 8 · p. 25

E. Evaluation Details

Evaluation details specify text-only model inputs, a pairwise artifact-based judge, and a two-anchor Elo estimator. They also describe how GDPval-AA inference costs are estimated and where those estimates are bounded.

E. Evaluation Details

  • Local evaluations use three rollouts per model–task pair; WorkPilot and matched Qwen bases are evaluated text-only, though judges may inspect rendered documents and native media.Ratings cover 220 tasks across nine sectors and 44 occupations, judged pairwise against the original task requirements.Source: E. Evaluation Details · p. 25, §E.1 · p. 26

E.1. Two-Anchor Elo Estimation

  • Targets are fitted against fixed GLM-5.3 and DeepSeek-V4-Flash-0731 anchors, then mapped to official Elo through three frozen calibration nodes and monotone interpolation or extrapolation.The fitted Bradley–Terry utility is estimated from pairwise comparisons; unresolved judge errors are excluded.Source: §E.1 · pp. 25–27

E.2. Pairwise Judge Prompt

  • A blinded pairwise judge compares submissions against original requirements using artifact evidence and correctness, completeness, usability, professional quality, and faithfulness; no task-specific rubrics are supplied.The judge returns a preference or tie with concise, evidence-grounded reasons.Source: §E.1 · p. 26, §E.2 · pp. 27–28

E.1. Two-Anchor Elo Estimation

  • WorkPilot-35B-A3B received 1565 GDPval-AA v2.1 Elo (95% task-bootstrap interval [1539, 1589]), conditional on the task population, anchors, judge, and calibration.Strict and permissive estimates coincide because no comparison receives a maximum-turn override.Source: §E.1 · p. 27
  • Validation MAE/RMSE were 24.11/27.86 Elo; frozen-procedure test MAE/RMSE were 19.00/22.77, and 9 of 10 models fell within ±40 Elo.Predicted and official Elo were strongly aligned across both sets (Pearson r=0.994; Spearman ρ=0.964; Figure 10).Source: §E.1 · p. 27, Figure 10 · p. 29

E.3. GDPval-AA Inference-Cost Accounting

  • Cost per GDPval-AA task uses logged input/output tokens, a 0.992 cache hit rate, and equal task weighting after averaging completed trials; reasoning tokens count as output.Tasks without completed trials and trials with recorded execution exceptions are excluded.Source: §E.3 · pp. 28–29
  • Cost estimates use endpoint proxies or shared model-size prices where needed and exclude judges, tools, sandbox compute, storage, and orchestration; provider prices can change over time.Cache-write charges are not added because usage is not consistently exposed in logs.Source: §E.3 · p. 29

Related papers

Loading 2610.37161v1…