PaperlayerView exact revision on arXiv

Source-linked AI summary

AssetForge: Asset-Augmented Task Synthesis for Business Automation Agents

Xiangrui Liu, Shuo Tang, Siheng Chen

arXiv:2610.18330v2cs.AI

TL;DR

Business automation training needs executable tasks, but rubrics alone do not provide the infrastructure to instantiate them. AssetForge pairs task-specific Rubrics with reusable construction assets and validates candidates; its accepted examples outperform matched Rubrics on AutomationBench public, with gains also reported on AppWorld.

  • Problem

    A rubric makes task-generation requirements explicit but does not provide the infrastructure needed to instantiate a business task.

    Source: §1 · p. 2

  • Method

    AssetForge pairs task-specific Rubrics with reusable application, state, policy/evidence, and assertion assets, then validates candidates with executable checks and model-based review.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    41.17% strict pass@1 for AssetForge versus 30.33% for matched Rubrics at 6,000 accepted examples; both AssetForge supervisors also improve AppWorld completion over the base model.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The comparison supports an advantage for AssetForge’s complete construction pipeline on AutomationBench public, with transfer gains reported on AppWorld.

    Source: §1 · p. 2, §5 · p. 9

    The experiments use synthetic tasks in simulated business applications and benchmark states rather than live enterprise accounts.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Training business automation agents requires executable tasks that connect a natural-language request to an application state and a verifiable end state. Traditional Rubrics requires Authors to recreate application interfaces, initial records, policy evidence, and outcome checks for each task, entangling business logic with repeated infrastructure work. We introduce AssetForge, an asset-augmented task-construction pipeline that pairs a reusable construction-asset library with task-specific Rubrics. The assets provide application interfaces, world-state components, policy/evidence carriers, and native assertion definitions; the Rubric directs their instantiation for a particular business task. Executable checks and an additional model-based review validate each candidate, return targeted feedback, and support bounded repair before acceptance. On AutomationBench v1.0.6 public-600, fine-tuning Qwen3.6-35B-A3B on 6,000 AssetForge examples yields 41.17% on the official strict pass@1 metric, 4.4 times the 9.33% base score (+31.83 points). With the same backbone and GLM-5.3 supervision, AssetForge reaches 41.17% versus 30.33% for traditional Rubrics, a 10.84-point gain (35.7% relative) at the same accepted-example scale (6,000 versus 6,000 examples). Both DeepSeek- and GLM-supervised AssetForge models improve AppWorld completion over the base model on Test-Normal and Test-Challenge. These results establish the advantage of AssetForge's complete construction pipeline on AutomationBench public: the coupled asset-Rubric interface directs Authors toward business logic, while executable checks and model-based review validate their tasks.

1 INTRODUCTION

Business automation tasks require coordinated work across applications and policy-dependent decisions, but a rubric alone does not supply the infrastructure to instantiate them. AssetForge combines reusable construction assets with task-specific Rubrics and validation, and its accepted examples improve benchmark results over matched Rubrics and the base model.

Motivation

  • A rubric specifies task requirements but does not provide the infrastructure to instantiate a business task; reusable assets supply application and state support while the Author specifies scenario-specific logic.Executable checks, model-based review, and repair validate the resulting task.Source: §1 · p. 2
  • Business automation workflows often span several applications and require policy-dependent decisions, motivating tasks with explicit world states and executable success conditions.AutomationBench public contains 600 scored tasks across Sales, Marketing, Operations, Support, Finance, and HR.Source: §1 · p. 2

Contributions

  • 41.17% strict pass@1 versus 30.33% for Rubrics at 6,000 accepted examples, while AssetForge-DS reaches 41.17% versus 9.33% for the base model.The AssetForge–Rubrics comparison uses the same GLM-5.3 supervisor; both AssetForge-supervised models also improve AppWorld completion over the base agent.Source: §1 · p. 2
  • AssetForge packages application interfaces, world-state components, policy/evidence carriers, and native assertion types for Authors to instantiate through task-specific Rubrics.Authors fill in records, business rules, and outcomes instead of rebuilding recurring infrastructure.Source: §1 · p. 2
  • The pipeline combines the asset–Rubric interface with executable checks, model-based review, and bounded repair to produce runnable, coherent accepted QA packages.Failed checks return actionable feedback, and revised tasks are checked again.Source: §1 · p. 2

2 RELATED WORK

Related work covers business-automation benchmarks, executable task synthesis, and rubric-guided generation or evaluation. AssetForge is positioned as a business-specific construction interface that joins reusable assets with task-level validation.

Business automation environments

Executable task synthesis

  • TaskCraft, DIVE, and NexForge construct tasks with resources or verifiers, while AssetForge focuses on reusable business-application, state, policy, and assertion support for Authors.FACET grounds instructions, reference solutions, and verifiers in shared executable state and repairs inconsistent artifacts (Shi et al., 2026).Source: §2 · p. 3

Rubric-guided synthesis and task validation

  • Rubrics serve distinct roles in evaluation, reinforcement-learning feedback, and data construction; OptimSyn more directly uses document-specific rubrics to condition QA generation (Fan et al., 2026a).OpenRubrics synthesizes evaluation rubrics for reward modeling and alignment (Liu et al., 2025).Source: §2 · p. 3
  • AssetForge uses the Author rubric to specify what to construct, assets to realize those requirements, and a code Gate plus model-based review to assess each candidate before acceptance.This separates construction requirements, executable task assertions, and review of overall task coherence.Source: §2 · p. 3

3 METHOD

AssetForge constructs executable business QA tasks by pairing task-specific Rubrics with reusable construction resources. Candidates are instantiated by Authors, checked for execution and coherence, and accepted only after passing both validation stages.

3 METHOD

  • AssetForge’s output is an accepted executable QA task package, validated through an asset-augmented pipeline rather than left as a rubric or unverified draft.The package contains a request, initial world, evidence package, and assertion contract.Source: §3 · p. 3

3.1 RUBRIC-GUIDED AUTHORING WITH CONSTRUCTION ASSETS

  • The task-specific Rubric supplies business logic, while reusable assets provide application interfaces, state components, assertion definitions, and locations for policy rules and evidence.Assets supply a readable location for a rule, not the rule itself; required outcomes need both an application operation and an applicable check.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2 CONSTRUCTING AN EXECUTABLE QA TASK

  • The Author derives a request, initial world, application context, and success assertions from one specification, using supported assets and task-specific assertion arguments.Assertions define successful completion without prescribing one textual answer or unique action sequence.Source: §3.2 · pp. 4–5
  • Authors select relevant capabilities, prepare initial records and evidence, and instantiate scenario-specific requests, worlds, and assertion arguments.Construction resources handle recurring infrastructure while the Author supplies the business logic.Source: §3.2 · p. 5
  • Before submission, native self-checks execute the candidate, verify intended final-state assertions, and block tasks already completed by doing nothing.The candidate has the same four task-level components used at acceptance: request, initial world, application context, and executable assertions.Source: §3.2 · p. 5

3.3 GATE, MODEL-BASED REVIEW, AND TASK-LEVEL FEEDBACK

  • The code Gate checks schema, native compatibility, runtime execution, and required self-checks; model-based review checks rubric fit, accessible evidence, task consistency, and assertion coverage.Both stages must pass on the same candidate version; feedback triggers revision through both checks again, preserving the business goal and required outcomes.Source: §3.3 · p. 5
  • Model-based review can flag rubric ambiguities for future versions, but it does not relax current task requirements or replace the accept/reject decision.The Gate supplies executable evidence and the review supplies semantic quality control; solver success is graded separately by native final-state assertions.Source: §3.3 · p. 5

4 EXPERIMENTS

The experiments compare AssetForge with Rubrics on AutomationBench public-600 and assess transfer to AppWorld, then examine supervision sources, acceptance, and data scale. The controlled results favor the complete AssetForge pipeline over Rubrics, with gains on AutomationBench and improvements over the base on AppWorld.

4.1 EXPERIMENTAL SETUP

  • AutomationBench public-600 has 100 tasks in each of six domains; the matched comparison uses Qwen3.6-35B-A3B and 6,000 examples per branch.The official pass@1 metric is strict first-trial completion; Rubrics and AssetForge-GLM use the same GLM-5.3 supervisor and nominal training recipe.Source: §4.1 · p. 6

4.2 MAIN RESULTS

  • AssetForge exceeds Rubrics in all six domains, with first-trial gains of 2–24 points and all nine cross-trial pairings favoring AssetForge.The first-trial gains are 12, 24, 9, 2, 10, and 14 points in Finance, HR, Marketing, Operations, Sales, and Support.Source: §4.2 · p. 6
  • 41.17% pass@1 for both AssetForge branches exceeds the 9.33% base; with GLM supervision, AssetForge's 41.17% exceeds Rubrics' 30.33% at 6,000 examples.Table 1 reports AutomationBench v1.0.6 public-600 results; pass@1 is the official first-trial metric.Source: §4.2 · p. 6, Table 1 · p. 7
  • Both AssetForge branches improve over the base on AppWorld Test-Normal and Test-Challenge for task and scenario completion.The controlled Rubrics–AssetForge comparison is on AutomationBench; AppWorld is an auxiliary transfer evaluation.Source: Table 2 · p. 7, §4.2 · p. 8

4.3 ABLATION STUDY

  • Both supervision branches reach 41.17% pass@1, while avg3 differs by 1.39 points (41.61% versus 40.22%) and domain leads are split.DeepSeek leads Finance, HR, Marketing, and Operations; GLM leads Sales and Support.Source: Figure 3 · p. 8, §4.3 · p. 8
  • 19.29-point first-submission acceptance gap (88.72% AssetForge versus 69.43% Rubrics) narrows to 19.18 points at final submission, with 6,000 accepted each.The acceptance stages use all initial candidates as their denominator and allow up to two repair rounds.Source: §4.3 · p. 8, Figure 4 · p. 9
  • At 6,000 examples, AssetForge scores 41.17% pass@1 versus Rubrics' 30.33%, a 10.84-point gain; Rubrics gains 5.83 points from 1,500 to 6,000 examples.The matched comparison fixes the backbone, supervisor, evaluation protocol, and accepted-example count; token volume and task structure remain construction outputs.Source: §4.3 · pp. 8–9, Figure 5 · p. 9

5 CONCLUSION

AssetForge combines reusable construction assets, task-specific Rubrics, executable checks, model-based review, and repair to turn task synthesis toward business-logic authoring. Its complete pipeline improves AutomationBench performance over both matched Rubrics and the base model, while evaluation remains limited to synthetic tasks in simulated applications.

5 CONCLUSION

  • 41.17% strict pass@1 on AutomationBench public-600 exceeds matched GLM-5.3 Rubrics at 30.33% (+10.84 points, 35.7% relative) and the base model at 9.33% (4.4×).The matched comparison uses 6,000 accepted examples for each pipeline.Source: §5 · p. 9
  • AssetForge pairs reusable application, state, evidence, and assertion support with task-specific Rubrics, then validates candidates through executable checks, model-based review, and repair.The coupled interface lets Authors focus on scenario-specific business logic rather than repeatedly rebuilding infrastructure.Source: §5 · p. 9
  • Both DeepSeek- and GLM-supervised AssetForge models improve AppWorld completion over the base model on Test-Normal and Test-Challenge.The conclusion also reports similar aggregate results under both supervisors.Source: §5 · p. 9
  • Experiments use synthetic tasks in simulated business applications, evaluate benchmark states rather than live enterprise accounts, and do not reproduce solver training traces.Appendices document evaluation settings, corpus statistics and audit results, the SFT specification, assets, task cases, and repeated-trial results; code is available via the repository linked in the abstract.Source: ETHICS STATEMENT · p. 10, REPRODUCIBILITY STATEMENT · p. 10

A EVALUATION DETAILS

The evaluation uses AutomationBench public-600 and repeats its same 600 tasks across trials. At matched accepted-example scale, AssetForge produces larger and more application-rich tasks than Rubrics, alongside fewer native assertions per task.

A EVALUATION DETAILS

  • AutomationBench v1.0.6 public-600 contains 600 tasks, with 100 tasks in each of six business domains, scored by native final-state assertions.The benchmark runs in its native environment.Source: A EVALUATION DETAILS · p. 13
  • The three trials repeat the same 600 tasks rather than evaluating 1,800 distinct tasks; trial-to-trial randomness comes from sampled model trajectories.The public-600 split is used for reproducible open evaluation.Source: A EVALUATION DETAILS · p. 13

B.1 CORPUS ACCOUNTING

  • Table 4 accounts for the matched 6,000-example Rubrics–AssetForge comparison using accepted-corpus token totals and mean lengths, not action-level execution details.The observed token volume and task structure emerge from each construction pipeline.Source: §B.1 · p. 13, Table 4 · p. 13
  • At approximately 6,000 examples, AssetForge yields 176M tokens versus Rubrics’ 128M, 3.86 versus 2.42 applications per task, and 22.2 versus 42.4 native assertions.These corpus differences are part of the end-to-end pipeline treatment, not separate budget adjustments or equal-token controls.Source: Table 4 · p. 13, §B.1 · p. 13

B.2 PUBLIC-BENCHMARK CONTAMINATION AUDIT

The contamination audit compares both complete 6,000-record training files against all 600 public-600 tasks. Five detectors report no flagged overlap, including for semantic similarity.

B.2 PUBLIC-BENCHMARK CONTAMINATION AUDIT

  • The audit covers both complete 6,000-record SFT training files against all 600 AutomationBench public-600 tasks; the files share 5,626 task IDs.The two training files are independently indexed.Source: §B.2 · pp. 13–14
  • Zero records were flagged against public-600 by any of five detectors; the largest semantic cosine was 0.9089, below the 0.9270 threshold and the null maximum.The semantic detector used BAAI/bge-small-en-v1.5 over the complete 6,000-by-600 matrix.Source: §B.2 · p. 14, Table 5 · p. 14

B.3 SFT SPECIFICATION

The appendix specifies a shared standard full-parameter assistant-only SFT procedure and accounts for the candidate and repair workload needed to obtain matched accepted corpora. Rubrics requires more initial candidates and repair calls than AssetForge.

B.3 SFT SPECIFICATION

  • All models use the same standard full-parameter assistant-only SFT procedure, while construction-only metadata is excluded from solver-facing input.The appendix provides hyperparameters and corpus-level accounting, but not internal message layouts or solver execution traces.Source: §B.3 · p. 14

B.4 ACCEPTANCE-CALL ACCOUNTING

  • Rejected initial candidates are repaired until acceptance or exhaustion of a two-round budget; Figure 4 uses initial candidate submissions as its denominator.The reported Gate, repair, and model-review call ranges are envelopes because the aggregate plot does not show the one-versus-two-round split for each failed candidate.Source: §B.4 · p. 14
  • Rubrics requires at least 3.36 times as many repair calls as AssetForge (5,058/1,504) for approximately 6,000 accepted tasks.Rubrics uses 27.1% more initial candidates; its minimum Gate count of 13,527 exceeds AssetForge’s maximum of 8,169.Source: §B.4 · p. 15, Table 7 · p. 15

C ASSETS AND WORKED TASK EXAMPLES

AssetForge supplies reusable construction resources that Authors instantiate through task-specific Rubrics, while the worked cases illustrate how both pipelines produce executable business tasks. The examples include a traditional Rubrics task that passed strict completion and an AssetForge billing task validated across solver lanes.

C.1 WHAT THE ASSETS PROVIDE

  • AssetForge’s inventory contains 61 construction assets, including application interfaces, shared state, policy/evidence, effect, and background resources.Authors select supported operations and success checks, then instantiate the resources for each business scenario.Source: §C.1 · p. 15, Table 8 · p. 15, Table 9 · p. 15

D.1 CASE A: TRADITIONAL RUBRICS AT 6,000 EXAMPLES

D TWO COMPLETE TASK CASE STUDIES

  • The paired cases illustrate that both pipelines combine a business request, seeded records, success assertions, and an execution trace.AssetForge integrates supplied interfaces and state-construction resources into rubric-guided authoring; aggregate comparisons are reported separately.Source: D TWO COMPLETE TASK CASE STUDIES · p. 16, §D.2 · pp. 18–19
  • Strict completion passed for the traditional support task, while all three solver lanes reached strict 1.0 on the AssetForge billing task.The traditional case links a Slack handoff to Zendesk records; the billing case links Gmail session logs to Slack policy evidence and Sheets context.Source: §D.1 · p. 17, §D.2 · p. 18

D.2 CASE B: ASSETFORGE AT 6,000 EXAMPLES

  • The AssetForge billing case uses Gmail, Google Sheets, and Slack, with 22 native assertions and a 24-step validated execution trace.The task card reports three solver lanes, all with strict score 1.0.Source: §D.2 · p. 18, Table 11 · p. 18, Table 13 · p. 19

D.3 BILLING TASK DETAILS

The billing task turns policy evidence and application records into executable success conditions, including protected no-action requirements. The section also describes reuse of construction resources across Finance and HR and distinguishes the AssetForge rubric’s guidance from traditional rubric requirements.

D.3 BILLING TASK DETAILS

  • The billing task’s 22 native Gmail assertions cover labels, replies, summary totals, and protected no-op conditions, and the validated execution passes every check.A no-action attempt fails, and model-based review accepts the task.Source: §D.3 · p. 20

D.4 REUSE ACROSS BUSINESS TASKS

  • The same five construction resources support Finance and HR tasks with different requests, initial records, and success checks.The examples have 25 and 22 assertions, respectively, illustrating reuse across domains while business logic and evidence change.Source: §D.4 · p. 20

E AUTHOR-RUBRIC EXCERPTS

  • Unlike the traditional rubric, the AssetForge rubric identifies supplied resources and explains how to derive a task record from a complete business specification.Both rubrics require executable tasks and validation; the additional model-based review checks consistency among the request, initial state, evidence, and assertions, complementing the executable Gate.Source: E AUTHOR-RUBRIC EXCERPTS · p. 20

F.1 REPEATED AUTOMATIONBENCH EVALUATIONS

Across repeated AutomationBench evaluations, AssetForge’s advantage persists across both supervision branches and inference runs. The reported comparisons cover domain-level scores and trial aggregates on the benchmark’s 600-task evaluations.

F.1 REPEATED AUTOMATIONBENCH EVALUATIONS

  • The reported local evaluations are distinct from published aggregate scores and are not combined with local domain scores or repeated-trial averages.Each trial in the strict-success comparison contains 600 tasks.Source: §F.1 · p. 20, Table 14 · p. 21
  • For AssetForge-DS, the lowest score across three trials exceeds the highest Rubrics score, with domain differences of 3, 18, 3, 3, 5, and 6 percentage points.The differences correspond to Finance, HR, Marketing, Operations, Sales, and Support; AssetForge-GLM exceeds Rubrics in all six domains in its first-trial breakdown.Source: §F.1 · p. 21, Table 15 · p. 22
  • For AssetForge and Rubrics, both AssetForge supervision branches and Rubrics report three trials; T1–T3 are success counts out of 100 in each domain, while final-row counts use 600 per trial and Avg3 is a percentage.Source: Table 15 · p. 22

F.2 DATA QUANTITY AND CONSTRUCTION SUPPORT

At the same accepted-example count, AssetForge-GLM outperforms the largest Rubrics configuration in the first-trial AutomationBench comparison. The reported gain is a pipeline-level comparison under a shared surrounding workflow, not an equal-token-budget comparison.

F.2 DATA QUANTITY AND CONSTRUCTION SUPPORT

  • 10.84 points separate AssetForge-GLM from the 6,000-example Rubrics result, with 247 versus 182 successes out of 600.The comparison holds the surrounding authoring, checks, review, repair, supervision, training, and evaluation protocol fixed.Source: §F.2 · p. 22
  • 5.83 points are the cumulative Rubrics gains as its training set grows fourfold.The successive gains are 2.67, 1.50, and 1.66 percentage points.Source: §F.2 · p. 22
  • The comparison matches accepted-example count, not token budget, and attributes the gain to the pipeline-level integration of the coupled asset–Rubric interface.The table reports the first complete evaluation for each configuration using the common first-trial protocol.Source: §F.2 · p. 22
Loading 2610.18330v2…