PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Agentic Workflow Generation

Shuofei Qiao, Runnan Fang, Zhisong Qiu, Xiaobin Wang, Ningyu Zhang, Yong Jiang, Pengjun Xie, Fei Huang, Huajun Chen

arXiv:2410.07869v3cs.CLcs.AIcs.HCcs.LGcs.MA

TL;DR

LLM workflow benchmarks have limited scenario coverage, simplistic linear structures, and weak evaluation standards. The paper introduces WORFBENCH and WORFEVAL for diverse, graph-structured workflow generation and quantitative chain- and graph-level assessment. Experiments find substantial gaps between linear and graph planning, while generated workflows improve downstream performance with less inference time.

  • Problem

    Existing workflow evaluations have restricted scenarios, simplistic linear structures, and lax standards for measuring workflow-generation ability.

    Source: §5.3 · p. 10

  • Method

    WORFBENCH benchmarks workflow generation across diverse scenarios and DAG structures, while WORFEVAL uses subsequence and subgraph matching for quantitative assessment.

    Source: §1 · p. 2, §2.1 · p. 2

  • Results

    Graph planning trails linear planning across models, with GPT-4 achieving 52.47% graph-workflow performance and generated workflows improving downstream performance while reducing inference time.

    Source: §1 · p. 2, §3.2 · p. 6

  • Takeaways & Limitations

    Workflow generation should be evaluated at both sequence and graph levels, and generated workflows can support more efficient downstream planning.

    Source: §1 · p. 2, §6 · p. 10

    The benchmark is limited by unresolved query-synthesis quality issues, uncovered scenarios, natural-language workflow representation, and a maximum of 20 topological orders.

    Source: §3.1 · p. 5, §A.8 · p. 25

Abstract

from arXiv · show

Large Language Models (LLMs), with their exceptional ability to handle a wide range of tasks, have driven significant advancements in tackling reasoning and planning tasks, wherein decomposing complex problems into executable workflows is a crucial step in this process. Existing workflow evaluation frameworks either focus solely on holistic performance or suffer from limitations such as restricted scenario coverage, simplistic workflow structures, and lax evaluation standards. To this end, we introduce WorfBench, a unified workflow generation benchmark with multi-faceted scenarios and intricate graph workflow structures. Additionally, we present WorfEval, a systemic evaluation protocol utilizing subsequence and subgraph matching algorithms to accurately quantify the LLM agent's workflow generation capabilities. Through comprehensive evaluations across different types of LLMs, we discover distinct gaps between the sequence planning capabilities and graph planning capabilities of LLM agents, with even GPT-4 exhibiting a gap of around 15%. We also train two open-source models and evaluate their generalization abilities on held-out tasks. Furthermore, we observe that the generated workflows can enhance downstream tasks, enabling them to achieve superior performance with less time during inference. Code and dataset are available at https://github.com/zjunlp/WorfBench.

1 INTRODUCTION

The paper introduces WORFBENCH and WORFEVAL to evaluate LLM workflow generation across diverse scenarios and complex graph structures, addressing limitations in prior benchmarks. It reports evaluations showing gaps between linear and graph planning while exploring workflow benefits for downstream performance and efficiency.

Motivation

  • Prior benchmarks have limited scenario coverage, emphasize linear subtask relationships, and rely heavily on GPT-3.5/4 despite hallucination and ambiguity concerns.These limitations motivate a more systematic evaluation of workflow generation.Source: §1 · p. 2

Benchmark design

  • WORFBENCH covers four agentic scenarios with complex workflow structures, strict quality control, and 18k training, 2146 test, and 723 held-out tasks.The scenarios are problem-solving, function calling, embodied planning, and open-grounded planning; workflows are modeled as dependency-based directed acyclic graphs.Source: §1 · p. 2

Evaluation protocol

  • WORFEVAL applies subsequence and subgraph matching to quantify LLM workflow-generation ability across chain and graph structures.The protocol is designed to assess both linear planning and graph planning rather than only end-to-end outcomes.Source: §1 · p. 2

Main findings

  • Even GPT-4 achieves only 52.47% on graph-structured workflow prediction, while evaluations reveal a substantial gap relative to linear workflow prediction.The study also trains two open-source models on the training set and evaluates generalization on held-out tasks.Source: §1 · p. 2

Downstream application

  • Generated workflows improve downstream model performance while reducing inference time through parallelizing and shortening planning steps.The workflows are used as chain-of-thought augmentation and prior knowledge in downstream tasks.Source: §1 · p. 2

2 WORFBENCH

WorfBench formulates workflow generation as predicting a DAG from a task and action list, then constructs and filters benchmark workflows before evaluating them with sequence and subgraph matching.

2.1 TASK FORMULATION

  • A workflow is modeled as a DAG whose nodes are executable subtasks and whose edges encode execution dependencies.All nodes must be executed according to their dependencies before the task is complete.Source: §2.1 · pp. 2–3

2.2 BENCHMARK CONSTRUCTION

  • The benchmark first creates a topological node chain and then adds numbered edges, with START and END nodes marking workflow boundaries.Construction methods vary across function-call, embodied, and open-grounded tasks.Source: §2.1 · p. 3, §2.2 · pp. 3–4

2.3 QUALITY CONTROL

  • Quality control filters invalid decompositions and inconsistent graphs, removing 15.36% of node-chain data and 29.77% of workflow-graph data.Graph validation uses topological sorting aligned to node-chain order, followed by sampling and manual test-set checks.Source: §2.3 · p. 4

2.4 WORFEVAL

  • WorfEval matches semantic nodes with thresholded bipartite alignment, scores chains by longest valid subsequences, and scores graphs with maximum common induced subgraphs.The resulting precision and recall measures quantify both predicted node-chain and workflow-graph quality.Source: §2.4 · pp. 4–5

3 EXPERIMENTS

The experiments evaluate 18 language models under standardized two-shot settings and compare linear node-chain planning with graph workflow planning. Results show that graph planning remains substantially harder, generalization to held-out tasks is limited, and errors often reflect missing environmental knowledge.

3.1 EXPERIMENTAL SETTINGS

  • Eighteen models are evaluated with identical carefully designed instructions and two-shot examples, default decoding hyperparameters except temperature 0.5, and semantic-matching threshold β=0.6.The models include closed-source and open-source systems ranging from 7B to 72B parameters.Source: §3.1 · pp. 5–6, Table 1 · p. 6

3.2 MAIN RESULTS

  • 15.01–20.05%: workflow graph scores trail node-chain scores across models, with GPT-4 averaging 52.47% graph-planning performance.The smallest disparity occurs for Llama-3.1-70B, while the largest occurs for GLM-4-9B.Source: §3.2 · pp. 6–7
  • 10%+ and 15%+: fine-tuned 7B models surpass GPT-4 on held-in tasks in f1chain and f1graph, but held-out-task advantages shrink to 2.5–3.5% on Seal-Tools and remain limited on harder tasks.The authors report especially weak generalization on embodied tasks, indicating that fitting training data alone does not yield robust structured workflow planning.Source: §3.2 · p. 7, Table 2 · p. 7
  • Four recurring low-f1graph error types are identified: granularity, explicitness, graph structure, and output format.The analysis attributes most errors to insufficient environmental knowledge, including weak understanding of task details and action dependencies.Source: §3.2 · pp. 7–8, Figure 4 · p. 7

4 THE ROLE OF WORKFLOW FOR AGENT PLANNING

The paper studies workflows as structured guidance for improving agent planning quality and inference efficiency. Experiments examine prior-knowledge and CoT-style workflow augmentation, parallel execution, and reduced planning steps.

4 THE ROLE OF WORKFLOW FOR AGENT PLANNING

  • The workflow generator for these downstream experiments is the trained Qwen-2-7B model.The section uses this model as the default workflow generator for function invocation and embodied planning studies.Source: §4 · p. 8

4.1 ENHANCE END-TO-END PERFORMANCE

  • Workflow knowledge improves planning for GPT-4, Llama-3.1-8B, and Qwen-2-72B, with larger gains on the more diverse and complex ALFWorld tasks.Workflows generated by a 7B model also guide the substantially larger 72B model.Source: §4.1 · p. 8, Table 3 · p. 8
  • Workflow-guided CoT improves function invocation by focusing reasoning on each workflow node and retrieving a relevant API for that step.The workflow participates actively in planning rather than serving only as external reference knowledge.Source: §4.1 · pp. 8–9, Figure 5 · p. 8

4.2 REDUCE END-TO-END INFERENCE-TIME

  • Graph-structured parallelization reduces average task-completion time by approximately one-fifth to one-third across different test sets.The analysis identifies the longest START-to-END path as the critical path and compares it with linear ToolLlama execution.Source: §4.2 · p. 9, Figure 6 · p. 9
  • Workflow knowledge also vertically reduces the number of planning steps by replacing blind environmental trial-and-error with more purposeful actions.Table 4 quantitatively analyzes average planning steps with and without workflow knowledge.Source: §4.2 · p. 9, Table 4 · p. 9

5 RELATED WORK

Prior agent research often emphasizes end-to-end task performance while underassessing the reasoning and planning capabilities underlying reliable behavior. Related workflow studies motivate structured prior knowledge and automatic generation, but existing evaluations remain narrow in scenarios, workflow structure, and evaluators.

5.1 LARGE LANGUAGE AGENTS

  • Many language-agent methods focus primarily on end-to-end task performance rather than directly assessing the reasoning and planning capabilities underlying stable, reliable behavior.Source: §5.1 · pp. 9–10

5.2 WORKFLOW AND LANGUAGE AGENT PLANNING

  • Workflows serve as intermediate representations connecting tasks to executable actions, while also improving interpretability, debugging, human involvement, and resistance to planning hallucinations.The paper relates its DAG workflow to Marked Graphs, a restricted subclass of Petri nets.Source: §5.2 · p. 10

5.3 AGENTIC WORKFLOW GENERATION AND EVALUATION

  • Human-designed workflows can constrain agent planning but are time-consuming and inflexible, motivating automatic workflow generation.Source: §5.3 · p. 10
  • Existing workflow evaluations mainly cover function-calling or reasoning, use single linear structures, and rely largely on GPT-4 or human evaluation.These limitations leave embodied interactive scenarios and more complex task structures underrepresented.Source: §5.3 · p. 10

6 CONCLUSION

The conclusion presents WORFBENCH and WORFEVAL as a benchmark and evaluation framework for agentic workflow generation across diverse scenarios and graph structures. It reports gaps between linear and graph workflow generation, evaluates generalization, and studies downstream planning performance and efficiency.

6 CONCLUSION

  • WORFBENCH provides a unified benchmark with miscellaneous scenarios and intricate graph-structured workflows.Source: §6 · p. 10
  • WORFEVAL quantitatively evaluates both linear and graph workflows to assess LLM agents’ workflow-generation capabilities.Source: §6 · p. 10
  • Experiments investigate performance gaps between traditional linear-structured and complex graph-structured workflow generation, along with open-source-model generalization on held-out tasks.Source: §6 · p. 10
  • The paper also examines workflows as tools for improving downstream planning performance and inference-time efficiency.Source: §6 · p. 10

REPRODUCIBILITY STATEMENT

A.1 SOURCE DATA INFORMATION

The benchmark draws on multiple public datasets covering tool use, embodied interaction, operating systems, reasoning, open-world planning, and SQL interaction. These sources provide trajectories, instructions, or process data for constructing workflow tasks.

A.1 SOURCE DATA INFORMATION

  • ToolBench and StableToolBench provide function-call instructions, with StableToolBench used for stable end-to-end API evaluation.Source: §A.1 · p. 20
  • ToolAlpaca contributes multi-function-call instructions after filtering out single-call examples.Source: §A.1 · p. 20
  • ALFWorld and WebShop supply embodied and online-shopping trajectories used to construct benchmark workflows.ALFWorld uses human-annotated gold trajectories, while WebShop uses trajectories collected through GPT-4 in Song et al. (2024).Source: §A.1 · p. 20
  • OS, LUMOS, and WikiHow contribute operating-system interactions, reasoning components, and open-world planning processes for workflow construction.Source: §A.1 · p. 21
  • Seal-Tools and InterCodeSQL add self-instructed function-call data and embodied SQL interaction data.Source: §A.1 · p. 21

A.2 HUMAN VERIFICATION

The test data was divided into five parts and assessed by five NLP volunteers for node-chain and workflow-graph quality. Data violating granularity, logic, or task-quality criteria was discarded.

  • Five NLP volunteers evaluated node chains and workflow graphs after the test data was divided into five parts.Source: §A.2 · p. 21
  • Granularity required the smallest executable subtasks, without combining actions or adding information unavailable from the task and action list.Source: §A.2 · p. 21
  • Workflow graphs were required to follow logical sequencing consistent with execution relationships between nodes.Source: §A.2 · p. 21
  • Data with violations of the granularity, logic, or task-quality criteria was discarded from the final test set.Source: §A.2 · p. 21

A.3 BENCHMARK STATISTICS

The benchmark contains 18,679 training instances and 2,146 test instances spanning four task types, with held-out tasks supporting generalization evaluation. Workflows average 4.17 nodes and are usually 2–10 steps long.

  • 18,679 training instances are evenly distributed across function-call, problem-solving, embodied, and open-grounded tasks, while the test set contains 2,146 instances.Held-out tasks comprise 33.69% of the test set for evaluating generalization.Source: §A.3 · p. 21
  • 33.69% of the test set is dedicated to held-out tasks for evaluating trained-model generalization.Source: §A.3 · p. 21
  • Most workflows contain 2 to 10 nodes, fewer fall within 10 to 20 nodes, and the benchmark-wide average is 4.17 nodes.Source: §A.3 · p. 21

A.4 ABLATION STUDY

Providing gold node chains improves workflow-graph prediction by removing granularity and explicitness problems, but dependency prediction remains challenging and overall performance remains subpar.

  • Providing gold node chains clearly improves performance when agents only predict workflow-graph edges.The ablation compares traditional f1graph without gold chains against edge generation assisted by gold chains.Source: §A.4 · p. 21
  • Gold node chains alleviate granularity and explicitness issues, while predicting graph relationships between nodes remains difficult.Source: §A.4 · p. 21
  • Even after supplying gold node chains, overall performance remains subpar because graph-dependency prediction is still challenging.Source: §A.4 · p. 21

A.5 TRAINING SETUPS

The paper describes benchmark training setups, O1 case studies, construction prompts, benchmark limitations, and topological-order sampling for efficient node-chain evaluation.

A.5 TRAINING SETUPS

  • A.5 TRAINING SETUPS: Qwen-2-7B and InternLM-2.5-7B are fully fine-tuned with DeepSpeed using identical hyperparameters and two-shot prompts.Training uses LlamaFactory and three NVIDIA 80GB A100 GPUs.Source: §A.5 · pp. 21–22, Table 6 · p. 22
  • A.5 TRAINING SETUPS: Figure 7 summarizes benchmark statistics, while Figure 8 shows workflow-step counts across the benchmark.Source: Figure 7 · p. 22, Figure 8 · p. 22

A.6 CASE STUDY OF O1

  • A.6 CASE STUDY OF O1: O1 misses parallel dependencies in one case and omits locating a potato in another because of insufficient environmental knowledge.The cases support the conclusion that planning complexity involves interaction with the environment and substantial environmental commonsense knowledge.Source: §A.6 · p. 22
  • A.6 CASE STUDY OF O1: Correct workflows can provide prior knowledge that enhances planning, especially for tasks requiring environmental world knowledge.Source: §A.6 · p. 23

A.7 PROMPT FOR BENCHMARK CONSTRUCTION

  • A.7 PROMPT FOR BENCHMARK CONSTRUCTION: Node-generation prompts convert trajectory information into distinct, concise planning-oriented node tasks while ignoring API names and execution errors.Source: §A.7 · p. 24
  • A.7 PROMPT FOR BENCHMARK CONSTRUCTION: Workflow-graph prompts ask planners to map subtasks into a topology with START and END nodes while representing dependencies and independence.Source: §A.7 · pp. 24–25

A.8 LIMITATIONS

  • A.8 LIMITATIONS: The benchmark excludes heterogeneous actions, code-form workflows, iterative environmental feedback, and graphs with choices or loops.The authors also note that some synthesized queries may themselves have quality issues and that uncovered scenarios remain possible.Source: §A.8 · p. 25

A.9 ANALYSIS OF THE TOPOLOGICAL ORDERS NUMBER IN NODE CHAIN EVALUATION

  • A.9 ANALYSIS OF THE TOPOLOGICAL ORDERS NUMBER IN NODE CHAIN EVALUATION: The method samples 20 topological orders to balance matching efficiency and evaluation accuracy.The choice reflects the observed distribution, in which most workflows have at most 10 topological sorts and counts above 20 are sparse.Source: §A.9 · p. 25, Table 7 · p. 25

Related papers

Loading 2410.07869v3…