Chat2Workflow: A Benchmark for Generating Executable Visual Workflows with Natural Language

• More than 70% of real-world agent deployments use off-the-shelf language models without weight tuning and rely on explicit workflow orchestration, which industry experience identifies as better suited to reliable, controllable use (Pan et al., 2025; Shi et al., 2025).Platforms such as Dify and Coze primarily define agent behavior through workflows.Source: §1 · p. 1
• Manual workflow engineering requires translating natural-language requirements into structured workflows, while implicit requirements and changing requests make correct control flow, tool use, and revisions difficult.Revisions must preserve workflow correctness and consistency.Source: §1 · pp. 1–2
• Chat2Workflow introduces a benchmark of 237 human-annotated instances across six domains, with realistic requests and target workflow components spanning diverse complexity levels.It evaluates whether generated workflows meet execution requirements and tests structure inference, tool selection, and executable, intent-aligned output.Source: §1 · p. 2
• 6.05%: an error-driven agentic baseline with auto-repair improves absolute resolve rates by this amount, but models still struggle with correctness and stability as complexity or requirements evolve.Current state-of-the-art language models often capture high-level intent but have difficulty producing correct, stable, executable workflows.Source: §1 · p. 2

2.1 Workflow Orchestration Platforms

• The benchmark targets visual workflows built on Dify and Coze, where connected functional nodes form DAGs stored as YAML and executed node by node.It covers six domains: AIGC, Research, Document, Education, Enterprise, and Developer.Source: §2.1 · p. 2, Figure 2 · p. 2

2.2 Task Formulation

• Generation takes current instructions and dialogue history to produce JSON DAGs; a Chain-of-Thought format outputs selected nodes, rationale, and workflow JSON converted to executable YAML.The DAG represents nodes and their parameters alongside directed edges that specify source, target, and output branch port.Source: §2.2 · p. 2

2.3 Node Selection and Processing

• The benchmark selects 20 frequently used node types, simplifies I/O to essential key-value inputs with defaults, and supplies a structured node knowledge base derived from official documentation.The knowledge base describes node type, purpose, parameters, referable variables, and supplementary information.Source: §2.3 · pp. 2–3

2.4 Dataset Construction

• Volunteers pruned and filtered production configurations, clustered and abstracted workflows into six scenarios, chained them into 2–4 instruction rounds, and instantiated each with three test cases.Test cases were drawn from adapted Hugging Face datasets, real-world internet materials, or LLM-synthesized data; deterministic tasks include reference answers.Source: §2.4 · pp. 3–4

2.5 Quality Control and Dataset Statistics

• Cross-auditing produced 27 tasks, 79 multi-turn instructions, and 237 test instances, with ambiguous, invalid, time-varying, or incoherent cases removed.Instructions had to support unambiguous manual recreation of key nodes and core functions matching the creator’s intent.Source: §2.5 · p. 4

2.6 Evaluation Protocol

• Pass rate checks structural and format validity plus YAML importability, while resolve rate requires successful execution producing outputs that satisfy all instruction requirements.Three volunteers reviewed GPT-5.1 judgments: strict agreement was 97.18% for pass and 93.09% for resolve; DeepSeek-V3 validation yielded 96.62% and 91.79%, respectively.Source: §2.6 · pp. 4–5

3.1 Experimental Setup

• The benchmark evaluates 15 representative models over three independent runs on Dify version 1.9.2.The set comprises four closed-source and eleven open-source models; GPT-5.1 and GPT-5.2 are selected for the advanced agentic evaluation.Source: §3.1 · p. 5

3.2 Main Results

• Every model has a lower Resolve Rate than Pass Rate, showing that syntactic compliance alone does not ensure successful workflow execution.GLM-4.6 has the largest average gap at 27.43%, reaching 47.48% in Education; Kimi-K2-Instruct has an 11.68% gap but poor absolute performance, especially in Developer tasks.Source: §3.2 · p. 5
• Gemini-3-Pro-Preview leads end-to-end task resolution at 60.20%, while the best open-source model, GLM-4.7, reaches 46.55%.The paper notes that even the top result remains far below the threshold for a reliable, deployable planner.Source: §3.2 · p. 5
• Scaling, alignment, and reasoning affect performance, but better format compliance does not necessarily improve task resolution.Qwen-3 improves as it scales from 8B to 235B; in Developer tasks GLM-4.6 has a 45.83% Pass Rate versus GLM-4.7's 41.67%, but a 5.55% versus 19.45% Resolve Rate. Qwen-3-Coder-480B-A35B-Instruct resolves 19.13% versus Qwen-3-235B-A22B at 24.47%, while Kimi-K2-Thinking outperforms its instruct counterpart across task domains.Source: §3.2 · p. 6
• The analysis groups workflow-generation errors into format, graph, semantics, and consistency failures.These include malformed syntax, structural or topological flaws, misunderstandings of node rules or environment constraints, and contradictions with user intent, requested I/O, or prior turns.Source: §3.2 · pp. 6–7

3.3 Analysis of User Requirement Changes from Chat Interactions

• Workflow quality generally declines as interactions accumulate, but the degradation curve flattens in later rounds.The analysis covers the first three rounds because the fourth has limited samples; GPT-5.2's slight third-round improvement is attributed to a changing denominator as two-round tasks leave scope.Source: §3.3 · p. 7

3.4 Case Analysis

• The case study shows distinct failures: Kimi-K2-Instruct creates an invalid edge, GLM-4.6 omits selected nodes from its declaration, and GPT-5.2 produces an executable workflow.The invalid edge connects an iteration node to its internal iteration-start node despite documentation restricting their relationship to inclusion only.Source: §3.4 · p. 7

3.5 An Error-Driven Agentic Baseline

• Resolve Rate rises by 4.78% for GPT-5.1 and 6.05% for GPT-5.2 with the error-driven agentic baseline.The OpenCode-based approach uses task-critical SKILL guidelines, Variable Summaries, a five-attempt execution loop with structural and semantic verification, and targeted auto-repair modules.Source: §3.5 · p. 7

4.1 LLM-based Agents

• Despite strong capabilities, tool-, collaboration-, API-, and prompt-based LLM agents often neglect intermediate planning and reasoning requirements, undermining stable execution and cross-domain reproducibility.Prior work explores general tools (Schick et al., 2023; Patil et al., 2024), collaborative frameworks (Hong et al., 2024), API repositories (Qin et al., 2024; Tang et al., 2023; Zhang et al., 2024a), and prompting methods (Wei et al., 2022; Yao et al., 2022; Zhu et al., 2026); the cited limitation is discussed by Ge et al. (2024), Xie et al. (2024), Liu et al. (2024), Yang et al. (2026), and Liang et al. (2026).Source: §4.1 · pp. 7–8

4.2 Workflow and Language Agent Planning

• Workflows decompose high-level goals into executable steps and logically organized plans, making reasoning transparent and improving interpretability and reliability.Prior work describes structural patterns and robotic process automation (van der Aalst et al., 2003; Commoner et al., 1971; Ivanˇci´c et al., 2019; Hofmann et al., 2020); workflow generation has shifted from labor-intensive manual design to LLM-based automation, including iterative synthesis and knowledge-augmented planning (Shen et al., 2023a; Zeng et al., 2023; Zhang et al., 2024b; Li et al., 2024; Ye et al., 2023; Zhu et al., 2024; Wornow et al., 2024; Huang et al., 2024).Source: §4.2 · p. 8

• Chat2Workflow evaluates whether LLMs can generate deployable visual workflows from natural language, providing a realistic testbed and concrete target for automated workflow engineering.The benchmark is intended to help bridge the gap between language model-driven workflow design and industrial deployment.Source: §5 · p. 8
• Even with an agentic framework, current models remain fragile under structural constraints and requirement changes.Source: §5 · p. 8

• The manually verified dataset may not encompass the near-infinite variety of logic in complex industrial business processes.Source: Limitations · p. 9
• Simplified node interfaces prioritize executability but may omit intricate parameter configurations required in some real-world deployments.Source: Limitations · p. 9
• The system includes only 20 high-frequency node types, covering most standard scenarios while leaving many valuable tools unincorporated.Source: Limitations · p. 9

Research

• Research has 5 tasks for obtaining, understanding, and integrating knowledge from unstructured data through in-depth reading, logical organization, and critical thinking.Inputs typically include papers, books, or research topics.Source: A Detailed Dataset Description · p. 12

Document

• Document has 5 tasks that transform files, images, and tables into structured results for downstream computation, reuse, and processing.The paper describes this domain as an “AI preprocessing layer” in data processing.Source: A Detailed Dataset Description · p. 12

Enterprise

• Enterprise has 4 tasks that use internal or business documents to automate, standardize, and support decisions in complex business processes.Source: A Detailed Dataset Description · p. 12

Developer

• Developer has 3 tasks for software development, system design, and technical understanding, aiming to lower technical barriers and improve engineering efficiency.Source: A Detailed Dataset Description · p. 12

Education

• Education has 4 tasks covering the teaching-and-learning life cycle, from learning plans and teaching content to assessment and feedback.The goal is a closed-loop system.Source: A Detailed Dataset Description · p. 12

AIGC

• AIGC has 6 tasks for turning often abstract or brief instructions into complete, publishable content or materials, emphasizing creative expression and multimodal output.Source: A Detailed Dataset Description · p. 12

B Experimental Settings

• A rule-based framework converts agent-generated JSON workflows into valid Dify YAML, with extensions and common models prepared for runnable, reproducible workflows.This lets agents use a simplified JSON representation while remaining compatible with Dify's execution environment.Source: B Experimental Settings · p. 12
• The setup uses CoT output, a 20-type node knowledge base, fixed models for selected nodes, Qwen-3-VL-Plus decoding settings, and opencode 1.3.17.Question Classifier and Parameter Extractor use Qwen-3-Max; LLM uses Qwen-3-VL-Plus, Text to Image uses Z-image-Turbo, and Text to Speech uses GPT-4o-Mini-TTS. Qwen-3-VL-Plus uses temperature 0.7 and max_tokens 32768, with other decoding hyperparameters at defaults; the system prompt and SKILL.md were to be added to the GitHub repository soon.Source: B Experimental Settings · p. 12

Round 1

• Round 1 extracts four learning fields from the user’s instruction and outputs a detailed self-study improvement plan in Markdown.The fields are interested learning field, learning goals, learning preferences, and average learning duration.Source: C Instructions for the Studyplanner Task · p. 12

Round 2

• Round 2 first generates a course syllabus, then iteratively generates knowledge points by chapter to build a full tutorial.The content should be rigorous and include rich examples, pros and cons, and precautions.Source: C Instructions for the Studyplanner Task · p. 12
• Round 2 outputs only the final tutorial after integrating it through template conversion.Source: C Instructions for the Studyplanner Task · p. 12

D More Related Work on Automated Workflow Evaluation

• Prior studies automate evaluation in tool-learning scenarios (Li et al., 2023; Shen et al., 2023b; Xu et al., 2025) and fine-grained planning tasks (Chen et al., 2024; Ye et al., 2024; Zheng et al., 2024), using format-validity and tool-consistency metrics.Source: D More Related Work on Automated… · p. 13
• These works often assess only abstract workflow elements (Qiao et al., 2024), omitting execution effects (Guo et al., 2024), structural dependencies (Lal et al., 2024), platform constraints (Shen et al., 2024), and changing requirements across rounds.They may catch obvious format errors but cannot ensure usability.Source: D More Related Work on Automated… · p. 13