Source-linked AI summary
Chat2Workflow: A Benchmark for Generating Executable Visual Workflows with Natural Language
Yi Zhong, Buqiang Xu, Yijun Wang, Zifei Shan, Shuofei Qiao, Guozhou Zheng, Ningyu Zhang
TL;DR
Manual engineering makes executable visual workflows costly and error-prone to create and revise, motivating automated generation from natural language. The paper introduces Chat2Workflow and an error-driven agentic baseline, finding a 6.05% absolute resolve-rate improvement while current models remain fragile under structural constraints and changing requirements.
Problem
Manual engineering of executable workflows is costly and error-prone, while complex or changing requirements make automated generation difficult.
Method
Chat2Workflow benchmarks natural-language generation of executable visual workflows, and an error-driven agentic baseline uses auto-repair mechanisms to improve generation.
Results
6.05% absolute resolve-rate improvement is achieved by the agentic baseline, while current models struggle to produce correct and stable workflows as complexity or requirement changes increase.
Takeaways & Limitations
Chat2Workflow provides a realistic testbed and concrete target for research on automated workflow engineering and industrial deployment.
Takeaways & Limitations
The benchmark may not cover the variety of logic in complex industrial processes, simplifies node interfaces, and currently includes only 20 high-frequency node types.
Abstract
from arXiv · showhide
At present, executable visual workflows have emerged as a mainstream paradigm in real-world industrial deployments, offering strong reliability and controllability. However, in current practice, such workflows are almost entirely constructed through manual engineering: developers must carefully design workflows, write prompts for each step, and repeatedly revise the logic as requirements evolve -- making development costly, time-consuming, and error-prone. To study whether large language models can automate this multi-round interaction process, we introduce Chat2Workflow, a benchmark for generating executable visual workflows directly from natural language, and propose a robust agentic baseline to improve performance. The benchmark is built from a large collection of real-world business workflows, with each instance designed so that the generated workflow can be transformed and directly deployed to practical workflow platforms such as Dify and Coze. Experimental results show that while state-of-the-art language models can often capture high-level intent, they struggle to generate correct, stable, and executable workflows, especially given complex and evolving requirements. Although our agentic baseline yields up to 6.05% resolve rate gains, the remaining real-world gap positions Chat2Workflow as a foundation for advancing industrial-grade automation. Code is available at https://github.com/zjunlp/Chat2Workflow.
1 Introduction
Agentic workflows are widely used for reliable, controllable applications but remain manually engineered. Chat2Workflow benchmarks their generation from natural-language requirements and shows that current models struggle with executable reliability as requirements become complex or evolve.
- Agentic workflows are prevalent in industrial applications, yet developers still largely create them through manual engineering.Production systems commonly use explicit workflows for orchestration, including platforms such as Dify and Coze.
- Complex, implicit requirements make it difficult to infer correct control flow and tool usage from natural language.Changing requirements also require workflow revision or regeneration while preserving correctness and consistency.
- Chat2Workflow is a benchmark for generating executable visual workflows from natural language, comprising 237 human-annotated instances across six domains.Instances include realistic requests and target workflow components spanning diverse domains and complexity levels.
- The benchmark evaluates whether models can infer structures, select tools, and produce executable, intent-aligned workflows.
- 6.05% absolute resolve-rate gain is achieved by the proposed error-driven agentic baseline with auto-repair mechanisms.Despite this improvement, current models still struggle with correct and stable workflows, particularly under complexity and evolving requirements.
2 Constructing Benchmark
Chat2Workflow constructs executable workflow benchmarks from real-world platform configurations through multi-turn task creation, structured workflow generation, and progressive execution-based evaluation.
- Workflow representation: Dify and Coze represent workflows as executable directed graphs stored in structured YAML, with nodes handling tools, inference, and data flow.The benchmark targets workflows that can be generated in JSON, converted to YAML, imported into platforms, and executed.
- Workflow generation: The generation method uses Chain-of-Thought output with node selection, design rationale, and a structured JSON workflow that is converted into executable YAML.The workflow JSON records nodes and connectivity, while the simplified interfaces expose essential variables and rely on defaults for secondary variables.
- Dataset construction: The benchmark construction pipeline prunes unsupported or untestable workflows, clusters them into six scenarios, abstracts functions, chains 2–4 interaction rounds, and instantiates test cases.The six scenarios are Research, Document, Enterprise, Developer, Education, and AIGC.
- Dataset construction: Chat2Workflow contains 27 tasks, 79 multi-turn instructions, and 237 test instances designed to assess complex execution requirements.Each instruction is accompanied by three test cases covering historical requirements and multi-branch logic.
- Evaluation: Evaluation first measures Pass Rate for structural and logical validity, then Resolve Rate for functional execution correctness.A passing subtask must align JSON with the reasoning sequence and predefined variables and convert into an importable YAML file.
- Evaluation: Strict three-evaluator agreement reached 97.18% for the pass stage and 93.09% for the resolve stage, with DeepSeek-V3 cross-validation at 96.62% and 91.79%.The protocol prioritizes formal compliance over deep semantic accuracy.
3 Experiments
Experiments show that current language models often produce syntactically valid workflows without reliable execution, and performance declines as requirements become more complex or evolve across dialogue rounds. The error-driven agentic baseline targets these failures with structured guidance, verification, retries, and auto-repair.
- Experimental Setup: 15 representative language models were evaluated on Chat2Workflow using average results across three independent runs on Dify 1.9.2.The evaluation included four closed-source and eleven open-source models.
- Main Results: All models achieve lower resolve rates than pass rates, showing that syntactic validity does not guarantee successful workflow execution.GLM-4.6 has the largest average disparity at 27.43%, reaching 47.48% in Education; the highest resolve rate is 60.20%.
- Main Results: 60.20% is the highest absolute resolve rate, achieved by Gemini-3-Pro-Preview, while GLM-4.7 reaches 46.55% as the best open-source model.The paper reports that even the strongest models remain below a reliable, deployable planner threshold.
- Main Results: Scale, alignment, and reasoning affect performance differently: larger models improve steadily, formatting alignment alone is insufficient, and reasoning variants generally perform better.For example, GLM-4.6 has a higher Developer pass rate than GLM-4.7 but a lower resolve rate, while Qwen-3-235B-A22B outperforms the larger Qwen-3-Coder-480B-A35B-Instruct in resolve rate.
- Requirement Changes: Most models’ workflow quality declines as interaction rounds increase, although the degradation curve later flattens for workflows that survive early modifications.The fourth round is omitted because of limited sample size, and GPT-5.2’s slight third-round improvement reflects a changing denominator.
- Agentic Baseline: The error-driven agentic baseline replaces zero-shot prompting with SKILL-based guidance, variable summaries, verification, retries, and targeted auto-repair.Its repair loop addresses redundant code fences, JSON decoding failures, topological sorting violations, and node-related errors.
4 Related work
Prior work equips LLM agents with tools, collaboration frameworks, APIs, and prompting methods, while workflows emphasize transparent, reliable execution through structured plans. The supplied discussion also highlights persistent instability in planning and reasoning across domains.
- LLM agents use general tools, collaborative frameworks, and extensive API repositories to solve complex real-world problems.
- Prompt-based methods have been shown to improve agent performance, but planning and reasoning paths can hinder stable execution and result reproduction.
- Workflows decompose high-level tasks into executable atomic steps organized by formal logical relationships.
- Structured workflows provide transparent reasoning paths and use established constraints to support complex task logic.
- Workflow research has shifted from labor-intensive manual design toward automated generation using LLMs and iterative synthesis.
5 Conclusion
Chat2Workflow benchmarks whether LLMs can generate deployable visual workflows from natural language and finds that current models remain fragile under structural constraints and changing requirements. It offers a realistic testbed and target for research connecting language-model workflow design with industrial deployment.
- Chat2Workflow evaluates LLMs’ ability to generate deployable visual workflows from natural language.
- Current models remain fragile under structural constraints and requirement changes, even when augmented with an agentic framework.
- The benchmark provides a realistic testbed and concrete target for future research on automated workflow engineering.
Limitations
Chat2Workflow’s manually verified dataset, simplified node interfaces, and limited node vocabulary constrain how fully it represents complex industrial workflows and deployment requirements.
- The dataset may not cover the near-infinite variety of logic found in complex industrial business processes.
- Simplified node interfaces may fail to capture intricate parameter configurations required in some real-world deployments.
- The current system includes only 20 high-frequency node types, leaving many valuable tools unrepresented.
A Detailed Dataset Description
The dataset spans six workflow domains covering knowledge work, preprocessing, enterprise operations, software development, education, and directly usable content generation. Each domain emphasizes a distinct input-output or process objective.
- Research: Research tasks obtain, understand, and integrate knowledge from unstructured papers, books, or research topics.
- Document: Document tasks transform files, images, and tables into structured results that are computable, reusable, and processable downstream.
- Enterprise: Enterprise tasks automate, standardize, and support decisions in complex business processes using mostly internal or business-related documents.
- Developer: Developer tasks address software development, system design, and technical understanding to improve engineering efficiency.
- Education: Education tasks cover teaching and learning from planning and content generation through assessment and feedback.
- AIGC: AIGC tasks generate complete, publishable, and consumable multimodal content from often abstract or brief instructions.
B Experimental Settings
The experimental setup prepares Dify and Coze-compatible workflow generation with standardized reasoning and a broad node-type knowledge base.
- Dify workflows are represented in YAML, with a rule-based framework converting agents’ JSON outputs into valid YAML.Common models and required Dify extensions are prepared to support runnable and reproducible workflows.
- The setup uniformly uses chain-of-thought reasoning and supports node types spanning language models, control flow, extraction, media, search, and visualization.The listed knowledge base includes nodes such as LLM, If-Else, Iteration, Text to Image, Google Search, and Echarts.
C Instructions for the Studyplanner Task
The Studyplanner task evolves from extracting learning requirements and producing a study plan to generating a complete, chapter-structured tutorial through iterative content generation.
- Round 1: Round 1 extracts four learning fields from a user instruction and produces a detailed Markdown self-study improvement plan.The fields are learning area, goals, preferences, and average learning duration.
- Round 2: Round 2 extends the workflow to generate a full tutorial by first creating a syllabus and then iteratively generating chapter knowledge points.The final tutorial is integrated through template conversion and should include examples, pros and cons, and precautions.
- Round 2: The task instructions explicitly describe Round 2 as a modification built on the existing workflow foundation.
D More Related Work on Automated Workflow Evaluation
The paper motivates automated workflow evaluation by distinguishing abstract workflow assessment from validation of concrete designs and execution results across evolving instructions.
- Prior studies evaluate workflow format validity, tool consistency, or fine-grained planning, but the paper identifies limitations in existing automated evaluation approaches.
- Workflow validity evaluation: The Dify evaluator judges workflow validity from selected nodes, design principles, the parsed workflow JSON, and mandatory ground-truth nodes.Its instructions prohibit assumptions, inferred nodes, and external knowledge.
- The evaluation prompts are presented as dedicated procedures for assessing pass rate and resolve rate.
- Execution quality evaluation: A separate evaluator judges whether workflow execution results satisfy the requirements of the current round instruction using all available information.
- Execution quality evaluation: Execution evaluation receives historical instructions whose rounds may add, modify, or refine workflow behavior relative to earlier rounds.
- Figures 10 and 11 show GPT-5.2’s second-round Studyplanner workflows on Dify and Coze, respectively.