Source-linked AI summary
FlowAgent: Achieving Compliance and Flexibility for Workflow Agents
Yuchen Shi, Siqi Cai, Zihan Xu, Yuei Qin, Gang Li, Hang Shao, Jiawei Chen, Deqing Yang, Ke Li, Xing Sun
TL;DR
Rule-based workflow agents can restrict LLM flexibility, while prompt-based agents can fall short on procedural compliance. FlowAgent combines PDL with controllers and evaluates agents on compliance and out-of-workflow behavior; experiments on three datasets report strengths in both.
Problem
Rule-based methods restrict LLM action space, while prompt-based methods can weaken enforcement of workflow procedures.
Source: §1 · p. 1Method
FlowAgent combines the hybrid natural-language-and-code PDL workflow representation with controllers that supervise agent decisions.
Source: §1 · p. 2Results
Experiments on three datasets show that FlowAgent adheres to workflows and effectively manages out-of-workflow queries.
Source: §7 · p. 8Takeaways & Limitations
The results support FlowAgent's dual strengths in procedural compliance and flexibility when handling out-of-workflow queries.
Source: §7 · p. 8Evaluation uses artificially defined, manually constructed workflows and simulated out-of-workflow interactions, so real-world applicability remains to be tested across broader authentic user demands.
Source: §8 · p. 9
Abstract
from arXiv · showhide
The integration of workflows with large language models (LLMs) enables LLM-based agents to execute predefined procedures, enhancing automation in real-world applications. Traditional rule-based methods tend to limit the inherent flexibility of LLMs, as their predefined execution paths restrict the models' action space, particularly when the unexpected, out-of-workflow (OOW) queries are encountered. Conversely, prompt-based methods allow LLMs to fully control the flow, which can lead to diminished enforcement of procedural compliance. To address these challenges, we introduce FlowAgent, a novel agent framework designed to maintain both compliance and flexibility. We propose the Procedure Description Language (PDL), which combines the adaptability of natural language with the precision of code to formulate workflows. Building on PDL, we develop a comprehensive framework that empowers LLMs to manage OOW queries effectively, while keeping the execution path under the supervision of a set of controllers. Additionally, we present a new evaluation methodology to rigorously assess an LLM agent's ability to handle OOW scenarios, going beyond routine flow compliance tested in existing benchmarks. Experiments on three datasets demonstrate that FlowAgent not only adheres to workflows but also effectively manages OOW queries, highlighting its dual strengths in compliance and flexibility. The code is available at https://github.com/Lightblues/FlowAgent.
1 Introduction
Workflow agents use LLMs to help users complete procedural tasks, but existing approaches trade flexibility against compliance. FlowAgent pairs PDL with supervised agent decisions to address both challenges and evaluate behavior beyond predefined workflows.
1 Introduction
- Rule-based workflows offer high compliance but constrain LLM flexibility, while prompt-based workflows allow autonomous flow management but risk procedural violations.Adding flexible features to rule-based systems can require many extra transitions; prompt-based errors can include reporting a booking that never occurred.Source: §1 · p. 1
- FlowAgent introduces PDL, combining natural-language flexibility with coding precision, and controllers that govern agent behavior according to PDL-defined nodes.The framework allows autonomous decisions subject to monitoring and legal constraints.Source: §1 · p. 2
- In a hospital appointment example, FlowAgent re-invokes the hospital-check API when a user switches hospitals and blocks a booking result until prerequisites are met.The controllers prevent the agent from reporting a result before the required booking API has run.Source: §1 · p. 2
- The paper evaluates compliance and flexibility on three datasets and augments existing datasets with a benchmark for out-of-workflow queries.The benchmark is intended to assess performance beyond routine flow compliance.Source: §1 · p. 2
2 Related Work
Prior work motivates evaluating LLM dialogue systems by task success and treating workflow execution as a distinct agent paradigm. FlowAgent targets the balance between process control and adaptability.
2.1 LLM-Driven Conversational Systems
- LLM-driven dialogue research motivates task-success metrics rather than relying only on modular accuracy.The related work connects this shift to workflow-guided interactions and the need to assess procedural compliance alongside adaptive flexibility.Source: §2.1 · p. 3
2.2 Agentic Workflow Architectures
- Agentic workflow research distinguishes generating procedures from executing within predefined structures, and FlowAgent focuses on the latter.Within workflow execution, rule-based systems can lack flexibility, while prompt-based systems may deviate from intended procedures.Source: §2.2 · p. 3
3 Preliminary and Background
The paper models workflows as structured task processes and workflow agents as sequential decision-makers interacting with users and tools. It distinguishes explicit rule-based control from prompt-based autonomous dialogue management.
3.1 Workflow
- A workflow is a directed acyclic graph whose nodes represent atomic operations and whose edges encode temporal and dependency relationships.The graph specifies the progression of a task, such as querying preferences, retrieving appointment slots, confirming details, and completing a booking.Source: §3.1 · p. 3
3.2 Workflow Agent
- A workflow agent makes sequential decisions in a user-and-tool environment, modeled as states, actions, and feedback over time.The history of actions and observations guides decisions, while the workflow provides guidance for the agent's actions.Source: §3.2 · p. 3
- Rule-based agents use explicitly programmed procedures with limited information and action space, whereas prompt-based agents let one language model manage decisions and dialogue.Rule-based systems program node transitions; prompt-based systems represent the workflow as text, with its graph structure implicitly encoded.Source: §3.2 · p. 4
4 Methodology
FlowAgent combines a workflow-description language with an execution framework that controls agent behavior. PDL structures workflow information and procedure logic, while controllers guide and constrain decisions.
4 Methodology
- FlowAgent is an execution framework integrated with PDL that uses controllers to promote reliable action execution without removing LLM autonomy.The methodology presents PDL and FlowAgent as complementary components for workflow representation and behavioral control.Source: §4 · p. 4, §4.2 · p. 4
4.1 PDL Syntax
- PDL contains workflow meta information, API and ANSWER node definitions, and procedural logic expressed through natural language and pseudocode.Its node preconditions encode dependencies, such as requiring check_hospital before check_department; the hybrid representation combines natural-language clarity with code accuracy.Source: §4.1 · p. 4
4.2 FLOWAGENT Architecture
- FlowAgent uses pre-decision controllers to guide choices and post-decision controllers to validate proposed actions, combining soft guidance with hard constraints.Pre-decision controllers can identify unreachable nodes from the dependency graph; post-decision checks can deny a transition when a prerequisite is incomplete.Source: §4.2 · pp. 4–5
- The framework includes modular controllers that address node dependencies, repeated API calls, and conversation length.The node-dependency controller can operate before or after a decision: it identifies inaccessible nodes in advance and validates proposed transitions afterward.Source: §4.2 · p. 5
5 Evaluation and Data
The evaluation measures workflow compliance at both turn and session levels, then assesses flexibility in out-of-workflow scenarios. It uses datasets built for workflow, user-profile, and conversation evaluations.
5.1 Compliance Evaluation
- Turn-level compliance uses reference-session Pass Rate and evaluates tool selection and parameter filling with Precision, Recall, and F1-score.Session-level evaluation simulates user interactions, measures primary-objective Success Rate, and tracks initiated subtasks as Task Progress.Source: §5.1 · pp. 5–6
5.2 Flexibility Evaluation
- Flexibility evaluation covers intent switching, procedure jumping, and irrelevant answers, measuring immediate turn-level responses and overall session performance.The framework applies the compliance metrics to sessions and turns that include out-of-workflow interventions.Source: §5.2 · p. 6
5.3 Data
- The test datasets use SGD, STAR, and in-house business-related data, with workflows, user profiles, and conversations for evaluation.The passage points to Tables 1 and 2 for dataset statistics and comparisons, but their contents are not provided here.Source: §5.3 · pp. 6–7
6 Experiments
The experiments compare FLOWAGENT with ReAct variants across workflow formats, assessing compliance in routine tasks, flexibility under OOW queries, and the contribution of its controllers. Results favor FLOWAGENT in session- and turn-level evaluations, while controller ablations show performance declines when controllers are removed.
6.1 Experimental Setup
- The setup compares FLOWAGENT with ReAct using natural-language, code, and flowchart workflow representations.Session-level evaluation uses GPT-4o-mini for user simulation and GPT-4-Turbo for judgment; GPT-4o and Qwen2-72B are the demonstrated bots.Source: §6.1 · p. 7
6.2 Session-level Experimental Results
- FLOWAGENT outperforms the three baselines on session-level Success Rate, Task Progress, and Tool F1.The evaluation compares ReAct using natural-language, code, and flowchart workflow representations.Source: §6.2 · p. 7
- Across three datasets, FLOWAGENT has the best results under OOW scenarios and shows only a slight performance decline.Figure 5 visualizes Task Progress under different settings.Source: §6.2 · pp. 7–8, Figure 5 · p. 8
6.3 Turn-level Experimental Results
- In turn-level evaluation, FLOWAGENT achieves the best performance in both in-workflow and OOW settings.The comparison reports Qwen2-72B results; Figure 5(b) compares Success Rate across models and settings.Source: §6.3 · p. 8, Figure 5 · p. 8
6.4 Ablation Studies
- Removing either the post-decision controller or the pre-decision controller negatively affects performance, supporting the controllers’ role in compliance.Table 5 presents the ablation study; “-post-pre” removes the pre-decision controllers in addition to the post-decision controllers.Source: §6.4 · p. 8, Table 5 · p. 8
7 Conclusion
The paper proposes PDL and FLOWAGENT to support workflow execution with compliance and flexibility. Experiments on three datasets show strong compliance and robust handling of out-of-workflow queries.
7 Conclusion
- PDL represents workflows, while FLOWAGENT controls agent behavior to target compliance without significantly compromising flexibility.The paper also reviews existing LLM-based workflow methods in terms of these two capabilities.Source: §7 · p. 8
- The authors construct datasets from existing data and design evaluation methods for compliance and flexibility.Source: §7 · p. 8
- Experiments on three datasets demonstrate strong compliance and robust flexibility when handling out-of-workflow queries.Source: §7 · p. 8
8 Limitations
The evaluation is bounded by manually constructed PDL workflows and simulated user interactions. The authors identify automated workflow generation and broader testing with authentic user demands as areas not covered by this study.
Workflow Generation
- The study evaluates LLM performance in manually constructed PDL workflows and does not explore automated workflow generation.The authors call for future work on dynamic workflow synthesis for varying and complex user demands without manual intervention.Source: §8 · p. 9
Dialogue Diversity and Evaluation
- OOW performance is evaluated with simulated user interactions; real-world applicability depends on testing a broader spectrum of authentic user demands.Source: §8 · p. 9
Appendices
The appendices provide a hospital-appointment PDL example and an example user profile. Together, they illustrate workflow components and the kinds of user information used in the profile.
A.1 PDL Example
- The PDL appointment example specifies requests, responses, and node preconditions for hospital and department checks, appointment queries, and registration.The listed nodes include check_hospital, check_department, query_appointment, recommend_other_hospitals, and registration steps.Source: §A.1 · p. 11
- The example procedure checks hospital and department availability, queries appointment slots, and can seek alternatives at other hospitals.Source: §A.1 · pp. 11–12
A.2 User Profile Example
- The user profile example covers user details, conversational style, workflow-related needs, and a possible interactive pattern.The appendix describes these as profile attributes, style, requirements, and possible dialogue process, respectively.Source: §A.2 · p. 12
B.1 Conversation Example on STAR Dataset
The appendix illustrates a STAR conversation in which the user changes an appointment time, then documents implementation prompts and how evaluation data were constructed. These materials span user simulation, agent behavior, and workflow formats.
B.1 Conversation Example on STAR Dataset
- On STAR, the user asks to change the appointment time to 4 PM; the bot checks availability, reports it is available, and asks whether to book it.The example labels the change as an intent-switching/detail-switching OOW.Source: §B.1 · p. 13
C.1 Implementation Details
- GPT-series and Qwen2-series models were run at temperature 0.2 for stable output results.The specified GPT models were gpt-4o-2024-05-13, gpt-4o-mini-2024-07-18, and gpt-4-turbo-2024-04-09; Qwen2 inference used vllm.Source: §C.1 · p. 13
C.2 Prompts
- The FLOWAGENT prompt directs the agent to follow PDL-aligned steps, answer irrelevant questions conversationally, and collect enough parameter values before calling APIs.OOW intentions can be injected through an additional user-profile constraint and adjusted during the conversation.Source: §C.2 · pp. 13–14
D.1 Data Construction
- The datasets combine SGD, STAR, and an in-house collection; workflows were represented as natural language, code, flowcharts, and PDL.Turn-level sessions were constructed with GPT-4o and OOW queries inserted into reference sessions; session-level dialogues used LLM-simulated users, agents, and systems.Source: §D.1 · pp. 15–16
E.1 Turn-level Evaluation Results
The section presents turn-level results for Qwen2-72B and explains why the OOW Success Rate may be higher than the IW rate or left unreported.
E.1 Turn-level Evaluation Results
- OOW turns can have higher turn-level Success Rate than IW turns because they typically require fewer complex judgments or API calls.This is a stated possible explanation, not a reported universal outcome.Source: §E.1 · p. 16
- The OOW turn-level Success Rate is left blank because fewer API calls can make its direct calculation highly variable.Source: §E.1 · p. 16
- The table reports turn-level experimental results for Qwen2-72B.The caption identifies it as Table 6; no cell values are supplied here.Source: Table 6 · p. 17