Source-linked AI summary
Generating Workflow DAGs from Natural Language with Non-Reasoning LLMs
Anand Iyer, Bhanu Khetharpal, Srinivas Upadhya, Ramkumar Rajagopal
TL;DR
The paper studies how to translate natural-language contact-center routing rules into executable workflow DAGs despite dense branches, exactness requirements, and token-cost constraints. It diagnoses emission-density failures and uses an intermediate representation, deterministic compiler, and registry selector to reduce model emission burden, achieving strong validity and condition accuracy while substantially improving GPT-5.3-chat performance.
Problem
The paper addresses translating natural-language routing rules into executable JSON workflow DAGs, where dense branching, exact configuration, and interactive token budgets make monolithic generation difficult.
Method
The system diagnoses emission-density failures and moves combinatorial graph construction into a deterministic, target-agnostic compiler driven by an intermediate representation and learned registry selection.
Results
GPT-5.3-chat judge validity improves from 56.4 to 80.6 and condition accuracy from 56.5 to 82.2, while JSON validity reaches 99–100% across compiler cells.
Takeaways & Limitations
The approach lets non-reasoning models reach production-relevant operating points at roughly half the token cost and statistically match a reasoning model’s out-of-the-box quality on the targeted semantic metric.
Takeaways & Limitations
Decomposition relocates rather than removes the bottleneck: recurring-timer cases still miss exact conditions in approximately 40% of cells, and content-filling can retain node-dropping failures.
Abstract
from arXiv · showhide
This paper addresses the problem of translating natural-language routing rules written by business administrators into executable workflow graphs for enterprise contact centers. Each target is a directed acyclic graph (DAG) of conditional actions with parallel branches, hit-first fallback chains, and per-branch Boolean predicates, encoded in the JSON dialect of a commercial routing platform. We show that neuro-symbolic decomposition enables lower-cost, non-reasoning large language models to generate complex workflow DAGs at production-relevant quality without expensive extended-reasoning models. Our central diagnostic is an emission-density bottleneck: on a 635-rule benchmark of manufactured synthetic data, models select the correct graph nodes with high accuracy but increasingly misconfigure attributes and Boolean grouping as the number of interdependent nodes emitted in one pass grows. We therefore move combinatorial graph construction from the model into a deterministic compiler driven by a compact intermediate representation, with a learned registry-selection front end that focuses generation on relevant vocabulary. Across four models, the full system reaches approximately 89% LLM-judge validity, approximately 90% exact-match condition accuracy, and 99-100% valid JSON while using roughly half the per-rule prompt tokens of a monolithic prompt. On GPT-5.3-chat, the method improves judge validity by 24 percentage points and achieves statistical equivalence to a reasoning model's out-of-the-box quality, although an approximately 8-point frontier gap remains. We also present a deployment path and transferable lessons for structured-generation applications.
1 Introduction
The paper studies translating natural-language contact-center routing rules into executable workflow DAGs and identifies dense structured emission as the central failure mode. It addresses this bottleneck by moving combinatorial graph construction into a deterministic compiler while retaining LLM-based interpretation.
- Problem and significance: Natural-language routing rules must become JSON workflow DAGs whose nodes encode actions and whose edges encode conditional and hit-first execution order.The target includes predicates, parallel branches, and dependencies rather than a flat action list.
- Problem and significance: Production rules are difficult because they combine dense conditional branches, exact behavioral requirements, and interactive cost constraints.Rules may combine up to 15 conditional branches, while one branch can trigger several actions.
- Motivation: Few-shot exemplars drive accuracy, but globally inlining growing vocabularies and scenario-specific examples increases cost and dilutes attention.The architecture instead pays for only the vocabulary and exemplars each rule needs.
- Thesis: The emission-density bottleneck is that models select graph nodes accurately but increasingly misconfigure attributes and Boolean structure as node density rises.The observed degradation is tied to emitting dense interdependent structure, not to reading the rule.
- Contribution: The proposed remedy is architectural: a deterministic compiler handles combinatorial graph construction while the non-reasoning LLM handles interpretation.The paper reports statistically equivalent judge validity to a strong reasoning model at a pre-specified ±5-point margin.
- Contributions: The paper presents a diagnostic, a gap-bridging result, and an enterprise workflow-DAG application built around this division of labor.The diagnostic generalizes as a lens for NL→graph tasks, while the application instantiates the approach on a commercial routing platform.
2 Task and Domain
The task uses a versioned registry and a 635-rule dense benchmark to generate workflow JSON from natural-language rules. The method combines learned registry selection with an intermediate representation and deterministic compilation to reduce emission burden and guarantee structural validity.
- Registry and task: The platform registry is a versioned vocabulary of event types, action types, conditions, and example JSONs used to retrieve relevant few-shot context.Adding a capability or scenario exemplar requires a registry entry rather than a code change.
- Input and output: Workflow output contains a trigger, action nodes, per-action Boolean predicates, and dependency edges for hit-first, parallel, and residual branches.The main challenge is assigning conditional structure to actions and relating branches, not identifying the trigger or action set.
- Benchmark: The benchmark contains 635 programmatically templated rules across seven structural families, with 1–15 conditional branches and more than 15 actions in dense cases.Templating supplies exact ground-truth workflows for deterministic accuracy evaluation.
- Configurations: The evaluation compares monolithic generation, selection followed by direct JSON generation, IR-only compilation, and the combined Selector+IR pipeline.The configurations isolate the effects of vocabulary selection and deterministic compilation.
- Intermediate representation and compiler: The IR pipeline maps natural language to low-syntax YAML and then deterministically compiles it into target workflow JSON.The compiler performs cross-products, hit-first chaining, and list-valued condition expansions while remaining target-schema agnostic.
- Diagnostic: Node-set accuracy is substantially higher than node-exact accuracy, showing that residual errors concentrate on attributes and parameters rather than selecting the wrong actions.Across configurations, node-set accuracy reaches 98.4–99.8% for GPT-5.2 and 99.5–100% for Opus 4.6, with lower ranges for GPT-4.1 and GPT-5.3-chat.
- Registry selection: Selection retrieves relevant events, actions, conditions, and examples before IR generation, keeping prompts focused as registry size grows.Similarity-based candidate generation is proposed to preserve high recall and bound selector input independently of vocabulary size.
- System principle: The combined pipeline feeds selected registry entries into IR generation and deterministic compilation, combining scaling with validity by construction.This division moves combinatorial emission from the model into the compiler.
5 Evaluation
Across 635 rules and four models, compiler-based decomposition improves dense workflow generation, especially for GPT-5.3-chat, while preserving structural validity and reducing token use. The evaluation also shows that gains mainly remove completeness and syntax errors; ordering and Boolean-logic errors remain, and a frontier gap persists against the strongest reasoning configuration.
- Headline results: 98.4–100%: stronger models achieve near-saturated node-set accuracy, while node-exact accuracy remains lower because attributes are mis-emitted.For GPT-5.3-chat, Selector+IR raises node-set accuracy by 10.2 points and node-exact accuracy by 20.3 points relative to monolithic generation.
- The emission-density curve: ∼0%: GPT-5.3-chat’s monolithic exact-conditions accuracy beyond 15 branch-grid pairs, versus above ∼55% for Selector+IR through the 13–15 band.All configurations decline as branch-grid density rises, but compiler configurations degrade more gracefully.
- Validity by construction: 99–100%: compiler cells achieve valid JSON, while IR-only produces zero uncompilable output across all four models.Selector-induced malformed IR leaves exactly one hard failure across 5,080 pipeline generations, or 0.02%.
- Headline results: 24.3 points: GPT-5.3-chat’s Selector+IR pipeline improves judge validity over monolithic generation, from 56.4% to 80.6%.Condition accuracy rises from 56.5% to 82.2% in the same comparison.
- Token efficiency: 0.44–0.50×: GPT-5.3-chat + Selector+IR uses this fraction of the tokens consumed by reasoning models’ monolithic runs while matching their monolithic quality.IR reduces completion size, and selection trims the registry context supplied to generation.
- What structure fixes, and what it does not: 49.5%: ordering errors are the largest failure category and barely change, while pipelines roughly halve incompleteness and De Morgan/exclusion errors.Externalizing emission converts omission errors into residual mis-ordering errors; cross-variable AND/OR flips also remain flat.
6 Deployment Status and Path
The system extends workflow authoring from templatized rules to text rules while scaling registry growth without expanding prompts. Its deployment path combines learned selection, deterministic compilation, and independent validation practices.
- Deployment Status and Path: The architecture accepts text rules and scales to a growing registry without an ever-expanding prompt.It is evaluated offline at scale on 635 rules across four models and targets the current system’s production cost envelope.
- Deployment Status and Path: New registry entries add events, actions, conditions, and disambiguation metadata while leaving the compiler and prompts unchanged.Few-shot exemplars are single-sourced from the registry, and similarity retrieval keeps selection high-recall and inexpensive as vocabulary expands.
- Deployment Status and Path: Blind author labeling agreed with the judge on 91% of sampled verdicts, with combined Cohen’s κ = 0.82.The sample contained 50 judge-PASS and 50 judge-FAIL items, and the conditions metric independently corroborated the evaluation.
- Deployment Status and Path: The architecture is domain-agnostic and is intended to transfer to business-process, ETL, agent-tool, and CI/build graphs.The transferable pattern is LLM → IR → deterministic compiler with learned selection and similarity prefiltering.
7 Lessons Learned
The paper identifies evaluation and decomposition practices that matter for structured generation. Reliable gains require aligning evaluation conventions and reducing what the model must emit, rather than merely splitting generation into stages.
- Lessons Learned: The evaluation practices include treating the scorer as fallible code and using greedy generation for paired statistics.These practices are identified as transferable lessons for measuring structured-generation systems.
- Lessons Learned: A shared judge rulebook prevents convention drift from falsely failing outputs that use equivalent Boolean forms.The paper fixes one rulebook across systems and rechecks anomalously poor configurations.
- Lessons Learned: Plan→expand and codelocked scaffolds failed because node dropping moved to content filling rather than disappearing.The supported remedy is to reduce emitted structure through an IR, list-valued conditions, or selection.
8 Conclusion
The paper addresses enterprise workflow generation with affordable non-reasoning models by moving dense graph construction into a deterministic compiler. The resulting system reaches production-relevant quality and statistically matches a reasoning model in a scoped comparison, while retaining a frontier gap.
- Conclusion: Commodity LLMs fail mainly when emitting dense structure, so an IR-driven deterministic compiler handles combinatorial graph construction.A learned registry selector with similarity prefiltering focuses the model on relevant vocabulary.
- Conclusion: On GPT-5.3-chat, the method statistically matches a reasoning model’s out-of-the-box quality, while an approximately 8-point frontier gap persists.The comparison is framed as an equivalence-test gap-bridging result rather than complete parity.
Data Availability
The evaluation data and platform vocabulary are proprietary and cannot be released because of confidentiality agreements. The paper reports aggregate metrics and dataset characteristics while providing enough methodological detail for reimplementation on comparable targets.
- Data Availability: The evaluation data and platform vocabulary are proprietary and governed by confidentiality agreements.Examples and schemas are not released.
- Data Availability: The paper reports aggregate metrics and dataset characteristics instead of releasing examples or schemas.Reported characteristics include benchmark size, scenario-family structure, and condition-density distribution.
- Data Availability: The methods are described in enough detail to reimplement on any comparable structured-workflow target.
A Self-preference bias — full argument (supports Sec. 5.1)
The paper treats self-preference bias as a legitimate concern because the judge and one generator share a model family, but argues that the primary bridge comparison avoids this confound and that remaining bias makes the reported claims conservative.
- Self-preference bias is a legitimate concern because Opus 4.6 is both a generator and the LLM judge.
- The primary bridge comparison uses non-Opus generations, so both are evaluated by the same Opus judge and self-preference cancels.The comparison is GPT-5.3-chat + Selector+IR versus GPT-5.2 monolithic.
- Bias in comparisons involving Opus inflates both its monolithic baseline and its best reasoning result, working against the paper’s bridge claim.
- Node-set accuracy measures correct action types, whereas node-exact accuracy additionally requires correct parameters.
- Node-set accuracy reaches 98–100% for GPT-5.2 and Opus 4.6, while residual error remains concentrated in parameters on correctly selected nodes.For GPT-5.3-chat, node-set accuracy moves from 82.4 to 92.6.
C Compile-failure breakdown (supports Sec. 5.5)
The decomposition nearly eliminates uncompilable output, while paired evaluation shows a large GPT-5.3-chat improvement and identifies scorer strictness and single-run variance as important interpretive boundaries.
- IR-only produces zero uncompilable output across all four models.Adding selection causes rare malformed IR, but nearly all such cases repair to valid JSON.
- Only one hard failure occurs across 5,080 pipeline generations, a 0.02% compile-failure rate.
- Exact-condition scores for Opus decline under Selector→JSON and IR-only because canonical compiler nesting differs from ground-truth structure despite logical equivalence.The judge forgives this scorer-strictness effect.
- Equivalence to Opus is established only at a ±7-point margin, not the pre-specified ±5-point margin used for strict parity claims.
- Single-run greedy generation leaves decode variance, prompt-paraphrase sensitivity, and few-shot-ordering sensitivity unquantified.These are identified as the main residual threat to point estimates.
E Error-taxonomy coding method and per-category numbers (supports Sec. 5.9)
The paper codes judge rationales to separate decomposition-sensitive completeness and syntax errors from persistent logic and topology errors, while emphasizing paired inference and scorer validation.
- The error taxonomy covers 2,348 failing cells and separates completeness/syntax errors from logic/topology errors.
- Pipelines reduce incomplete errors from 12.6 to 5.7 per 100 prompts, De Morgan errors from 8.3 to 3.7, and JSON errors from 1.2 to 0.0.
- Cross-variable AND/OR connective flips remain essentially flat, while Ordering remains the largest and most stubborn residual error.
- The conditions scorer resolves gates through timer cascades only across success edges, preventing fallback branches from inheriting a gate.The scorer is validated against the judge and unit tests.
G Full per-config token breakdown (supports Sec. 5.7)
The token analysis explains why the full selector-plus-IR pipeline is cheaper despite multiple calls: compact IR reduces completions, while selection avoids repeatedly sending the full registry.
- Selector→JSON costs approximately 1.08–1.12× the monolithic baseline because it performs two full-size calls.
- Selector+IR reaches approximately 0.56× the monolithic token cost by combining compact IR with a selected registry subset.
- The selector trims the monolithic approximately 19K-token registry prompt to a compact approximately 2.2K-token menu before assembly.