PaperlayerView exact revision on arXiv

Source-linked AI summary

Natural-Language Workflows Are Not Software Yet: Artifact-Driven Compilation for Reliable Agent Execution

Xiangzhe Xu, Hanxi Guo, Guangyu Shen, Siyuan Cheng, Xiangyu Zhang

arXiv:2608.21341v1cs.SE

TL;DR

Natural-language workflows are attractive as reusable agent procedures, but implicit dependencies and context-heavy control flow undermine reliable execution. Artic compiles them into artifact-driven workflows with explicit state and control structure, optimizes enforcement burden, and validates faithfulness through local obligations and scenario-based dry runs. Across 488 instances from 11 real-world workflows, it improves task resolve rate by 28 percentage points over the original text workflow and increases consistency across evaluation settings.

  • Problem

    Implicit data dependencies and complex, context-heavy control flow make natural-language workflows difficult for agents to execute faithfully.

  • Method

    Artic compiles natural-language workflows into artifact-driven workflows with explicit artifact dependencies and control transfers, using constrained optimization and faithfulness validation.

  • Results

    28 percentage points: Artic improves task resolve rate over the original text workflow across 488 problem instances from 11 real-world domain workflows.

  • Takeaways & Limitations

    Artifact-driven compilation improves workflow enforcement and consistency across realistic domain workflows and execution settings.

Abstract

from arXiv · show

Natural-language workflows offer a software-like interface for agents: domain experts can write reusable procedures, and agents can execute them as instructions. This promise is not yet reliable. Workflow descriptions often leave data dependencies implicit, so the executor must infer which prior results a step should use; agents can also fail to follow long or branching instructions under context pressure. We propose Artic, an artifact-driven workflow compiler that transforms a natural-language workflow into an artifact-driven workflow in which each step declares the artifacts it reads and writes, constraints gate produced artifacts, and explicit control transfers route execution. This representation exposes the enforcement burden placed on agent execution, allowing the compiler to identify steps that depend on too much state or contain difficult control logic and refine them through constrained optimization. To validate the LLM-assisted transformation, Artic decomposes faithfulness checking into local obligations and uses scenario-based dry runs to test whether compiled workflow regions conform to the source workflow. We evaluate Artic on 488 problem instances from 11 real-world domain workflows; it improves task resolve rate by 28 percentage points over the original text workflow. We also show that workflows compiled by Artic are 32 and 56 percentage points more consistent in cross-model and repeated-execution setups, respectively.

I. INTRODUCTION

Natural-language workflows provide a reusable, software-like interface for agents, but implicit data dependencies and complex control flow make faithful execution unreliable. ARTIC compiles these workflows into artifact-driven forms with explicit state, control transfers, optimization, and faithfulness validation.

  • Motivation: Natural-language workflows let domain experts specify reusable procedures for agent execution, but they do not yet provide software-like reliability.Agents may execute instructions variably rather than following them exactly.
  • Motivation: Implicit data dependencies force agents to infer which prior results a step should use from the accumulated context.This resembles a program whose intermediate results are exposed as global variables rather than declared dependencies.
  • Motivation: Branches and heavy intermediate context increase the risk of skipped steps, incorrect control transfers, and lost information during context compaction.The motivating example shows failures in selecting relevant medical information and interpreting an “or” condition as “and.”
  • Artifact-driven execution: ARTIC represents workflows as steps that explicitly read and write artifacts, use constraints to gate progress, and route execution through explicit control transfers.This semi-structured representation preserves natural-language actions while making workflow state and execution gates inspectable.
  • Compilation: ARTIC uses constrained optimization to reduce context pressure and decision complexity, identifying workflow regions that are difficult for agents to enforce faithfully.Programmatic analyses of explicit artifacts and control transfers provide feedback to the LLM during compilation.
  • Validation: ARTIC validates compiler output through local faithfulness obligations and scenario-based dry runs before accepting the transformed workflow.The validator decomposes whole-workflow checking into smaller obligations and tests compiled regions on concrete cases.
  • Evaluation: 28 percentage points: ARTIC improves task resolve rate over the original text workflow on 488 instances from 11 real-world domain workflows.Compiled workflows are also 32 percentage points more consistent across models and 56 percentage points more consistent across repeated executions.

III. METHOD

The compiler drafts artifact-driven workflows, analyzes their structure to reduce execution burden, and validates that the compiled workflow preserves the source procedure. The representation formalizes artifact transformations, control flow, and artifact-state semantics for executable workflows.

  • Compiler pipeline: The compiler asks an LLM to draft an artifact-driven workflow, then uses program analysis to identify excessive context, live-artifact, or control-flow burden.The draft isolates each step’s needed context and decomposes overloaded steps.
  • Compiler pipeline: Faithfulness validation combines static checks, local inductive obligations, and dry-run simulations before accepting the compiled workflow.Diagnostics can feed back into the drafting stage when validation fails.
  • Workflow language: Artifact-driven workflows represent procedures as transformations over named artifacts, with agent, sequential, branch, and loop statements as core structures.Each agent step performs a local natural-language instruction over an artifact transformation.
  • Workflow semantics: Artifact states map each artifact identifier to its current data and semantic constraints, while reads restrict the store and updates replace produced entries.The semantics therefore track both artifact contents and constraints during execution.
  • Workflow semantics: Big-step semantics define how a statement transforms one artifact state into another, with agent execution producing updates that must pass constraint checks.Sequential composition and conditional execution operate over the resulting artifact states.

B. Optimization Objectives

Artic optimizes compiled workflows by exposing control-flow, context, compaction, and artifact-materialization burdens. Its objectives favor decomposing difficult instructions while avoiding unnecessary intermediate artifacts.

  • Optimization strategy: The compiler decomposes complex instructions into multiple steps so each step retains only enforceable instructions.This optimization targets the execution burden created by excessive context or control structure.
  • Control-flow complexity: Control-flow burden estimates the latent workflow’s branching complexity using cyclomatic complexity over a fine-grained artifact-driven representation.The representation makes prompt structure analyzable through program analysis.
  • Context pressure: Context burden estimates the context length carried by primitive bodies or guards and combines pressure across sequential, branch, and loop structures.Branches use the larger route because only one route executes, while loops scale by estimated iteration count.
  • Compaction pressure: Compaction burden increases when many live artifacts cross a compaction point and must be preserved for future execution.The compiler uses live-variable analysis and the worst internal compaction point to estimate this burden.
  • Artifact cost: Artifact cost measures the overhead of materializing intermediate results, so optimization reduces pressure without introducing unnecessary artifacts.The objective is based on the estimated sizes of materialized artifacts.

C. Faithfulness Validation

ARTIC validates LLM-generated artifact-driven workflows by decomposing faithfulness into local obligations and comparing source and compiled executions on concrete scenarios.

  • Validation decomposes whole-workflow faithfulness into smaller obligations aligned with the artifact-driven language structure.Each local obligation is intended to be simpler and more reliably validated than checking the workflow at once.
  • Coverage-guided scenarios vary predicates and expected routes or artifacts, enabling concrete tests of source–compiled consistency.Independent dry runs of the source and compiled workflows are compared on each scenario.
  • ARTIC uses a deterministic orchestrator, runtime artifact management, and subagents that produce parsed artifacts during execution.The executable package separates control transfer, local natural-language instructions, artifact paths, versions, and subagent invocation.
  • The evaluation covers 11 domains from SOP-Bench and χ-Bench, comparing textual execution, skill rewriting, code generation, and workflow baselines.Instances are resolved only when the prescribed procedure is followed and the correct domain decision is made.
  • The validator checks that intermediate artifacts form valid conceptual boundaries before composing prefix and continuation validations.An intermediate artifact must contain exactly the state on which the remaining source workflow may rely.

B. RQ1: Does Artifact-Driven Compilation Improve Workflow Enforcement?

Across executor models and workflow representations, ARTIC’s compiled workflows achieve the strongest task-resolution performance and substantially greater execution consistency than text workflows.

  • ARTIC compares compiled artifact-driven workflows against textual execution, skill rewriting, direct code generation, and natural-language-to-workflow baselines.Table I reports task resolve rates for the workflow representations, while Table II compares ARTIC with existing workflow-conversion systems.
  • 85% on GLM-4.7-Flash, 82% on GPT-OSS-120B, and 85% on Qwen3-235B are ARTIC’s average task resolve rates across executor models.These are the best average task resolve rates for every evaluated executor model.
  • 72% to 88% is the compiled workflow’s cross-model performance range on Medical, versus 28% to 60% for the text workflow.The compiled workflow’s range is half as wide as the text workflow’s range.
  • 80% of Medical instances are consistent across six executor models with ARTIC, compared with 48% for the text baseline.An instance is consistent when all six models either pass or fail it under the same representation.
  • At k = 10 repeated runs, about 72% of compiled-workflow cases pass versus 16% for text workflows.From k = 1 to k = 10, compiled workflows degrade by 16 percentage points, compared with 35 points for text workflows.
  • Compiled workflows reduce average per-agent input tokens by 63% and output tokens by 50% compared with text workflows.The reduction reflects decomposition of challenging steps into smaller subagents across the Table I samples.

D. RQ3: What Is the Compilation Cost?

Compilation is an offline cost paid once per workflow, with studied SOP-Bench domains showing sub-five-minute median first-package time and under-$3 average cost. The compiler also uses dry-run feedback and other components to identify and correct transformation errors while preserving workflow constraints.

  • Under 5 minutes is the median first-package compilation time with GLM-5 for the studied SOP-Bench domains.The average compilation cost is under $3.
  • Optimization contributes the most in ablations, while dry-run feedback and validation also contribute meaningfully to performance.Observed component effects range from 8 to 32 percentage points; optimization decomposes steps with excessive context or decision burden.
  • Dry-run simulation compares source and compiled workflows without oracle labels and localizes disagreements to specific predicates.A referral-source case exposed a compiled decision that checked for a referral message but not a qualifying source.
  • Artic targets enforcement rather than arbitrary outcome optimization by making intermediate artifacts, control transfers, and validation obligations explicit.The generated workflow must preserve the procedure intended by domain experts.

A. Faithfulness Validation Formal Sketch

The formal validation sketch treats compilation as observational refinement rather than syntactic equivalence. It validates sequential, branching, and looping decompositions by composing locally validated subworkflows under consistent artifact-state observations.

  • Validation checks whether compiled workflow behavior preserves observable artifact behavior of the source under dry-run simulation.The goal is not syntactic equivalence between source and compiled representations.
  • Observable correspondence compares final outputs, route decisions, and intermediate artifacts whose values affect later observable behavior.Surface wording may differ when decisions, required fields, and constraint-relevant content remain equivalent.
  • The validation relation is one-directional: compiled observable behavior must be allowed by the source, without reproducing every interpretation of underspecified text.Runtime scratch artifacts and repair feedback may differ when they do not affect observable behavior.
  • The dry-run judgment maps a workflow and input artifact store to an output store produced by a policy rollout.The notation records the artifact effects of executing the workflow under policy π.
  • Scenario validation generates representative input stores, runs source and compiled workflows, and checks whether their resulting stores satisfy observational correspondence.Predicate coverage guides corner-case selection.
  • Sequential, branch, and loop induction rules compose recursively validated refinements into a refinement of the original natural-language region.Branch validation requires guard consistency, while loop validation uses an operational iteration relation.

APPENDIX B IMPLEMENTATION DETAILS

The runtime lowers compiled workflows into deterministic packages with a step interface, artifact checks, explicit routing, versioned artifacts, and retry handling. Validators check generated modules before execution.

  • The deterministic orchestrator invokes one step module at a time, checks its artifacts, and follows the returned control route.The per-step interface separates artifact production from artifact validation.
  • Each step’s exec_step transforms declared input artifacts into declared outputs, while check_artifact enforces schema and quality gates.The orchestrator continues only after the produced artifacts pass validation.
  • The lowering validator checks syntax, function signatures, output types, routing targets, and undeclared reads.Program analysis and model-assisted prompt/entity analysis detect fields referenced outside the step interface.
  • Artifact versioning preserves prior committed values when retries or loops produce replacements.Stable names, versions, and metadata support diagnostics and resumption from the latest valid state.
  • Failed quality gates trigger feedback and step reinvocation before artifact commitment.Runtime checks preserve retry feedback, prevent silent commits, and require selected routes to exist in the workflow.

APPENDIX C ADDITIONAL EVALUATION RESULTS

Additional evaluation examines repeated execution, environmental robustness, context pressure, ablations, compiler-model stability, and compilation overhead. The results attribute substantial performance differences to optimization while showing stable generated-workflow performance across compiler models.

  • Pass^k measures the probability that all k repeated executions of an instance succeed.The repeated-run study executes each instance ten times and uses an unbiased estimator averaged over instances.
  • Robustness testing compares text and compiled workflows under transient request failure and context compaction at 10K and 20K tokens.The study selects cases where both workflows pass in the original setting before applying perturbations.
  • The per-agent token-pressure plot measures local context burden rather than total workflow cost.The measurement concerns one subagent invoked by the artifact workflow.
  • Removing optimization reduces Medical-domain pass rate to 48%, compared with 72% without validation and 80% without dry-run feedback.Removing all three components leaves no passing cases; optimization produces the largest drop because it addresses context and decision burden.
  • Compiler-model pass rates are 80% for GLM-5, 76% for GPT-5.4, and 84% for Sonnet-4.6, compared with 48% for the text workflow.These results use GLM-4.7-Flash execution on the Medical domain in χ-Bench.
  • Compilation-time and cost estimates cover the first package-producing run for each retained SOP-Bench domain and exclude later targeted repairs.Figure 18 reports reconstructed time and prompt/response-character-based cost records.
Loading 2608.21341v1…