Source-linked AI summary
Substrate-Portable Execution for Production LLM Workflows
Tarun Gopinath, Atul Kulkarni, Vijay Rajakumar, Shrikar Katti, Parthasarathy Govindarajen
TL;DR
Production LLM workflows must serve real-time, asynchronous, and batch workloads despite runtimes with different execution requirements. The paper presents a binding-adaptive platform that compiles one typed dataflow workflow into substrate-specific execution, including first-class inference. Across dozens of production configurations, the bindings showed no detectable output-quality difference, while batch execution enabled lower-cost offline inference; portability remains bounded by operational differences and time-sensitive tools.
Problem
Production agents share logic across real-time, asynchronous, and batch workloads, but existing runtimes couple workflows to one execution model and limit substrate-specific inference strategies.
Method
The platform defines workflows once as typed dataflow graphs, represents inference as a graph node, and compiles them to Local streaming, SWF asynchronous orchestration, or Flink batch execution.
Results
Across dozens of production configurations and five orchestration patterns, all 95% Bradley-Terry confidence intervals included zero, indicating no statistically detectable quality preference for any binding.
Takeaways & Limitations
Binding selection can depend on cost, latency requirements, and operational characteristics rather than output quality, while batch execution provides access to lower-priced batch inference.
Takeaways & Limitations
The guarantee is functional portability rather than identical operational behavior, and wall-clock-dependent tool outputs may differ when batch execution occurs later.
Abstract
from arXiv · showhide
Production LLM agents execute tool-calling loops, retrieval chains, and compositional workflows in multiple modes, yet execution semantics are often coupled to one runtime. We encountered this portability problem in Rufus, a conversational AI assistant with a large tool catalog that serves millions of Amazon customers. Rufus supports real-time serving, asynchronous background tasks, and high-volume batch workloads such as evaluation and content pregeneration. Each mode has distinct service-level objectives and typically uses a separate runtime. Reusing streaming orchestration makes asynchronous and batch workloads blocking and prevents use of batch inference APIs, which offer a 50 percent discount at published prices. We present a binding-adaptive agent execution platform that separates workflow definition from execution substrate. Developers define a workflow once as a typed dataflow graph. The platform compiles the graph to in-process streaming for real-time serving, durable AWS SWF orchestration for asynchronous execution, or distributed Apache Flink stream processing for batch inference. No workflow code changes are required. LLM inference is represented as a suspendable graph node whose behavior depends on the substrate: streaming delivery online, durable retry asynchronously, and batched submission offline. We validated dozens of production agent configurations across five orchestration patterns: single-inference RAG, iterative ReAct, compositional PreAct, conditional routing, and multi-agent deep research. Across all three bindings, we found no detectable difference in output quality. Batch execution reduced per-query inference cost in line with published batch API pricing while operating alongside the streaming path at production scale.
1 Introduction
Production agents must reuse shared logic across real-time, asynchronous, and batch workloads, but current runtimes couple workflows to execution modes with different requirements. The paper introduces execution-mode portability for Rufus by compiling one workflow definition across heterogeneous substrates.
- Case study: Rufus is a conversational AI assistant used by millions of Amazon customers and integrates a large catalog of MCP endpoints.Its endpoints support functions including product search, knowledge retrieval, and price tracking.
- Workload diversity: Real-time serving, asynchronous background tasks, and batch workloads have distinct latency, durability, and throughput requirements.These modes support live interaction, long-running tasks such as deep research, and offline evaluation or content pre-generation.
- Portability gap: Shared prompts, tools, and orchestration logic become blocking when asynchronous and batch workloads reuse real-time execution.This coupling also places offline workloads on infrastructure provisioned for low-latency serving.
- Approach: The platform defines agent workflows once as typed dataflow graphs and compiles them to multiple execution substrates.The approach targets real-time, asynchronous, and batch execution without changing workflow definitions.
2 System Design
The system represents agents as typed, binding-agnostic dataflow graphs and compiles them into substrate-native operations. First-class inference exposes execution properties needed for streaming, durable asynchronous orchestration, and cross-invocation batching.
- Design challenge: Existing orchestration systems treat LLM calls as opaque blocking operations, coupling one workflow to a single execution strategy.This prevents asynchronous batch submission and aggregation of independent inference requests.
- Typed dataflow DSL: Agent workflows are typed dataflow graphs authored through equivalent Java and JavaScript interfaces.The Java API provides compile-time checks, while the JavaScript DSL supports hot reloading and rapid exploration; both frontends produce the same combinator model and immutable graph.
- Workflow patterns: The restricted combinator interface supports patterns such as speculative PreAct while preserving behavioral equivalence across bindings.Arbitrary I/O or runtime graph mutation would make that guarantee infeasible.
- First-class inference: Representing inference with .infer(...) exposes a stateless, remotely executable operation that can be submitted asynchronously and aggregated for batch processing.Inference dominates workflow cost and latency, making its execution strategy central to each binding’s operational profile.
- Binding compilation: The binding compiler traverses the workflow graph and translates nodes into substrate-native operations for Local, SWF, or Flink execution.A shared control plane manages configuration, registration, and lifecycle across bindings.
- Execution bindings: Local streams tokens synchronously for low latency, SWF provides durable traceable asynchronous execution, and Flink batches requests across invocations.The bindings respectively target progressive rendering, long-running coordination, and scalable offline execution.
3 Evaluation
The evaluation tests whether identical agent workflows preserve quality across Local, SWF, and Flink bindings, while comparing their latency, cost, and operational profiles. Across five orchestration patterns, no binding showed a statistically detectable quality preference, while batch execution offered cost advantages with substantial latency overhead.
- Evaluation setup: The evaluation covers five orchestration patterns—RAG, ReAct, Routing, PreAct, and Deep Research—using more than 500 expert-authored and synthetic prompts per pattern.All bindings run against the same Bedrock endpoints and dataset under Local streaming, SWF asynchronous execution, and Flink batch execution.
- Cross-binding quality equivalence: Every reported 95% Bradley-Terry confidence interval includes zero, indicating no statistically detectable quality preference for any binding.ReAct and Deep Research have wider intervals because they produce fewer decisive comparisons; Routing and PreAct have narrower intervals.
- Binding overhead: 1.1–3.4× SWF latency and 33–152× Flink latency exceed the Stream baseline, with Flink delay driven primarily by provider-side batch scheduling.Deep Research has the smallest Flink multiplier because specialist fan-out executes in parallel, amortizing latency across concurrent calls.
- Cost and operational implications: Flink batch execution can reduce evaluation cost consistently with published batch API pricing while still detecting model downgrades and prompt regressions.For the representative Sonnet ReAct agent, tokens account for approximately 100% of per-query cost in every binding, while non-token infrastructure accounts for less than 0.5%.
- Cost and operational implications: The absence of detectable quality differences makes binding selection primarily a trade-off among cost, latency requirements, and operational characteristics.SWF provides durable coordination for long waits, whereas its per-step scheduling latency makes it unsuitable for traffic sensitive to time to first token.
4 Limits of Portability
Portability preserves workflow behavior across substrates, but not operational behavior. Time-sensitive tools, substrate-specific failures, and execution-context dependencies define the practical boundary of the guarantee.
- Functional portability yields statistically indistinguishable outputs for the same input, but latency, failures, and other operational behavior can differ across bindings.The guarantee covers successful workflow behavior rather than identical runtime characteristics.
- Time-sensitive tool outputs can change when batch execution occurs later, potentially altering continuation predicates independently of the binding.The divergence is attributed to the tool’s wall-clock dependence.
- Asynchronous and batch substrates add correlation, partial-result, token-limit, and completion-window failure modes absent from direct synchronous returns.These risks are considered acceptable for offline workloads without strict latency SLOs.
- Portable workflows must isolate side effects in tool calls and avoid dependencies on execution context, such as streamed-token timing for user-interface behavior.These constraints separate portable logic from binding-specific logic.
5 Related Work
Related systems address dataflow portability, durable execution, workflow composition, inference serving, or evaluation, but generally emphasize different layers or execution goals. This work combines typed agent workflows with substrate-specific inference execution across streaming, asynchronous, and batch modes.
- Apache Beam defines data pipelines once for equivalent backends, whereas this work targets substrates with different latency, cost, and failure semantics.The comparison distinguishes correctness-equivalent runners from intentionally heterogeneous execution bindings.
- Agent frameworks commonly represent LLM calls as function invocations, while this platform exposes inference as a substrate-aware operation in a typed DSL.The distinction enables compilation across execution modes rather than inheriting one blocking strategy.
- Durable-execution systems prioritize asynchronous orchestration, while cross-invocation batching and streaming token delivery are outside their primary design goals.The platform operates at the agent-programming layer and incorporates first-class inference.
- Inference-serving systems optimize GPU use below the workflow layer and are complementary to this platform’s execution abstraction.The related systems target serving efficiency rather than portable agent orchestration.
- Evaluation frameworks separate evaluation from production execution, whereas this platform reuses production workflow definitions to reduce behavioral drift.Cloud batch APIs target single-turn inference, so multi-turn tool workflows require separate inference submission and orchestration.
6 Conclusion
The paper presents a binding-adaptive platform that compiles one typed workflow definition across heterogeneous runtimes. Across production configurations and orchestration patterns, output quality showed no detectable binding-specific difference, while batch execution separated offline work and enabled batch pricing.
- One typed dataflow workflow definition compiles across real-time, asynchronous, and batch runtimes.Inference is represented as a first-class operation, allowing runtime-specific execution strategies.
- Across dozens of production configurations and five orchestration patterns, output quality showed no detectable binding-specific difference.The evaluation covered Amazon Bedrock and conversational shopping.
- The results are scoped to Amazon Bedrock and conversational shopping; providers or domains with different response characteristics may behave differently across bindings.This defines the paper’s main empirical scope boundary.
- Batch execution provides access to batch inference pricing while separating offline workloads from real-time serving.The broader conclusion is that runtime-layer differences can preserve equivalence when the workflow interface is sufficiently restricted.