Source-linked AI summary
Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective
Noppanat Wadlom, Junyi Shen, Yao Lu
TL;DR
Agentic workflows generate redundant, interdependent LLM calls, while existing serving systems optimize calls individually rather than across workflows. Helium models these workloads as query plans with LLM operators, combining proactive caching and cache-aware scheduling. Across reported workloads, Helium reaches up to 1.56× speedup and approaches theoretical scheduling optimality, though dynamic control flow and unpredictable external APIs remain scope boundaries.
Problem
Agentic workflows create redundant overlapping work, while existing LLM serving systems lack visibility into broader workflow structure and cross-call dependencies.
Method
Helium models agentic workloads as query plans with LLMs as first-class operators, using proactive caching and cost-based, cache-aware scheduling to maximize reuse.
Results
Helium achieves up to 1.56× speedup on primitive workflows, with an average optimality gap of 0.9% and a maximum of 3.6%.
Takeaways & Limitations
Workflow-aware optimization can apply query-optimization principles to reduce redundant computation and improve end-to-end efficiency for agentic LLM serving.
Takeaways & Limitations
Helium’s DAG abstraction struggles with dynamic control flows, and external API latency is unpredictable under its current best-effort handling.
Abstract
from arXiv · showhide
Agentic workflows are composed of sequences of interdependent Large Language Model (LLM) calls, and they have become a dominant workload in modern AI systems. These workflows exhibit extensive redundancy from overlapping prompts and intermediate results due to speculative and parallel exploration. Existing LLM serving systems, such as vLLM, focus on optimizing individual inference calls and overlook cross-call dependencies, leading to significant inefficiencies. This paper rethinks LLM and agent serving from a data systems perspective and introduces Helium, a workflow-aware serving framework that models agentic workloads as query plans and treats LLM invocations as first-class operators. Helium integrates proactive caching and cache-aware scheduling to maximize reuse across prompts, KV states, and workflows. Through these techniques, Helium bridges classic query optimization principles with LLM serving, achieving up to 1.56x speedup over state-of-the-art agent serving systems on various workloads. Our results demonstrate that end-to-end optimization across workflows is essential for scalable and efficient LLM-based agents.
A Data Systems Perspective (Extended)
The section identifies the paper’s authors and their institutional affiliation.
- The paper lists Noppanat Wadlom, Junyi Shen, and Yao Lu as authors.
- The authors are affiliated with Singapore.
- The listed contact addresses use the comp.nus.edu.sg domain.
CCS Concepts
The paper is categorized under query optimization and multi-agent systems, with keywords emphasizing LLMs, agentic workflows, and query optimization.
- The paper’s computing classification includes query optimization.
- The paper’s computing classification includes multi-agent systems.
- The listed keywords are large language models, agentic workflows, and query optimization.
1 Introduction
Agentic workflows combine multiple LLM calls and speculative exploration, creating cross-call redundancy that individual-call serving systems cannot address. Helium applies query-plan optimization, proactive caching, and cache-aware scheduling to optimize these workflows end to end.
- Agentic workflows execute multiple LLM invocations in goal-driven sequences and have become a dominant workload in modern AI systems.
- Speculative and parallel exploration can issue overlapping or repeated sub-queries, producing substantial redundancy and inefficiency.
- Existing serving engines optimize individual LLM calls but lack visibility into broader multi-call workflow structure.
- Helium represents workflows as query-plan DAGs whose nodes retrieve data or invoke LLMs and whose edges carry data or prompt flow.
- Helium recognizes shared prefixes during compilation, pre-warms KV caches, rewrites plans, and maximizes KV-state reuse across batch workflows.
- Helium models agentic workflows as query plans with LLMs as first-class operators and adds proactive caching plus cache-aware scheduling.
2 Background and Motivation
Agentic workflows resemble data-processing DAGs but introduce stateful, expensive LLM operators, speculative redundancy, and cache-management challenges. Helium addresses these challenges through proactive cross-workflow caching and cost-based optimization within a defined deployment scope.
- Agentic workflows resemble traditional data-processing DAGs while generating redundancy across prompts and results through speculative execution.
- Existing LLM serving work primarily optimizes standalone streams through techniques such as PagedAttention, continuous batching, and prefix caching.
- Workflow frameworks simplify DAG construction but commonly treat LLM operators as black-box units.
- Current systems redundantly reprocess shared conversational context and use passive, opportunistic prefix caching.
- Helium proactively reuses KV cache across operators and workflows while pairing that cache with a cost-based optimizer for plan rewriting.
- Helium assumes semantic preservation and on-premise multi-GPU execution, while limiting the scope of supported agents.
3 System Overview
Helium represents batch agentic workflows as DAG-based query plans and processes them through parsing, optimization, and execution phases. Its optimizer removes redundant computation and substitutes cache hits, while the processor schedules optimized operators and proactively caches shared prefixes.
- Helium represents each workflow as a procedural template whose dependent LLM, retrieval, and transformation calls form a DAG over batched inputs.
- The logical optimizer rewrites DAGs to prune redundant nodes, consolidate identical subgraphs, and replace deterministic cache hits with CacheFetch operators.
- Helium builds a templated radix tree to expose prompt structure and dependencies for cost-based, cache-aware worker assignment and execution ordering.
- Static prompt prefixes identified during planning are proactively precomputed and cached, allowing later workflow executions to reuse their KV states.
4 Query Optimizer Design
Helium’s query optimizer identifies and removes structural redundancy before execution, then produces an optimized logical plan that separates cache retrieval and parallel execution intent from physical worker placement.
- The optimizer searches for sharing from entire sub-workflows down to prompt prefixes and applies two stages of optimization.
- Initial Plan Pruning: Initial plan pruning removes dead operators and merges structurally identical subgraphs, simplifying speculative workflows before later optimization.
- Caching is restricted to deterministic operators, uses LRU eviction, and adds negligible CPU-bound optimization overhead relative to LLM inference.
- The resulting logical plan expresses caching and parallelism without fixing workers or timelines, enabling runtime physical planning.
5 Query Processor Design
Helium’s query processor models prompt sharing and operator dependencies with a templated radix tree, then uses cache-aware scheduling and proactive KV caching to reduce redundant work. Its cost model accounts for token usage, worker capacity, prefix reuse, and precedence delays while optimizing workflow makespan.
- The templated radix tree represents static and dynamic prompt structure while its leaf dependencies form the workflow DAG used for scheduling.
- Proactive Cache Management: Proactive caching precomputes invariant prompt-prefix KV states during the first execution and reuses them across subsequent batches.
- Scheduling Problem Formulation: The scheduling objective minimizes the final completion token step subject to dependency and precedence-delay constraints.
- Solver by Cache-Aware Scheduling: The scheduling problem is NP-hard, so Helium uses a cost-based greedy algorithm operating on workflow-level TRTs rather than batch-sized call sets.
- Helium’s cache-aware scheduler assigns operators to workers and orders execution using a cost model designed to balance load and maximize shared-prefix KV reuse.
6 Implementation
Helium implements its workflow DSL as a lazy symbolic dataflow system and integrates it with separate vLLM workers. The implementation also requires operator profiling to support scheduling decisions, with profiling overhead amortized across later workflows.
- The Python DSL records primitive operator calls and dependency edges in a symbolic DAG instead of executing them immediately.
- Each worker runs a dedicated vLLM engine process, while Helium augments vLLM to pin precomputed prefix KV caches.
- Helium requires statistics such as average output token counts to make effective scheduling decisions.
- Offline profiling uses a separate query set, and its small one-time cost can be amortized across workflows.
7 Evaluation
Helium consistently outperforms agent-serving baselines across workflows, while ablations and sensitivity studies show that global plan optimization, proactive caching, and cache-aware scheduling drive its performance and robustness.
- 100.92× is Helium’s maximum speedup over naive vLLM, whose sequential execution prevents batch-computation gains.
- 1.56× is Helium’s maximum speedup over KVFlow, with larger gains on workflows exhibiting high prefix sharing.
- 23.35% is the largest ablation performance drop, caused by disabling plan pruning and retaining redundant operators that obscure prefix identification.
- 0.9% is Helium’s average optimality gap, with a maximum of 3.6%, indicating near-optimal schedules compared with substantially worse baselines.
- Helium remains more resilient under constrained KV-cache memory because cache-aware scheduling anticipates pressure and organizes execution for reuse.
- 2.07× is Helium’s maximum speedup over LangGraph in high-sharing Debate scenarios, while Helium remains advantageous with divergent prefixes.
- 552 KiB versus 14.8 MiB is Helium’s peak scheduling-structure memory at 16 branches compared with SGLang’s RadixCache metadata.
8 Related Work
Prior work improves individual LLM inference, agent orchestration, or workflow execution, but does not jointly eliminate prompt redundancy and schedule across workflows for prefix reuse.
- LLM inference optimizations improve throughput, KV-cache management, resource use, and generation speed at multiple system levels.
- Agentic workflow frameworks simplify development and orchestration but miss system-level optimizations.
- Recent workflow-aware systems improve scheduling, caching, or composability, yet do not target prompt-redundancy elimination and workflow-level cache-aware scheduling together.
- Helium differs from LLM-enabled data-system approaches by treating LLMs as white-box operators and combining query optimization with proactive caching and cache-aware scheduling.
9 Limitations and Future Work
Helium’s current design prioritizes cache locality and scheduling efficiency, while its DAG abstraction and external-API handling leave important extensions for future work.
- Helium’s current design prioritizes cache locality and scheduling efficiency, with implications discussed as future work.
- Expressiveness of DAG Abstraction: The DAG abstraction struggles with dynamic control flows such as conditional looping and runtime-dependent dynamic mapping.
- External API Calls: External tools introduce unpredictable latencies, so Helium currently handles them through best-effort execution rather than fully modeled global optimization.
10 Conclusion
Helium models agentic workflows as query plans with LLM operators, then combines proactive caching, cache-aware scheduling, and dataflow abstractions to improve end-to-end serving efficiency.
- Helium models agentic workflows as query plans with LLMs as first-class operators and combines proactive caching with cost-based, cache-aware scheduling.
- The templated radix tree captures shared prefixes and data dependencies among LLM operators by translating an optimized workflow DAG into a prefix-based representation.
- Operators are topologically ordered, converted into prefix templates with static tokens and dynamic antecedent inputs, and inserted into the templated radix tree.
- The radix-tree construction takes O(N+|E|) for topological sorting and prefix-template generation before operator insertion costs are added.
- The scheduling algorithm has polynomial time complexity in the templated radix tree and dependency graph, with recursive scheduling as the dominant phase.
- Helium’s DSL separates workflow definition from execution through a lazy, symbolic dataflow system with input/output, prompt, and model operators.
D Prefix Cache Utilization
Helium achieves the strongest prefix-cache results for Qwen3-8B and remains competitive for Qwen3-14B, while global scheduling determines whether cache reuse improves end-to-end latency.
- Helium achieves the highest prefix-cache hit rates for Qwen3-8B across all baselines and remains competitive for Qwen3-14B.
- 27.5% to 37.3%: Helium outperforms all baselines on Qwen3-8B prefix-cache hit rate.
- For Qwen3-14B, Helium’s advantage reaches 14.9%, while vLLM has a marginally higher hit rate of +0.3%.
- Qwen3-14B’s larger KV-cache footprint increases memory pressure and limits Helium’s ability to retain precomputed prefixes.
- A high cache-hit rate alone is insufficient because workflow-aware scheduling is needed to translate cache reuse into end-to-end performance gains.
- 29.4%: Parrot trails Helium for Qwen3-8B because prefix-based routing creates worker imbalance, evictions, and lower cache hit rates.