PaperlayerView exact revision on arXiv

Source-linked AI summary

Parrot: Efficient Serving of LLM-based Applications with Semantic Variable

Chaofan Lin, Zhenhua Han, Chengruidong Zhang, Yuqing Yang, Fan Yang, Chen Chen, Lili Qiu

arXiv:2405.19888v1cs.LGcs.AI

TL;DR

LLM applications use complex multi-request workflows, while request-centric public services lack the application-level information needed for end-to-end optimization. Parrot exposes that information through Semantic Variables, and evaluations report up to 11.7× speedup or 12× higher throughput over state-of-the-art solutions.

  • Problem

    Request-centric public LLM services cannot observe application workflows, dependencies, scheduling preferences, or repeated content, limiting end-to-end optimization.

  • Method

    Parrot uses Semantic Variables to annotate prompt variables and connect multiple LLM requests into data pipelines exposed to the service.

  • Results

    Up to 11.7× speedup or 12× higher throughput was achieved compared with state-of-the-art solutions on popular production and open-source LLM applications.

  • Takeaways & Limitations

    Exposing inter-request correlations enables joint optimizations for the end-to-end performance of LLM-based applications.

  • Takeaways & Limitations

    The paper focuses on Parrot’s mechanisms and a subset of possible inter-request scheduling features, leaving broader features for future consideration.

Abstract

from arXiv · show

The rise of large language models (LLMs) has enabled LLM-based applications (a.k.a. AI agents or co-pilots), a new software paradigm that combines the strength of LLM and conventional software. Diverse LLM applications from different tenants could design complex workflows using multiple LLM requests to accomplish one task. However, they have to use the over-simplified request-level API provided by today's public LLM services, losing essential application-level information. Public LLM services have to blindly optimize individual LLM requests, leading to sub-optimal end-to-end performance of LLM applications. This paper introduces Parrot, an LLM service system that focuses on the end-to-end experience of LLM-based applications. Parrot proposes Semantic Variable, a unified abstraction to expose application-level knowledge to public LLM services. A Semantic Variable annotates an input/output variable in the prompt of a request, and creates the data pipeline when connecting multiple LLM requests, providing a natural way to program LLM applications. Exposing Semantic Variables to the public LLM service allows it to perform conventional data flow analysis to uncover the correlation across multiple LLM requests. This correlation opens a brand-new optimization space for the end-to-end performance of LLM-based applications. Extensive evaluations demonstrate that Parrot can achieve up to an order-of-magnitude improvement for popular and practical use cases of LLM applications.

1 Introduction

LLM applications use multi-request workflows, but request-centric public services lack the application-level information needed to optimize their end-to-end performance. Parrot exposes this information through Semantic Variables, enabling joint optimizations and up to 11.7× speedup or 12× higher throughput.

  • Problem: Request-centric services optimize isolated calls because they cannot identify application membership, request connections, or similarities, producing sub-optimal end-to-end performance.This lost application-level information prevents services from exploiting cross-request optimization opportunities.
  • Problem: Dependent consecutive requests incur network overhead and lose co-scheduling opportunities because clients must receive one result before issuing the next request.Colocating and executing dependent requests consecutively on the service side can avoid this interactive overhead.
  • Problem: Different tasks require different scheduling policies: Map requests benefit from batching for throughput, while scarce Reduce requests should prioritize latency.Request-level services cannot distinguish these task preferences and therefore optimize individual latency blindly.
  • Problem: Repeated static system prompts account for over 94% of tokens across users in one production search-engine analysis, wasting storage, computation, and memory bandwidth.The repetition arises because public services process common prefixes independently in each request.
  • Parrot: Semantic Variables annotate prompt regions and connect requests into data pipelines, exposing prompt structure and inter-request correlations to the service.The abstraction can represent instructions, examples, inputs, and outputs while linking multiple LLM requests.
  • Results: Up to 11.7× speedup or 12× higher throughput shows Parrot outperforming state-of-the-art solutions on popular production and open-source LLM applications.The evaluation covers practical applications and reports both latency and throughput improvements.

2 Background

LLM services expose text completion APIs in which clients submit prompts and receive generated text. Their backends queue requests and dispatch them to GPU-based inference engines.

  • LLM Service: Most LLM services provide conditional generation through a text completion API.The client supplies a text prompt and receives generated text.
  • LLM Service: A request scheduler dispatches queued LLM requests to inference engines that use GPU sets to conduct generation.The service provider may operate one or multiple inference clusters.
  • LLM-based Applications: Long-document analytics use map-reduce or chain workflows because limited context windows require chunking, multiple partial-result requests, and later combination.Map tasks summarize chunks before a Reduce task combines them, while chain workflows combine results incrementally.

A B Queue

The supplied material identifies LLM engines and a scheduler, while Figure 3 attributes current-service overhead to network and queuing from chatty application-service interaction.

  • Architecture: The depicted service architecture includes multiple LLM engines coordinated by a scheduler.The supplied architectural labels identify the engines and scheduler but do not provide further queue measurements.
  • Latency Breakdown: Figure 3 attributes current LLM-service overhead to network latency and queuing caused by chatty interaction between applications and services.The figure describes this overhead as eliminated in Parrot.

3 Problems of Serving LLM Applications

Existing public LLM services lose application-level workflow information, so they optimize individual requests rather than end-to-end application performance. This creates overhead from dependent requests, misaligned scheduling, and redundant prompt computation.

  • Excessive Overhead of Consecutive Requests: 30–50% of LLM API-call latency comes from outside the engine on average, exceeding 70% in the worst cases.The overhead grows with prompt length and can cause timeouts and resubmissions.
  • Excessive Overhead of Consecutive Requests: Dependent requests incur network and queuing delays because clients must receive one response before submitting the next request.Parrot instead batches consecutive requests so one step’s output can feed directly into the next.
  • Overview: Public services cannot distinguish application workflows or performance objectives because request-centric APIs expose only individual requests.This prevents them from exploiting dependencies, heterogeneous scheduling preferences, and prompt commonality.
  • Redundant Computations: More than 94% of prefix tokens can be reused across requests, while MetaGPT and AutoGen show 72% and 99% redundancy, respectively.Repeated common prompts waste storage, computation, and GPU memory bandwidth.

4 Parrot Design

Parrot exposes application structure through Semantic Variables and uses that information to analyze request relationships and prompt sharing. Its manager then schedules requests and applies end-to-end performance criteria across the resulting application workflow.

  • System Overview: Parrot Manager schedules analyzed requests across cluster LLM engines to optimize application-level end-to-end performance.The system is designed as a cluster-level manager coordinating LLM requests with the engines that serve them.
  • 4.1 Semantic Variable: Semantic Variables annotate prompt input and output variables, preserving structure that public services can analyze across requests.Unlike ordinary orchestration placeholders rendered before submission, Semantic Variables remain visible to the service.
  • 4.1 Semantic Variable: Connecting Semantic Variables across semantic functions forms a data pipeline and reveals request dependencies through data-flow analysis.For example, the code variable connects code-generation and test-generation requests sequentially.
  • Performance Criteria: Applications can fetch output variables asynchronously and attach end-to-end criteria such as latency or throughput to those operations.Criteria attached to final outputs are propagated to middle variables and used as scheduling hints.
  • Inter-Request Analysis: Parrot derives a request DAG and prompt structure from Semantic Variables for inter-request analysis and optimization.The DAG stores request and variable nodes, while PrefixHash captures shared prefixes at positions separated by Semantic Variables.

5 Optimizations with Semantic Variable

Parrot uses Semantic Variables to expose request dependencies and application objectives, enabling graph-based execution, objective-aware scheduling, and shared-prefix optimization.

  • 5.1 Data Pipeline: Semantic Variables let Parrot execute dependent requests through a graph-based executor, avoiding unnecessary client-side composition and maximizing batching opportunities.Requests are dispatched when producer requests finish, enabling immediate downstream execution.
  • 5.2 Performance Objective Deduction: Parrot derives scheduling preferences from application-level performance criteria and propagates throughput objectives to requests generating a Semantic Variable.Throughput preference is especially useful for offline bulk document analysis.
  • 5.2 Performance Objective Deduction: Latency-sensitive applications may require throughput-oriented Mapping requests because Parrot optimizes completion time for the entire request DAG rather than each request independently.Parrot analyzes requests in reverse topological order from latency-critical Semantic Variables.
  • 5.3 Sharing Prompt Prefix: Parrot shares common prompt prefixes by tracking prefix-to-request mappings and using a specialized attention kernel that reduces redundant memory loading.The kernel combines PagedAttention and FlashAttention ideas for shared-prefix decoding.
  • 5.4 Scheduling: Parrot’s scheduler orders requests topologically, groups related requests, and allocates task groups together while balancing throughput and latency objectives.Requests without identified sharing opportunities are scheduled independently.

6 Discussion

Parrot’s design supports integration with orchestration frameworks while leaving dynamic control flow, native function execution, and several broader scheduling optimizations for future work.

  • Dynamic Applications and Function Calling: Parrot currently supports cloud-side orchestration of LLM requests but leaves dynamic control flow and native function execution on the client side.The authors avoid offloading native functions to reduce security risks from malicious injection.
  • Dynamic Applications and Function Calling: The authors leave extensions for conditional connections, native code submission, and speculative branch launching as future work.These extensions are described as especially applicable to trusted private LLM services.
  • Other Applications of Inter-Request Analysis: Parrot’s inter-request analysis could support additional scheduling features, but this paper focuses on its mechanisms and selected use cases.Examples of deferred areas include outlier handling, failures, fairness, starvation, delay scheduling, and heterogeneous clusters.
  • Parrot with LLM Orchestration Frameworks: Parrot can integrate with LLM orchestration frameworks by extending their LLM-service calls with Semantic Variables.Template placeholders in these frameworks resemble Semantic Variables, but rendering before submission removes prompt-structure information from the service.

7 Implementation

Parrot implements Semantic Variables, request management, scheduling, and optimized LLM serving through a Python-based front end, centralized manager, and universal engine abstraction.

  • System Architecture: Parrot’s front end exposes Semantic Variables and SemanticFunctions, while a centralized manager handles request communication, variables, and scheduling.The implementation includes an LLM engine built from vLLM, xFormers, and custom kernels.
  • Semantic Variable APIs: Parrot preserves input and output placeholders, supports transformations around value exchange, and exposes APIs for setting and fetching Semantic Variables.Fetching a variable returns an error when intermediate engine, communication, or transformation steps fail.
  • Kernel Optimization: A custom GPU kernel combines PagedAttention and FlashAttention principles to reduce repeated shared-prefix KV-cache loading during attention decoding.Shared-prefix tiles are loaded into shared memory once, reducing memory transactions between L2 cache and shared memory.
  • Universal Engine Abstraction: Parrot’s universal engine abstraction requires stateful generation and KV-cache sharing across requests so heterogeneous engines can support its optimizations.The cluster manager controls engines with differing models, tokenizers, and KV-cache layouts.
  • LLM Engine: Parrot’s engine separates prompt processing from token generation through Fill and Generate operations, allowing finer-grained dependency handling and more parallel execution.Fill processes initial prompt tokens and stores KV-cache state, while Generate performs iterative decoding.

8 Evaluation

Parrot is evaluated across representative LLM applications and serving workloads, showing end-to-end latency and throughput gains from application-aware coordination. The improvements come from reducing communication and queueing overhead, grouping related tasks, sharing prompt context, and scheduling heterogeneous workloads appropriately.

  • 8.2 Data Analytics on Long Documents: 1.88× lower end-to-end latency than HuggingFace and 1.38× lower latency than vLLM were achieved for chain summarization.The advantage primarily came from reducing client interaction and network latency.
  • 8.2 Data Analytics on Long Documents: 2.38× lower end-to-end latency than vLLM was achieved for chain summarization under background requests.Parrot processes the next dependent chunk immediately, avoiding repeated queue entry and additional client-side interaction.
  • 8.2 Data Analytics on Long Documents: 2.37× acceleration over the baseline was achieved for map-reduce summarization by grouping mapping tasks and using larger batches to improve throughput.The baseline optimized each request independently with a limited 4096-token capacity, which harmed application-level end-to-end performance.
  • 8.3 Serving Popular LLM Applications: 1.1×–1.7× speedup over vLLM was achieved for shared-prompt workloads, with Parrot loading shared prompt tokens only once during attention.Figure 16 reports 1.58× and 1.84× speedups over vLLM with paged attention, including 40 ms per-output-token latency at batch size 32.
  • 8.4 Multi-agent Applications: 2.45× faster performance than a throughput-centric baseline was achieved for multi-agent programming through prompt-structure analysis and shared-context reuse.The shared-prefix mechanism contributed 2.35× acceleration, while a specialized GPU kernel added 1.2× speedup at 16 files.
  • 8.5 Scheduling of Mixed Workloads: Up to 5.5× and 1.23× normalized-latency improvement over latency-focused and throughput-focused baselines, respectively, was achieved for chat applications.For map-reduce applications, Parrot reached 3.7× over the latency-focused baseline and 1.05× over the throughput-focused baseline.

9 Related Works

Related work spans deep learning serving, LLM orchestration, and DAG-aware optimization. Parrot differs by exposing application-level dependencies and commonality through Semantic Variables to optimize end-to-end application performance.

  • Deep Learning Serving Systems: Parrot is orthogonal to deep learning serving systems because Semantic Variables expose application knowledge for data-flow analysis and end-to-end optimization.Existing serving systems address batching, caching, placement, scheduling, and model parallelism for model deployment.
  • LLM Orchestrator Frameworks: LLM orchestration frameworks provide workflow patterns such as chain and map-reduce, whereas Parrot targets their execution through application-aware serving.LangChain is identified as a framework for creating and managing LLM application workflows.
  • DAG-aware System Optimizations: Parrot extends DAG-aware system optimization to LLM requests by learning application-level relationships and using them for serving decisions.Prior systems exploit dependency graphs for data analytics and serverless workloads.

10 Conclusion

Parrot treats LLM applications as first-class citizens and uses Semantic Variables to expose request dependencies and commonality. Its evaluation reports up to 11.7× application-level optimization, while future work includes fairness in end-to-end performance.

  • 10 Conclusion: Parrot targets end-to-end LLM application performance rather than optimizing individual requests.The system treats LLM applications as first-class citizens.
  • 10 Conclusion: Semantic Variables expose request dependencies and commonality, enabling a new optimization space for LLM applications.This abstraction is the paper’s central mechanism for application-aware serving.
  • 10 Conclusion: Up to 11.7× improvement was achieved for LLM-based applications in the evaluation.The conclusion presents this as the paper’s overall efficiency result.
  • 10 Conclusion: Future work should study scheduling features such as fairness of end-to-end application performance.The conclusion identifies fairness as a direction rather than a completed capability.
Loading 2405.19888v1…