PaperlayerView exact revision on arXiv

Source-linked AI summary

Clipper: A Low-Latency Online Prediction Serving System

Daniel Crankshaw, Xin Wang, Giulio Zhou, Michael J. Franklin, Joseph E. Gonzalez, Ion Stoica

arXiv:1612.03079v2cs.DCcs.LG

TL;DR

Online machine-learning applications require accurate, robust predictions with low latency under heavy query loads, but existing frameworks mainly support training rather than deployment. Clipper interposes a modular, layered serving system with caching, batching, and adaptive model selection across frameworks. Evaluations show bounded latency, scalable throughput, robust predictions, and parity with TensorFlow Serving on throughput and latency while providing additional functionality.

  • Problem

    Most machine-learning frameworks emphasize training and provide limited support for scalable, accurate, and reliable inference deployment.

  • Method

    Clipper uses model abstraction and model selection layers with caching, adaptive batching, and dynamic selection or combination of predictions across models.

  • Results

    Across four benchmark datasets, Clipper bounded latency, scaled workloads, produced accurate and robust predictions, and achieved throughput and latency parity with TensorFlow Serving.

  • Takeaways & Limitations

    Clipper’s modular architecture supports deployment across frameworks while enabling model composition and online adaptation without modifying underlying frameworks.

  • Takeaways & Limitations

    Clipper treats served models as black boxes and therefore does not optimize their execution; slow models remain slow.

Abstract

from arXiv · show

Machine learning is being deployed in a growing number of applications which demand real-time, accurate, and robust predictions under heavy query load. However, most machine learning frameworks and systems only address model training and not deployment. In this paper, we introduce Clipper, a general-purpose low-latency prediction serving system. Interposing between end-user applications and a wide range of machine learning frameworks, Clipper introduces a modular architecture to simplify model deployment across frameworks and applications. Furthermore, by introducing caching, batching, and adaptive model selection techniques, Clipper reduces prediction latency and improves prediction throughput, accuracy, and robustness without modifying the underlying machine learning frameworks. We evaluate Clipper on four common machine learning benchmark datasets and demonstrate its ability to meet the latency, accuracy, and throughput demands of online serving applications. Finally, we compare Clipper to the TensorFlow Serving system and demonstrate that we are able to achieve comparable throughput and latency while enabling model composition and online learning to improve accuracy and render more robust predictions.

1 Introduction

Clipper addresses the difficulty of deploying machine-learning models for real-time applications by providing a layered serving architecture. Its caching, batching, and adaptive model-selection techniques target latency, throughput, accuracy, and robustness across frameworks.

  • Motivation: Inference must serve substantially more queries than training while meeting real-time requirements in user-facing applications.Training may take hours to days, whereas inference must respond quickly under heavy load.
  • Challenge: Deploying machine-learning services is difficult because developers must integrate multiple evolving frameworks while meeting scalability and responsiveness demands.Existing deployment, optimization, and maintenance work is described as difficult and error-prone.
  • Architecture: Clipper uses model abstraction and model selection layers to simplify prediction serving and improve latency, throughput, accuracy, and robustness.The abstraction layer hides framework heterogeneity; the selection layer dynamically selects and combines predictions.
  • Techniques: Caching, adaptive batching, bandit methods, and ensembles optimize serving performance and prediction quality without requiring framework-specific redesign.The system also supports online adaptation of model selection and uncertainty estimation.
  • Implementation: Clipper supports several major machine-learning frameworks, with each integration added using fewer than 25 lines of code.The implementation includes Spark MLLib, Scikit-Learn, Caffe, TensorFlow, and HTK.
  • Evaluation: Across four benchmark datasets, Clipper achieved predictions below 20ms, scaled across machines, adapted model combinations, and matched TensorFlow Serving’s throughput and latency while supporting more functionality.The evaluation also reports dynamic accuracy-latency trade-offs under heavy query load.

2 Applications and Challenges

Machine-learning systems have emphasized computationally expensive training while providing limited support for scalable, accurate, and reliable inference. Clipper is evaluated against application workloads spanning object recognition and speech recognition, including settings where user feedback can guide model improvement.

  • Lifecycle: Training estimates models from data and may require multiple passes over large datasets, taking hours or days to complete.Systems research has concentrated heavily on training infrastructure.
  • Lifecycle: Inference evaluates a model to render predictions, but most frameworks primarily support offline batch inference rather than scalable online serving.The paper identifies inference as an increasingly important source of system challenges.
  • Online Feedback: User feedback can help identify failing models and support more personalized predictions as applications are used.Feedback may be explicit, such as labels, or implicit, such as clicks.
  • Object Recognition: Object-recognition evaluation uses MNIST, CIFAR-10, and ImageNet to span increasingly difficult tasks, larger inputs, and varied model requirements.The datasets cover models with different computational demands and accuracies.

Automatic Speech Recognition

Speech recognition illustrates the need for real-time inference and personalization because models can be large, costly, and specialized to users’ dialects. Clipper’s speech benchmark uses TIMIT and HTK to evaluate personalization and online model selection with simulated user interactions.

  • Application: Speech recognition maps spoken audio to word sequences and may use large, complex models built from specialized sub-models.Models are often personalized to accommodate dialect and accent variation.
  • Challenge: Online speech inference is essential for user experience and can be computationally costly because models may require large tensor-product evaluations.Real-time inference also enables applications such as real-time translation.
  • Personalization: User feedback provides an implicit signal that can help select models specialized for an individual’s dialect.The paper connects rapid feedback incorporation with improved user experience.
  • Benchmark: The benchmark uses the TIMIT corpus and HTK with recordings from 630 speakers across eight English dialects.Users are randomly drawn from the test corpus and represented through prerecorded speech interactions.

Complexity of Deploying Machine Learning

Prediction-serving systems must reconcile framework heterogeneity with strict latency and throughput requirements. Clipper addresses these constraints through a common interface, adaptive batching, and straggler mitigation while preserving latency objectives.

  • Framework Complexity: The large and specialized framework ecosystem makes deployment difficult because applications may need multiple frameworks or framework changes over time.Frameworks differ in APIs, models, hardware requirements, and application focus.
  • Framework Complexity: Clipper’s model abstraction layer isolates applications from framework variability and simplifies adding models or frameworks to running applications.It provides a common prediction interface.
  • Latency: Prediction latency is the time required to render a prediction, and serving systems must keep both latency and tail latency bounded.Sophisticated models may have latencies of 50-100ms.
  • Throughput: Batching can improve throughput through hardware and library optimizations but may worsen prediction latency under serving workloads.This creates a direct tension between throughput and latency.
  • Throughput: Clipper adaptively batches requests while meeting latency objectives and uses straggler mitigation to bound tail latency for complex models.These mechanisms allow model developers to experiment with complex models without affecting serving latency.

Model Selection

Clipper addresses the difficulty of selecting among evolving models by using adaptive online selection and ensembling to incorporate feedback and combine models across frameworks.

  • Model Selection: Offline evaluation and static model choices can become biased or outdated as models, contexts, and data distributions change.The paper identifies concept drift, feature corruption, context dependence, and ensemble opportunities as limitations of conventional selection.
  • Model Selection: Clipper uses adaptive online model selection and ensembling to incorporate feedback and automatically select and combine predictions across frameworks.The approach is designed to support models spanning multiple machine learning frameworks.
  • Experimental Setup: The evaluation covers three object-recognition benchmarks and one speech-recognition benchmark using image-labeling and phonetic-transcription tasks.Object recognition uses raw pixels as input, while speech recognition predicts phonetic transcriptions from raw audio using Hidden Markov Models.
  • Experimental Setup: The experiments ran primarily on single servers with Haswell-EP CPUs and 256 GB of RAM, while TensorFlow models used Nvidia Tesla K20c GPUs.The cluster scaling experiment additionally used 10Gbps and 1Gbps networks.

3 System Architecture

Clipper routes prediction requests through model selection and model abstraction layers, combining feedback-driven selection with framework-independent execution, caching, and adaptive batching.

  • 3 System Architecture: The model selection layer dispatches queries to one or more models and combines predictions using feedback to improve accuracy, estimate uncertainty, and provide robustness.Applications submit requests through REST or RPC APIs, after which selection determines the relevant model calls.
  • 3 System Architecture: The model abstraction layer provides a common prediction interface, resource isolation, and workload optimization for batch-oriented frameworks.It hides framework differences while supporting model execution through the serving stack.
  • 3 System Architecture: The abstraction layer checks the prediction cache before placing requests into model-specific adaptive batching queues.Batches are sent through cross-language RPC to model containers running models in their native frameworks.
  • 3 System Architecture: After model evaluation, Clipper caches returned predictions and sends them to model selection, which combines predictions into a final result and confidence.Application feedback returns through the same interface and is joined with predictions to guide future selection and combination.

4 Model Abstraction Layer

The model abstraction layer standardizes deployment across frameworks while using caching and adaptive batching to reduce repeated work and improve throughput under latency constraints. Its batching and replication mechanisms support substantial throughput gains, including cluster scaling, while exposing trade-offs from workload and network conditions.

  • 4 Model Abstraction Layer: A common interface, prediction cache, adaptive batching component, and isolated model containers let Clipper share optimizations across frameworks and scale concurrent models.Lightweight RPC provides uniform communication and simplifies adding new frameworks.
  • 4.3 Batching: Batching aggregates point queries into dynamically resized mini-batches, amortizing RPC and framework overhead while increasing latency because queries wait for batch completion.Batching exploits data-parallel inference through GPU or BLAS acceleration.
  • 4.2 Prediction Cache: Caching serves frequent queries without model evaluation and also joins predictions with feedback for adaptive model selection.In a four-model ensemble, caching increased feedback processing throughput 1.6x, from roughly 6K to 11K observations per second.
  • 4.2 Prediction Cache: The cache implements a non-blocking request and fetch API for model-query predictions and uses LRU eviction through the CLOCK algorithm.With adequate capacity, frequent queries remain cached; model-selection changes do not invalidate cache entries.
  • 4.3.1 Dynamic Batch Size: Up to 26x throughput improvement was achieved for a Scikit-Learn SVM while meeting a 20ms latency SLO.AIMD and quantile-regression strategies performed nearly identically; Clipper selected AIMD as the simpler adaptive default.
  • 4.3.2 Delayed Batching: A 2ms batch delay produced a 3.3x throughput improvement for the Scikit-Learn SVM, while delayed batching did not improve throughput for the Spark SVM.The benefit depended on fixed batch-processing costs and efficient parallel inference under bursty or moderate load.
  • 4.4.1 Container Replica Scaling: With four GPU replicas on a 10Gbps-connected cluster, throughput rose 3.95x from 19,500 qps to 77,000 qps; a 1Gbps network became the bottleneck when adding a second remote machine.The experiment shows aggregate scaling benefits alongside a network boundary for larger distributed inputs.

5 Model Selection Layer

Clipper’s model selection layer uses feedback to select or combine deployed models, supporting online adaptation, accuracy improvements, and uncertainty estimation. Its policies include generic bandit-based selection and linear ensembles, with explicit trade-offs between accuracy, latency, computation, and ensemble size.

  • Model Selection Layer: The model selection layer uses feedback to dynamically select or combine deployed models for more accurate and robust predictions.Deploying multiple candidate models lets Clipper adapt selection throughout an application’s lifetime.
  • Model Selection Layer: Clipper exposes a select, combine, and observe API so users can introduce new model selection techniques.The interface also isolates policy state and supports many contextualized policy instances.
  • 5.1 Single Model Selection Policy: Exp3 selects a single model using noisy feedback with minimal computational overhead, while Exp4 adaptively combines predictions from multiple models.These policies span a trade-off between computation overhead and accuracy.
  • 5.2 Ensemble Model Selection Policies: Linear ensembles compute weighted averages of base-model predictions and achieve a 5.2% relative reduction in ImageNet error rate.The reduction comes from combining off-the-shelf models; ensembles marginally reduce overall error on CIFAR and ImageNet.
  • 5.2 Ensemble Model Selection Policies: Within the first 5K queries, both single-model and ensemble policies converge near the best model’s error rate and mitigate errors when that model degrades.The policies learn to divert queries after model 5’s cumulative error rate spikes.
  • 5.2 Ensemble Model Selection Policies: Agreement among concurrently evaluated models provides a confidence measure, while larger ensembles improve accuracy and uncertainty estimation at additional computational and tail-latency cost.Straggler mitigation preserves the latency objective but can reduce ensemble size.

6 System Comparison

Clipper is evaluated against TensorFlow Serving on three TensorFlow object-recognition models with different inference costs. Despite its modular architecture, Clipper achieves comparable throughput, with C++ containers nearly matching TensorFlow Serving and Python containers incurring a 15–18% throughput reduction.

  • System Comparison: TensorFlow Serving uses static batching and serves one model at a time, whereas Clipper supports latency objectives, dynamic model selection, and model composition.TensorFlow Serving’s batch size requires manual tuning to meet latency requirements.
  • Workloads: The comparison evaluates latency and throughput on MNIST, CIFAR-10, and ImageNet TensorFlow networks ranging from 4-layer to 22-layer models.The models are a 4-layer CNN, 8-layer AlexNet, and 22-layer Inception-v3.
  • Results: 15–18%: Python model containers reduce throughput relative to TensorFlow Serving, while Clipper’s C++ containers achieve nearly identical performance.The comparison spans all three evaluated models.
  • Results: The throughput bottleneck is GPU inference, and Clipper’s RPC and modular-system overheads are minimal for these workloads.Both systems queue additional work to saturate the GPU.
  • Results: Comparable performance across the evaluated models indicates that Clipper’s modular architecture and broader features do not reduce performance on core serving tasks.This conclusion concerns the tested prediction-serving workloads.

7 Limitations

Clipper treats models as black-box components below its serving layer, which limits optimization of model execution and management of base-model training. Consequently, slow or outdated models remain constrained by their underlying frameworks and ensemble capabilities.

  • Execution Optimization: Clipper’s black-box model abstraction limits its ability to optimize execution within the underlying machine learning frameworks.Slow models remain slow when served from Clipper, unlike tightly integrated TensorFlow Serving models.
  • Training Management: Clipper does not manage training or retraining of base models within their respective frameworks.This boundary is a direct consequence of the architecture’s separation from model development.
  • Training Management: If all deployed models are outdated or inaccurate, Clipper cannot improve accuracy beyond what ensembles can accomplish.Model selection operates over the available base models rather than updating them.

8 Related Work

Related systems address prediction-serving latency, throughput, or application-specific needs, but are generally vertically integrated around particular models or frameworks. Clipper instead combines cross-framework serving with mechanisms for latency, throughput, accuracy, and online adaptation.

  • Latency and Throughput: LASER and Velox use caching, while TensorFlow Serving relies on batching and hardware acceleration rather than caching.LASER also uses straggler mitigation; neither LASER nor Velox discusses batching.
  • Online Learning: LASER and Velox incorporate forms of feedback or context, but neither supports the full combination of real-time feedback, bandits, and cross-framework learning described for Clipper.TensorFlow Serving has no mechanism to achieve the corresponding prediction-serving functionality.
  • General Serving Systems: LASER, Velox, and TensorFlow Serving address prediction-serving concerns but focus on a single model or framework through vertical integration.LASER supports linear ad-targeting models, Velox uses Apache Spark, and TensorFlow Serving targets TensorFlow.
  • Clipper’s Scope: Clipper supports a wide range of machine learning models and frameworks while addressing latency, throughput, and accuracy in one serving system.This contrasts with the vertical integration of LASER, Velox, and TensorFlow Serving.
  • Parameter Server: Parameter servers reduce latency and provide caching for distributed model training, rather than serving predictions.They coordinate model-parameter updates in training systems.
  • General Serving Systems: Prediction-serving systems differ from data-serving systems because computation, rather than I/O, is the dominant cost.This changes resource allocation, batching, and latency-hiding strategies.

9 Conclusion

Clipper addresses latency, throughput, and accuracy in prediction serving through a layered architecture between applications and machine-learning frameworks. Its evaluation demonstrates scalable, accurate serving with performance comparable to TensorFlow Serving.

  • Clipper interposes a layered architecture between end-user applications and machine-learning frameworks to address latency, throughput, and accuracy.
  • A common prediction interface isolates applications from framework diversity, allowing new frameworks and models without application changes.
  • The model abstraction layer achieves up to a 26x throughput improvement while maintaining strict tail-latency bounds, while model selection combines predictions for robust, contextualized outputs.
  • Across four benchmark datasets, Clipper bounded latency, scaled heavy workloads across nodes, and matched TensorFlow Serving on throughput and latency with minimal performance penalty.
Loading 1612.03079v2…