Source-linked AI summary
TensorFlow-Serving: Flexible, High-Performance ML Serving
Christopher Olston, Noah Fiedel, Kiril Gorovoy, Jeremiah Harmsen, Li Lao, Fangwei Li, Vinu Rajashekhar, Sukriti Ramesh, Jordan Soyke
TL;DR
Production deployment of ML models has received less systematic attention than training, despite growing complexity from versioning, multiple models, and hardware acceleration. TensorFlow-Serving addresses this gap with a modular, platform-agnostic framework spanning lifecycle management, inference, batching, canonical binaries, and hosted service. It is open-source and used widely inside Google, including through the TFS2 hosted service.
Problem
Production ML deployment has received little systematic attention despite increasing serving complexity from model versions, multiple models, and hardware acceleration.
Method
TensorFlow-Serving combines modular lifecycle and inference libraries, a canonical binary, and a hosted service with flexible model integration and performance optimizations.
Results
TensorFlow-Serving handles about 100,000 requests per second per core when RPC and TensorFlow layers are factored out, while substantially reducing tail-latency problems during model loading.
Takeaways & Limitations
The library, binary, and service support production systems inside Google, with hundreds of projects generating tens of millions of inferences per second.
Takeaways & Limitations
The core libraries are ML-platform-agnostic, but the system’s other layers contain little TensorFlow-specific logic and would require generalization for broader use.
Abstract
from arXiv · showhide
We describe TensorFlow-Serving, a system to serve machine learning models inside Google which is also available in the cloud and via open-source. It is extremely flexible in terms of the types of ML platforms it supports, and ways to integrate with systems that convey new models and updated versions from training to serving. At the same time, the core code paths around model lookup and inference have been carefully optimized to avoid performance pitfalls observed in naive implementations. Google uses it in many production deployments, including a multi-tenant model hosting service called TFS^2.
1 Introduction
TensorFlow-Serving addresses the growing gap between extensive ML training infrastructure and comparatively underdeveloped production serving. It provides a reusable framework spanning flexible deployment options, performance-conscious serving, hosted access, and open-source availability.
- Motivation: Production ML serving received little systematic attention compared with model training, while ad-hoc systems became difficult to reuse as applications matured.Supporting versioning, experimentation, availability, latency, acceleration, and larger models increased serving complexity.
- System scope: TensorFlow-Serving comprises a modular C++ library, a canonical server binary, and a hosted service for TensorFlow and other ML models.The library supports composing or customizing modules for different lifecycle, model, and storage requirements.
- Hosted service: The hosted service codifies practices such as validating model quality before serving new versions and logging inferences to detect training/serving skew.These practices were previously not widely adopted across teams.
- Flexibility: The framework supports production variations including in-place version transitions, multiple model types, and different model-storage or data-conveyance mechanisms.These variations are realized through module configuration, composition, and custom implementations.
- Availability: The libraries and binary are open-source, while the hosted service is available through Google Cloud Platform.The service is also available to teams within Google.
- Platform scope: TensorFlow-Serving treats models as black boxes through its core libraries, making the system largely ML-platform-agnostic.The other layers contain little TensorFlow-specific logic and could be generalized.
2 Library
The TensorFlow-Serving library separates model lifecycle management from inference and connects extensible modules through APIs. It supports flexible model discovery, version transitions, inference interfaces, and batching while incorporating optimizations aimed at tail latency and resource use.
- Library architecture: The library manages which model versions reside in memory and serves inference requests, with optional cross-request batching.Its two parts are lifecycle-management modules and inference-serving modules.
- Model lifecycle management: A lifecycle chain uses Sources, Source Routers, Source Adapters, and a Manager to discover, transform, sequence, and expose model versions.Most modules treat models as generic black-box servables, including non-TensorFlow objects such as lookup tables.
- Model lifecycle management: The aspired versions API lets Sources declare which servable versions should be memory-resident through a uni-directional, idempotent interface.The API is templated by the data type passed with each version and supports adapter chains and custom components.
- Version transitions: Production deployments can support canarying and rollback, while AspiredVersionsManager chooses availability-preserving or resource-preserving version transitions.The resource-preserving policy addresses models for which two versions cannot fit in memory simultaneously.
- Performance: AspiredVersionsManager uses wait-free servable access, manager-thread memory freeing, memory release on unload, and isolated load and inference thread pools.These choices target latency hiccups and interference between loading and inference.
- Inference and batching: The library provides TensorFlow RPC interfaces, standardized tf.Example support, logging, and batching primitives for multiple servables or versions.Batching can share a device such as a GPU through dynamic queues scheduled round-robin.
- Inference and batching: Batch and Unbatch graph operations can selectively batch GPU/TPU subgraphs, sequence-model loop bodies, or separate encode and decode subgraphs.This approach is described as more flexible than batching complete Session::Run() calls, though it was not yet fully vetted.
3 Canonical Binary and Hosted Service
TensorFlow-Serving packages a common file-based serving configuration as a binary and extends the abstraction to a hosted service, TFS2, that manages serving jobs on users’ behalf. TFS2 supports multi-tenant model lifecycle operations, resource-aware placement, scaling, and production-oriented partitions.
- Canonical binary: A vanilla setup combines a file-system-monitoring Source, a TensorFlow Source Adapter, and a Manager, and is packaged as a binary for simpler deployment.This avoids requiring most users to configure the lower-level library directly.
- Hosted service: TFS2 raises the serving abstraction from running model-serving jobs to serving models while managing the jobs for users.The hosted service is intended to free users from running jobs themselves.
- Hosted service: TFS2 accepts high-level commands such as adding or removing models and versions, then assigns models to serving jobs based on resource fit.Its Controller manages updates, canaries, rollbacks, RAM estimation, and transactional state.
- Hosted service: TFS2 disseminates models through Synchronizer jobs and routes inference requests according to which models are successfully loaded.The system coordinates model state across configured data centers.
- Deployment: Separate Temp and Prod instances support experimentation and robust production serving, with partitions specialized by hardware or geography.Partitions include hardware such as TPUs and locations such as South America.
- Deployment: TFS2 serving jobs use the same binary provided for standalone deployment, reducing maintenance and enabling staged binary-release canaries.Releases can be canaried in Temp before broader rollout.
- ML infrastructure: TensorFlow-Serving integrates with training, quality validation, robustness validation, and training/serving-skew detection pipelines.Successful model versions can be injected into standalone serving jobs or TFS2.
4 Project Status, Performance and Adoption
TensorFlow-Serving progressed from project launch to broad production adoption, while delivering high throughput and substantially reduced tail-latency interference during model loading.
- Project Status: TensorFlow-Serving was launched incrementally, with the library open-sourced in winter 2016 and production TFS^2 following in winter 2017.The binary entered Cloud Machine Learning Engine in fall 2016; Temp and Prod TFS^2 instances launched in fall 2016 and winter 2017.
- Performance: 100,000 requests per second per core is achievable when RPC and TensorFlow layers are factored out.The remaining throughput bottlenecks lie in those two layers.
- Performance: Tail latency was substantially reduced during concurrent model or version loading compared with the initial naive implementation.The paper attributes this improvement to the optimized serving system and reports further details elsewhere.
- Adoption: The library, binary, and service form-factors are all used in Google production systems and user-facing products.Adoption spans hundreds of Google projects and tens of millions of inferences per second overall.