Source-linked AI summary
Serving DNNs like Clockwork: Performance Predictability from the Bottom Up
Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kaufmann, Ymir Vigfusson, Jonathan Mace
TL;DR
Model serving systems struggle to provide low-latency guarantees while sharing accelerators across many models and handling tail latency. Clockwork builds a distributed serving system around predictable DNN execution and reports superior goodput, more concurrent models, and fewer SLO violations than prior systems.
Problem
Existing model serving systems do not support sharing accelerators across many low-request-rate models while also controlling tail latency.
Method
Clockwork preserves predictable DNN performance by consolidating resource and scheduling decisions in a centralized controller coordinating predictable workers.
Results
Clockwork achieves superior goodput, serves considerably more models concurrently, and violates substantially fewer SLOs than prior model serving systems.
Takeaways & Limitations
Clockwork supports thousands of models concurrently per GPU while maintaining close to maximal possible goodput and isolating low-latency workloads from background batch workloads.
Takeaways & Limitations
Clockwork requires control or guarantees over major bottleneck resources, including exclusive worker-machine and dedicated-GPU access and mostly predictable controller-worker network latency.
Abstract
from arXiv · showhide
Machine learning inference is becoming a core building block for interactive web applications. As a result, the underlying model serving systems on which these applications depend must consistently meet low latency targets. Existing model serving architectures use well-known reactive techniques to alleviate common-case sources of latency, but cannot effectively curtail tail latency caused by unpredictable execution times. Yet the underlying execution times are not fundamentally unpredictable - on the contrary we observe that inference using Deep Neural Network (DNN) models has deterministic performance. Here, starting with the predictable execution times of individual DNN inferences, we adopt a principled design methodology to successively build a fully distributed model serving system that achieves predictable end-to-end performance. We evaluate our implementation, Clockwork, using production trace workloads, and show that Clockwork can support thousands of models while simultaneously meeting 100ms latency targets for 99.9999% of requests. We further demonstrate that Clockwork exploits predictable execution times to achieve tight request-level service-level objectives (SLOs) as well as a high degree of request-level performance isolation.
1 Introduction
Clockwork builds model serving around the predictable execution times of DNN inference, preserving predictable responsiveness across a distributed system. Its evaluation shows improved latency-goal attainment, comparable or better goodput, resource sharing, and scaling to thousands of models per worker.
- Problem: Existing model serving systems assume unpredictable component latency, allowing variability and fairqueuing to propagate tail latency across higher layers.The paper identifies software bottlenecks and execution-time variability as obstacles to consistently bounded request latency and SLOs.
- Predictability: DNN inference is a deterministic sequence of mathematical operations with predictable execution time on a GPU.Clockwork uses this observation as the foundation for predictable model serving.
- System design: Clockwork preserves predictable responsiveness through workers that exclusively load models and execute inference, plus a centralized controller that schedules requests.Workers handle one or more GPUs and immediately abort infeasible schedules before resuming the next request at its specified time.
- Evaluation: Compared with Clipper and INFaaS, Clockwork more effectively meets latency goals while providing comparable or better goodput.The evaluation uses a wide range of DNN models and production workload traces.
- Evaluation: Clockwork shares resources more effectively between models and scales to thousands of models per worker.The reported evaluation includes unpredictable, bursty, and cold-start clients.
2 Background and Motivation
Model serving must deliver millisecond-scale latency under hard hardware-cost constraints, while tail latency remains difficult because complex systems introduce performance variability. Clockwork’s motivation is that DNN inference itself is highly predictable—though concurrent execution can trade that predictability for throughput—raising whether a distributed serving system can preserve it end to end.
- Model serving: Model serving users submit inference requests for pre-trained DNNs through an API, placing inference on the critical path of interactive applications.Latency SLOs commonly specify millisecond-scale response times, such as a 10ms average or 40ms 99th-percentile response time.
- Model serving: Specialized ML hardware is needed for interactive latency but is expensive, so systems must use GPUs efficiently across workloads that may lack sufficient request volume for dedicated resources.Existing systems rely on dedicated GPUs and extensive batching for heavily used models, whereas insufficient-volume, specialized, and experimental applications may not justify dedicated hardware.
- Low-latency inference: Tail latency is especially challenging because variability arises from scheduling, concurrency interference, power-saving modes, and network queuing across complex distributed systems.The paper frames the design challenge as meeting tight tail-latency SLOs under resource constraints.
- Observation: DNN inference is predictable: DNN executions exhibit negligible latency variability because inferences have no conditional branches, and this observation extends beyond GPUs to TPUs and, where appropriate, CPUs.A DNN inference is conceptually deterministic: it processes a fixed-size input through a predefined sequence of tensor multiplications and activation functions.
- Observation: DNN inference is predictable: 0.03% was the maximum gap between the 99.99th-percentile and median latency for 11 million isolated ResNet50v2 inferences on a Tesla V100 GPU.The experiment used TVM 0.7, random inputs, and batch size 1; concurrent execution gained up to 25% throughput but substantially increased latency variability.
- Motivation: The resulting systems question is whether measured and accurately predicted DNN execution times can preserve predictable responsiveness in distributed model serving.This question motivates building upward from predictable core inference execution.
3 Predictable Performance
Clockwork pursues predictable performance by designing the system bottom up and restricting lower-layer choices that introduce variability. This makes predictable execution the common case, allowing rare unpredictability to be treated as an error rather than handled solely through reactive mechanisms.
- Sources of variability: System performance variability is primarily governed by how constituent components are assembled, and throughput or average-latency optimizations do not fix tail latency.Variability can originate in application, operating-system, hardware, network, and workload layers.
- Sources of variability: Worst-case execution, redundant parallel work, and reactive feedback mechanisms trade resource utilization or responsiveness for reduced variability.Feedback mechanisms include autoscaling, request throttling, and load balancing after unusual delays.
- Consolidating choice: Clockwork restricts choices available to lower system layers because predictable tasks became variable when lower layers could choose how to execute them.This bottom-up design applies across the systems stack, including hardware and operating-system behavior.
- Consolidating choice: By consolidating choices in upper layers, Clockwork narrows lower-layer execution paths so upper layers can predict performance, resource utilization, and request execution times.The resulting layer’s performance becomes nearly deterministic, though the strategy has a cost.
- Imperfect predictability: The design does not require perfect predictability: it aims to make predictable executions the common case and treat rare residual unpredictability as an error.Unpredictable components such as CPU-cache management and workload shifts can remain after upper-layer choice consolidation.
4 Design
Clockwork consolidates performance-critical decisions in a centralized controller and uses predictable worker actions to make model serving performance predictable. Its design explicitly manages memory, scheduling, batching, and GPU execution while limiting hardware and state-induced variability.
- Architecture: Clockwork uses a centralized controller with workers that provide predictable performance, placing the most performance-critical execution choices in the topmost layer.The controller centrally queues requests and maintains a global view of workers and their state.
- Scheduling: The controller estimates LOAD plus INFER duration against request SLOs, schedules loaded models immediately, batches requests when deadlines allow, and cancels requests that cannot meet their deadlines.Workers execute one INFER and one LOAD action at a time, enabling the controller to sequence actions explicitly.
- Timing model: 8 ms and 3 ms are the approximate LOAD and INFER times, respectively, for ResNet50.These measurements illustrate why explicit controller scheduling can account for model-loading and inference costs.
- Action abstraction: Each controller-issued action, including LOAD and INFER, has a predicted execution time and designated execution window derived from worker state and prior actions.The action-command abstraction communicates worker state changes or tasks instead of relying on traditional RPC calls.
- Sources of variability: Clockwork addresses unpredictability from memory and hardware schedulers by exposing explicit resource actions and restricting execution, while external interference remains a source of variance requiring tolerance.Managed memory and caches, hardware interactions, and external factors are identified as distinct challenges; one-at-a-time GPU execution has closely comparable throughput to concurrent execution.
- GPU execution: Two orders of magnitude is the reduction in performance variability achieved by running a single EXEC at a time, with only minimal inference-throughput loss.A single DNN inference can efficiently utilize the GPU while restricting the hardware scheduler to one predictable option.
5 Implementation
Clockwork’s 26KLOC implementation consolidates scheduling and resource-management choices in its controller by using a custom predictable runtime and explicitly managed worker state. Its workers execute timestamped actions with preallocated memory, while the scheduler selects models and batch sizes using execution profiles and deadline-aware queues.
- 26KLOC of C++ implements Clockwork’s design decisions to consolidate choice in its controller.
- Clockwork uses its own model runtime, reusing key components of TVM, instead of delegating scheduling and memory management to existing execution frameworks.
- Model loading: Workers pre-load serialized models into main memory; 768GB RAM can support thousands of models, while JIT compilation and CUDA-kernel caching are disabled for predictability.
- Managing model weights in memory: Clockwork preallocates GPU memory, executes models one at a time, and uses paging so the controller can represent worker memory by total free pages.Paging eliminates external fragmentation and does not affect memory-transfer latency.
- Actions: Each worker-GPU has dedicated action-type executors that process actions by earliest timestamp and reject actions whose latest timestamp has passed.INFER actions use dedicated INPUT, EXEC, and OUTPUT executors; EXEC checks that weights and inputs are present before running GPU kernels.
- Managing worker state and Scheduling INFER: The controller’s scheduler tracks memory state, profiled durations from the past 10 actions, and pending actions to predict execution and coordinate workers.Scheduling selects models and batch sizes using per-model batch queues, prioritizing larger batches while dropping requests that are no longer satisfiable.
6 Evaluation
Clockwork’s evaluation shows that predictable execution enables reliable low-latency serving across controlled, heterogeneous, and realistic workloads. It meets tight SLOs, sustains high goodput across thousands of models, and adapts to shifting bottlenecks without substantial interference.
- Workload scaling: 100 ms: Clockwork’s maximum request latency never exceeded the SLO while serving 201 activated models and shifting GPU, memory, and PCIe bottlenecks.At 3.5 minutes, 201 models reached GPU memory capacity, after which model swapping drove PCIe utilization to 100%.
- SLO performance: 10 and 22 ms: Clockwork satisfied tight SLOs at 600 and 1200 r/s regardless of model count, and managed 74 ms at 2400 r/s.The reported satisfaction metric equals 1 when all requests complete successfully within their SLO.
- Realistic workloads: 9,638 r/s: Across a six-hour Azure Functions trace, Clockwork averaged matching offered load and goodput, with 58 failures among 208 million requests and no timeouts.All GPUs remained fully utilized, yet no request exceeded the 100 ms SLO.
- Realistic workloads: 987 models: Approximately 25% of models performed cold starts each minute, but cold-start requests averaged 126 r/s, or 1.3% of all requests.The results demonstrate sustained load across varied realistic workloads comprising thousands of models.
- System scalability: 103,387 r/s: Peak goodput increased linearly with worker count until 110 workers, when the bottleneck shifted from worker utilization to Clockwork’s controller.The reported peak goodput is the median across three experiment repetitions.
7 Discussion
The discussion identifies Clockwork’s scope and deployment limitations while outlining how predictable execution could extend to pipelines, other accelerators, and additional guarantees. It also notes assumptions about resource control, network behavior, security, and fault tolerance.
- Scope and extensions: Clockwork excludes user-defined, CPU-bound data preprocessing and postprocessing; safely and predictably executing them remains a research topic.The system currently focuses on DNN inference rather than the surrounding application-defined processing steps.
- Scope and extensions: Performance predictability could support end-to-end guarantees for DNN pipelines and cascades and influence designs for embedding models, reinforcement learning, and training.The paper identifies more sophisticated pipeline scheduling and broader questions about the nature and limits of predictability.
- Hardware: Clockwork’s approach generalizes beyond GPUs to inference-specific accelerators, while CPUs remain suitable for some models such as sequential RNNs.TPUs are cited as an example of hardware emphasizing software control, high-level operations, and explicit memory hierarchies.
- Limitations: Predictability depends on controlling major bottleneck resources: Clockwork assumes exclusive worker-machine and GPU control and mostly predictable controller-worker network latency.Shared settings make preserving predictability more challenging.
- Limitations: 12 workers reached the testbed network’s limit under a sustained single-model workload because the prototype routes all inputs and outputs through its central controller.The experiment disabled inputs to test beyond that network limit.
- Limitations: Clockwork does not address security or fault tolerance, has not verified safety properties, and would require a fault-tolerant centralized scheduler for large-scale serving.Its predictable worker design could nevertheless make grey failures easier to detect.
8 Related Work
Related model-serving systems trade away execution predictability by abstracting over backend frameworks or relying on cloud-provider control, whereas Clockwork uses backend predictability and latency SLOs for proactive scheduling. Real-time systems provide related predictability goals but generally assume periodic or sporadic workloads, known request sets, or known timing bounds.
- Model serving: Clipper and INFaaS wrap existing execution frameworks, sacrificing predictability and control over model execution.Clipper provides a unifying abstraction, while INFaaS exploits heterogeneous execution strategies.
- Model serving: Unlike Clockwork, virtualized-cloud and serverless model-serving systems do not use backend predictability and latency SLOs for proactive scheduling.These systems jointly consider model throughput, latency, and accuracy for model selection.
- Real-time systems: Real-time systems typically assume periodic or sporadic workloads with known minimum inter-arrival times and worst-case execution times, or inference requests known in advance.Soft-real-time systems consider weaker timeliness notions similar to latency SLOs.
9 Conclusion
Clockwork addresses growing model-serving demands by meeting aggressive tail-latency SLOs while supporting thousands of diverse DNN models concurrently on each GPU and scaling across worker machines.
- Conclusion: Clockwork fulfills aggressive tail-latency SLOs for interactive applications.The paper frames this capability as addressing increasingly tight response-time requirements.
- Conclusion: Clockwork supports thousands of DNN models with different workload characteristics concurrently on each GPU.This enables concurrent service for a growing number of models despite differing workload characteristics.
- Conclusion: Clockwork scales out to additional worker machines for increased capacity.The system is designed to expand capacity beyond individual GPUs by adding worker machines.
A Artifact Appendix · A.1 Abstract
The artifact provides Clockwork’s source code, environment setup, and experiment automation needed to build the system and reproduce the paper’s graphs and results.
- A.1 Abstract: The artifact includes Clockwork’s prototype source code.It supports building the prototype from source.
- A.1 Abstract: The artifact provides instructions for building Clockwork from source.
- A.1 Abstract: The artifact includes directions for preparing the experimental environment.
- A.1 Abstract: A Docker instance with all dependencies pre-installed can be launched using the provided instructions.
- A.1 Abstract: Scripts, descriptions, and instructions support running the experiments automatically or manually.
- A.1 Abstract: The artifact enables reproducing the graphs and results presented in the paper.
A.2 Artifact check-list
The Clockwork artifact includes the system and its compiled models and dataset, uses permissive open-source licenses, and includes an attributed Azure Functions trace.
- Artifact size: Clockwork occupies 210MB, while the compiled models and dataset total 12GB.The checklist reports these artifact size figures separately.
- Licenses: Clockwork, TVM, and the CUDA Common Library use the Apache License 2.0, while Catch2 uses the Boost Software License 1.0.These are the licenses listed in the artifact checklist.
- Dataset: The Azure Functions Trace 2019 is released under CC-BY Attribution.The checklist identifies its attribution license.
A.3 Description … A.8 AE Methodology
The artifact appendix specifies the hardware, software, data, installation, workflow, evaluation, customization, and methodology needed to reproduce Clockwork’s experiments. It provides scripts, documentation, and environment requirements, including a 12-worker-machine Azure Functions experiment.
- A.3.2 Hardware dependencies: Exact reproduction requires machines with at least 768GB RAM, 16 CPU cores, one 32GB Tesla V100 GPU, and a 10Gbps network.The large-scale Azure Functions experiment requires 12 worker machines; most other experiments require fewer.
- A.4 Installation: A Dockerfile is provided to facilitate the build process.The artifact also supplies separate documentation for prerequisites, building, environment setup, and configuration.
- A.3.4 Data sets: The Azure Functions data set is available from the project’s tracedatasets repository.The appendix links directly to the data-set location used by the artifact.
- A.5 Experiment workflow: Experiments can be run with repository scripts or manually, with manual setup recommended first to understand system components and verify configuration.The experiments repository includes the scripts and instructions for both workflows.
- A.6 Evaluation and expected results: Table 3 organizes each reproduction experiment by paper figure, execution time, documentation, and scripts.The experiments repository is structured according to Section 6 and links each experiment to its corresponding description and instructions.
- A.7 Experiment customization: Each experiment’s documentation provides manual execution directions, while environment customization instructions are supplied separately.The customization guide is hosted in the Clockwork repository.
- A.8 AE Methodology: The submission, reviewing, and badging methodology is documented by the USENIX OSDI 2020 artifact-evaluation call.The appendix provides the corresponding methodology link.