PRETZEL: Opening the Black Box of Machine Learning Prediction Serving Systems

1 Introduction

• At inference time, pipelines serve direct user requests and therefore need low latency, high throughput, and graceful performance degradation during load spikes.The trained pipeline is deployed as a whole, although its operators have different system characteristics across training and inference.Source: §1 · p. 1
• Black-box serving emphasizes ease of deployment and offers pipeline-agnostic optimizations, while model density can raise memory pressure and co-location can increase tail latency.Similar pipelines may share operators because of A/B testing, personalization, and fine-tuning.Source: §1 · p. 2
• PRETZEL compiles pipelines into white-box model plans that reuse parameters and computation across similar plans, then pools memory and CPU resources and schedules requests to execution units.The design applies end-to-end and multi-pipeline optimizations to trained pipelines.Source: §1 · p. 2
• On 500 production-like pipelines, PRETZEL improved memory footprint by 25×, reduced 99th percentile latency by 5.5×, and increased throughput by 4.7× on average.The comparisons were with ML.Net and end-to-end solutions such as Clipper.Source: §1 · p. 2
• The paper contributes an analysis of black-box serving limitations, design principles for white-box serving, a system implementation, and an experimental evaluation.The principles target inference optimization and resource sharing across pipelines.Source: §1 · p. 2

• Black-box deployment hides pipeline structure and prevents principled sharing of optimizations or control over end-to-end execution.Source: §2 · p. 3

Memory Waste

• Containerization prevents sharing runtimes and resources, even though 250 sentiment-analysis variants reuse common operators and many Ngram configurations.Source: §2 · pp. 3–4

Prediction Initialization

• Hot predictions with memory allocated and code JIT-compiled are more than two orders of magnitude faster than the worst cold case.Source: §2 · p. 4, Figure 4 · p. 4
• 57.4% of cold-prediction time goes to pipeline analysis and function-chain initialization, and 36.5% to JIT compilation, leaving the remainder for computation.Source: §2 · p. 4

Infrequent Accesses

• Evicting infrequently accessed pipelines saves residency but makes later requests incur loading and warm-up penalties that can violate SLAs.Source: §2 · p. 4

Operator-at-a-time Model

• Operator-at-a-time execution ignores interactions between operators; in the sentiment-analysis example, pipelining linear regression with n-gram operators can eliminate Concat and intermediate buffers.Source: §2 · p. 4

Coarse Grained Scheduling

• Coarse container-level scheduling can replicate popular pipelines while leaving less popular ones underutilized, reducing total resource utilization.Source: §2 · pp. 4–5

White Box Prediction Serving

• A white-box runtime can co-locate pipelines, share resources, keep unpopular pipelines warm, and use reservations when a pipeline needs exclusive resources.Source: §3 · p. 5

End-to-end Optimizations

• End-to-end computation units should avoid data-path allocation, separate per-operator routines when possible, and prediction-time reflection or JIT compilation.Source: §3 · p. 5

Multi-model Optimizations

• Multi-model optimization should store shareable components once, share execution units, pool resources, and reuse partial results such as featurization outputs.Source: §3 · p. 5

• PRETZEL treats models as database queries, using DAG optimization, multi-query optimization, view materialization, and shared vector and thread pools.Source: §4 · p. 5
• Flour, Oven, the Object Store, Runtime, Scheduler, and FrontEnd translate pipelines, optimize plans, share parameters, execute plans, schedule work, and submit requests.Source: §4 · p. 5
• Offline, PRETZEL translates ML.Net pipelines into Flour transformations, fuses and AOT-compiles stages, then registers plans with shared physical stages and parameters; online, the Scheduler binds stages to shared execution units.Source: §4 · pp. 5–6

4.1.1 Flour

• Flour provides a language-integrated DAG representation between ML frameworks and PRETZEL; optional training statistics help the compiler tailor physical plans to model characteristics.Statistics include maximum vector size and dense or sparse representations.Source: §4.1.1 · p. 6

4.1.2 Oven

• Oven splits stages at Concat or n-to-1 operations requiring full materialization, then rewrites stage graphs to merge equal transformations and remove unnecessary branches.It also pushes linear models through Concat when possible.Source: §4.1.2 · p. 7
• Each logical stage maps to a parameterized, lock-free AOT physical computation, enabling implementations to be shared across pipelines without prediction-path memory allocation.Logical plans also retain pipeline-specific parameters and statistics, including maximum vector size.Source: §4.1.2 · p. 8

4.1.3 Object Store

• The Object Store reuses parameters already present across plans, identifying equality by checksums of serialized objects.Shared state such as language dictionaries can reduce memory footprint.Source: §4.1.3 · p. 8

4.2.1 Runtime

• At registration, identical physical stages load once; preinitialized vector pools and Executor thread pools reduce memory and thread allocation time during prediction.Source: §4.2.1 · p. 8
• Single predictions use request-response, which inlines a JIT-compiled plan on the handling thread; batch requests route stages to the Scheduler for placement by runtime and resource status.The engine choice is configured at plan registration.Source: §4.2.1 · p. 9

4.2.2 Scheduler

• Executors pull stage events from low-priority new-plan and high-priority started-stage queues; vectors are allocated per pipeline and retained through its execution.The priority queues let started pipelines run earlier and return memory sooner.Source: §4.2.2 · p. 9
• Reservation-based scheduling gives plans exclusive pipeline-specific memory or computation pools, while parameters and physical stage objects remain shared.Source: §4.2.2 · p. 9

4.3 Additional Optimizations

• PRETZEL caches results of shared physical stages, checks reuse by hashing inputs, and evicts entries with LRU when a memory threshold is reached.Source: §4.3 · p. 9
• The FrontEnd supports prediction-result caching with LRU eviction as an external optimization.Source: §4.3 · p. 9
• Delayed batching buffers inference requests for a user-specified interval before submitting them together; these external optimizations are complementary to PRETZEL’s other techniques.Source: §4.3 · p. 9

5 Evaluation

• The evaluation uses 250 SA and 250 AC pipelines; SA pipelines benefit from sub-plan materialization, while the more diverse AC pipelines do not.The AC task predicts event attendance; its most complex pipeline combines several models and featurizers.Source: §5 · p. 10

5.1 Memory

• 164MBs of memory lets PRETZEL load all 250 AC pipelines, using about 25× less than ML.Net and 62× less than ML.Net + Clipper; only PRETZEL with Object Store loads all SA pipelines.The comparison measures the memory needed to load all 250 pipelines in each category.Source: Figure 8 · p. 10, §5.1 · p. 10
• 2.8 seconds lets PRETZEL load 250 AC pipelines, versus 270 seconds for ML.Net; PRETZEL loads 250 SA pipelines in 37.3 seconds, while ML.Net swaps after 75 in about 9 minutes.For the other configurations, exceeding 32GB leads to disk swapping and system instability.Source: §5.1 · pp. 10–11

5.2 Latency

• The latency evaluation measures isolated single-model requests, comparing hot predictions with first-request cold predictions and separately measuring client HTTP latency.The experiments also include prediction-only micro-benchmarks and end-to-end comparisons.Source: §5.2 · p. 11, §5.2.1 · p. 11, §5.2.2 · p. 11

5.2.1 Micro-benchmark

• PRETZEL is 3.2× (SA) and 3.1× (AC) faster than ML.Net at P99hot, and about 9.8× (SA) and 5.7× (AC) faster at P99cold.The comparison measures hot and cold predictions on a single core; Figure 9 annotates P99 values.Source: §5.2.1 · p. 11, Figure 9 · p. 11
• 460.6× (ML.Net) versus 33.3× (PRETZEL) is the SA worst-case latency relative to P99hot; for AC, the corresponding values are 21.2× and 7.5×.For the P99cold/P99hot ratio, ML.Net is about 13.3× (SA) and 4.6× (AC), versus about 4.2× and 2.5× for PRETZEL.Source: §5.2.1 · p. 11
• 2.0× average latency improvement from sub-plan materialization applies to SA pipelines, with no pipeline showing deterioration.Around 80% of SA pipelines show more than 2× speedup; the technique does not apply to AC pipelines.Source: §5.2.1 · p. 11, Figure 10 · p. 11

5.2.2 End-to-end

• 4.3ms (PRETZEL) versus 9.3ms (ML.Net + Clipper) is SA client-observed P99 latency; for AC, the values are 7.3ms and 18.0ms.Figure 11 also compares end-to-end latency with prediction-only latency.Source: §5.2.2 · p. 12, Figure 11 · p. 12

5.3 Throughput

• The throughput micro-benchmark scores all 500 models in batches of 1,000, allocating 2–13 CPU cores to serve requests and 3 cores to generate them.Source: §5.3 · p. 12
• 2.6× (SA) and 10× (AC) higher throughput than ML.Net; PRETZEL scales on par with ideal scaling as CPU cores increase.Figure 12 plots queries per second against CPU core count.Source: §5.3 · p. 12, Figure 12 · p. 12

5.4 Heavy Load

• The heavy-load test submits requests according to a Zipf distribution with α = 2, increasing load to measure throughput and latency changes.The experiment first measures PRETZEL alone, then compares systems end to end.Source: §5.4 · p. 12, §5.4.1 · p. 12

5.4.1 Micro-benchmark

• About 25k queries per second is where PRETZEL’s throughput stabilizes under increasing load, while latency for latency-sensitive pipelines rises linearly.All 500 models are in memory; half receive batch-size-1 requests and half receive batches of 100.Source: §5.4.1 · p. 12, Figure 13 · p. 12
• One reserved core and its vectors prevent latency degradation for the reserved model as load rises, with a maximum improvement of 3 orders of magnitude and similar system throughput.Source: §5.4.1 · p. 13

5.4.2 End-to-end

• PRETZEL throughput keeps increasing to around 300 requests per second in the end-to-end test; above that point, throughput and latency fluctuate.The test compares PRETZEL with ML.Net + Clipper using 250 AC pipelines.Source: §5.4.2 · p. 13, Figure 14 · p. 13

Off-line Phase

• Multiple logical and physical stage classes make maintenance harder, and each backend requires its own operator implementations.Source: §6 · p. 13
• PRETZEL currently supports only ML.Net-authored pipelines, and porting models from other frameworks may require non-trivial work.The authors aim to target unified formats such as ONNX.Source: §6 · p. 13

On-line Phase

• Fine-grained stage scheduling can add buffering and context-switching overhead; garbage collection can also cause latency spikes and long-tail latencies.The authors also note model failures may jeopardize the system and the current runtime is single-node.Source: §6 · p. 13

Prediction Serving

• Prediction-serving systems including Clipper, TensorFlow Serving, and Rafiki focus on deployment and reuse, while PRETZEL applies end-to-end and multi-query optimizations.Clipper and Rafiki use Docker containers connected through RPC; TensorFlow Serving allows pipeline splitting, but not automatically.Source: §7 · p. 13

Optimization of ML Pipelines

• PRETZEL’s Oven optimizations are presented as generalizing across intermediate representations, while a proposed Tensor Comprehension extension could express featurization pipelines and support neural-network featurizers.The proposed extension could also add code generation for heterogeneous devices.Source: §7 · p. 14
• PRETZEL rewrites logical and physical plans; KeystoneML has a query optimizer focused on distributed training, while PRETZEL also provides logical-level optimizations.PRETZEL, KeystoneML, and MauveDB provide sub-plan materialization.Source: §7 · p. 14

Scheduling

• Clipper and Rafiki schedule requests using latency targets and ensembles; PRETZEL addresses resource sharing between models through scheduling.The discussion relates this scheduling problem to distributed scheduling in cloud computing.Source: §7 · p. 14

8 Conclusion

• An end-to-end optimizer and compiler generate efficient model plans to decrease latency, while shared parameters and operators plus concurrent scheduling target memory footprint, resource utilization, and throughput.Source: §8 · p. 14
• The paper frames prediction serving as a database problem, applying end-to-end and multi-query optimization strategies to machine-learning pipelines.Source: §8 · p. 14
• Production-like pipeline experiments show PRETZEL delivers order-of-magnitude improvements over previous approaches across performance metrics.The conclusion describes improvements across different performance metrics without giving exact values here.Source: §8 · p. 14