HOLMES: Health OnLine Model Ensemble Serving for Deep Learning Models in Intensive Care Units

1 INTRODUCTION

• ICU serving must deliver accurate predictions at low latency over dynamic streams of multi-patient, multimodal data, a requirement prior accuracy-focused work largely overlooked.Source: §1 · p. 1
• HOLMES reaches competitive accuracy within a 200 ms latency budget while balancing accuracy (ROC-AUC) against latency.Source: Figure 1 · p. 1
• ICU tasks impose different priorities: length-of-stay and discharge predictions support resource optimization, while rapidly changing conditions make latency crucial for mortality prediction.Source: §1 · p. 2
• ICU serving is complicated by disparate data frequencies, unreliable sensor collection, heterogeneous patient needs, and limited computing resources.A unified model across modalities is described as unnecessary and nearly impossible; heavy, inflexible models can be difficult to deploy.Source: §1 · p. 2
• 95% accuracy for stepdown readiness prediction with sub-second latency, while HOLMES simultaneously serves streaming data for up to 100 beds.The experiments use a realistic simulated streaming environment.Source: §1 · p. 2
• HOLMES combines a model zoo of specialized models with an ensemble composer and a real-time serving system to explore accuracy/latency tradeoffs for clinical use cases.The composer selects a subset of models under computational-resource and latency constraints.Source: §1 · p. 2

2 RELATED WORK

• HOLMES combines automatic ensemble construction from specialized models with a latency-aware system that can serve clinical use cases across differing prediction-latency requirements.Source: §2 · pp. 2–3
• Prior ICU deep-learning work commonly builds unified models for all patients and modalities, which the paper describes as inflexible to data heterogeneity and not yet serving real-world deployment.Source: §2 · p. 3
• TensorFlow Serving primarily supports single-model serving, whereas clinical pipelines need framework-level composition of multiple heterogeneous models; it also does not easily support models trained in frameworks such as PyTorch.Source: §2 · p. 3
• HOLMES builds ML pipeline-serving functionality on Ray, whose stateful and stateless computation primitives are suited to serving; the paper contrasts this with Spark MLlib for soft real-time use.Source: §2 · p. 3
• HOLMES composes models as building blocks rather than neural-network operators, extending Bayesian optimization to sequential model-based optimization and adding genetic exploration.Source: §2 · p. 3

3.1 Overview

• HOLMES composes and serves model ensembles through a model zoo, an ensemble composer, and a real-time serving system.The composer optimizes validation accuracy subject to latency constraints and selects a model set given patient count and system resources.Source: §3.1 · p. 3

3.2 Model Zoo

• The model zoo contains specialized models for individual modalities and time windows, including smaller models that can fit limited resources.The composer can be triggered by changes to the zoo, patient count, or resource constraints; new modalities can be added without retraining the entire monolithic model.Source: §3.2 · p. 4

3.3 Ensemble Composer

• HOLMES searches for the highest-accuracy ensemble that meets a specified latency threshold, addressing the potentially prohibitive latency of large ensembles.Ensemble composition is formulated as an optimization problem and solved with an efficient search algorithm in a Bayesian optimization framework.Source: §3.3 · p. 4

3.3.1 Problem setup.

• The composer selects a binary model-subset vector to maximize validation accuracy while respecting a latency constraint under a system configuration.The accuracy profiler evaluates a selected ensemble, while the latency profiler estimates its latency under the specified configuration.Source: §3.3.1 · p. 4

3.3.2 Ensemble Composer exploration.

• Bayesian optimization approximates accuracy and latency profilers with surrogate models, while genetic search explores the high-dimensional binary ensemble space.The algorithm iteratively adds candidates with true profiled values to its evaluated set and returns the best objective value found when its profiler-call budget is reached.Source: §3.3.2 · p. 5

3.3.3 Prediction ensemble.

• HOLMES makes its final prediction with a bagging ensemble of the selected models’ predictions for an ICU input.The outcome measure is Y, and x denotes an instance of ICU data input.Source: §3.3.3 · p. 5

3.4 Real-time Serving

• The serving system profiles candidate ensemble latency and serves the deployed pipeline, buffering streaming patient data before ensemble queries.Network calculus estimates queueing latency using the maximum horizontal distance between arrival and service curves.Source: §3.4 · pp. 5–6

4.1.1 Model zoo training.

• The CICU model zoo contains 60 ECG models, alongside random-forest vital-sign models and logistic-regression lab models used in ensemble predictions.The 60 ECG models vary by lead, filter count, and residual-block count.Source: §4.1.1 · pp. 6–7

4.2.1 Overall performance compared with baselines.

• Under a 200 ms latency constraint, HOLMES achieves the best performance across all reported measurements.Table 2 summarizes the methods’ prediction performance under this constraint.Source: §4.2.1 · p. 7, Table 2 · p. 7

4.2.2 Tradeoff space exploration efficacy.

• HOLMES reaches the 200 ms latency constraint quickly and continues improving its ensemble, while random, accuracy-first, and latency-first searches exceed the budget and NPO misses optimal ROC-AUC.Across fixed latency budgets, HOLMES also outperforms NPO in ROC-AUC; its search trajectory stays on the Pareto frontier.Source: §4.2.2 · p. 7, Figure 6 · p. 7, Figure 7 · p. 8
• As exploration steps increase, both surrogate models’ R2 scores improve, helping HOLMES predict promising ensembles before profiling them.The R2 scores compare predicted accuracy and latency with profiler measurements on an independent validation set.Source: §4.2.2 · p. 8, Figure 8 · p. 8

4.3.1 Online vs. offline inference latency.

• For a single patient over 60 minutes, conventional batching incurs inference latency an order of magnitude higher than HOLMES.HOLMES reevaluates every 30 seconds, with inference spikes up to the order of 10^-1 seconds.Source: §4.3.1 · p. 8, Figure 9 · p. 8

4.3.2 Varying resource constraints.

• At 64 patients generating 16,000 queries per second, HOLMES serves a 10-model ensemble within 1.15 seconds at the 95th percentile.At fixed 16,000 queries per second, two GPUs provide lower latency than one GPU.Source: §4.3.2 · p. 8, Figure 10 · p. 8

5 CONCLUSION

• HOLMES navigates the accuracy/latency tradeoff to achieve the highest accuracy within specified sub-second latency targets.The conclusion reports that HOLMES outperforms conventional offline batched inference in both clinical prediction accuracy and latency by an order of magnitude.Source: §5 · p. 9

A.1 Exploring in HOLMES using genetic search

• The procedure takes a candidate set B, sample count N1, mutation degree S, and probabilities for genetic exploration and mutation.Source: §A.1 · p. 10
• Each iteration randomly picks three candidates from B and can take a random-exploration branch.Source: §A.1 · p. 10
• The procedure also includes recombination and mutation exploration and avoids adding duplicates.Source: §A.1 · p. 10

• Figure 11 compares ROC-AUC and latency across iterations for different exploration algorithms.Source: Figure 11 · p. 10

A.3 Comparison of accuracy and latency from the optimal ensemble selected by different algorithms

• Under a 0.2-second latency constraint, HOLMES selects a higher-accuracy ensemble than the compared method.The figure caption also reports comparable latency utility between HOLMES and Latency First (LF).Source: Figure 12 · p. 10
• Increasing the observation window causes a small increase in latency, and Figure 13 distinguishes model execution, serving delay, queueing delay, and end-to-end latency.Timeit denotes plain PyTorch GPU execution; TS is serving delay, TQ is queueing delay, and TQ + TS is end-to-end latency.Source: §A.3 · p. 10
• Figure 13 presents the effects of different history aggregation choices.Source: Figure 13 · p. 10

A.5 Deep model description in the Model Zoo.

• Table 3 presents descriptions of deep models in the Model Zoo.Source: Table 3 · p. 10

A.6 An Alternative Formulation of Accuracy Sensitive Constraint

• The main formulation maximizes accuracy subject to a latency upper bound L.Source: §A.6 · p. 11
• For accuracy-sensitive tasks, the objective function and constraint in Eq. (1) can be switched.Source: §A.6 · p. 11
• The alternative formulation uses A as the minimum accuracy for the ensemble and is beyond the paper's scope, though the same search algorithm can solve it equivalently.Source: §A.6 · p. 11