Source-linked AI summary
Serving deep learning models in a serverless platform
Vatche Ishakian, Vinod Muthusamy, Aleksander Slominski
TL;DR
The paper asks whether serverless computing is suitable for serving large neural-network models. It evaluates AWS Lambda with MXNet across models and start conditions, finding acceptable warm latency but cold-start delays that can distort latency and threaten stringent SLAs.
Problem
The paper examines whether a serverless computing platform can be used for neural-network inference as machine-learning adoption expands.
Method
The study measures performance and cost for three image-recognition models using MXNet on AWS Lambda under cold-start and warm-start conditions.
Results
Warm executions have acceptable latency, while cold starts add significant overhead and create a bimodal latency distribution.
Takeaways & Limitations
Serverless can support deep-learning prediction services, but cold-start behavior must be considered when stringent SLAs apply.
Takeaways & Limitations
Cold-request latency remains high because container bootstrapping adds overhead, limiting suitability for applications with stringent SLAs.
Abstract
from arXiv · showhide
Serverless computing has emerged as a compelling paradigm for the development and deployment of a wide range of event based cloud applications. At the same time, cloud providers and enterprise companies are heavily adopting machine learning and Artificial Intelligence to either differentiate themselves, or provide their customers with value added services. In this work we evaluate the suitability of a serverless computing environment for the inferencing of large neural network models. Our experimental evaluations are executed on the AWS Lambda environment using the MxNet deep learning framework. Our experimental results show that while the inferencing latency can be within an acceptable range, longer delays due to cold starts can skew the latency distribution and hence risk violating more stringent SLAs.
I. INTRODUCTION
Serverless computing offers a simplified, scalable model for cloud applications as machine learning adoption expands. This work evaluates whether serverless platforms can serve deep learning models effectively, with cold starts posing an SLA risk.
- Serverless platforms simplify application development by abstracting operational concerns, charging for execution time, and enabling rapid deployment of event handlers.
- Deep learning supports services including text analytics, natural language processing, speech recognition, and image recognition.
- Inference requires less computational power than training and uses trained neural networks without training data.
- The study measures user-perceived performance and cost for three trained MXNet models running on AWS Lambda.
- Cold-start delays can skew latency distributions and risk violating stringent service-level agreements.
A. Serverless Computing
Serverless platforms dispatch events to containerized functions and reuse containers to reduce repeated startup overhead. Deep learning models are trained separately and then used for inference on new data.
- A serverless platform receives events, dispatches functions, launches containers, returns responses, and stops functions when they are no longer needed.
- Container setup and bootstrapping add invocation latency, while container reuse reduces this cold-start overhead for subsequent calls.
- AWS Lambda also relies on container technology to provide serverless capabilities.
- Deep learning models have advanced through larger training datasets, GPU accelerators, and improved model designs.
- After training, neural networks apply learned weights to new data for tasks such as language processing, speech recognition, and image classification.
III. EXPERIMENTS
The experiments evaluate deep-learning inference on AWS Lambda with MXNet across three image-recognition models and multiple operational conditions. They measure response time, prediction time, and execution cost while distinguishing cold and warm starts.
- The evaluation tests whether serverless computing can support neural-network inference and plans comparison with other platforms and frameworks.
- Three image-recognition models represent different sizes: SqueezeNet at 5 MB, ResNet-18 at 45 MB, and ResNeXt-50 at 98 MB.
- The AWS Lambda functions package both models and images as dependencies to factor out download delays during prediction.
- AWS Lambda allocates CPU, network bandwidth, and disk I/O in proportion to selected memory, which ranges from 128MB to 1536MB.
- The study measures user-observed response time, model prediction time, and total Lambda execution cost.
- Cold starts require container initialization, whereas warm starts execute with an already-running container and avoid that initialization overhead.
A. Cold and warm evaluations
The evaluation separates cold-start and warm-start measurements using distinct request schedules. Results are reported with 95% confidence.
- Cold-start measurements use five sequential requests separated by 10 minutes to encourage container reinitialization.
- Warm-start measurements discard one initial request, then issue 25 requests at one-second intervals.
- All reported experimental results use 95% confidence.
B. Warm Results
Warm-start experiments compare latency, prediction time, and cost across Lambda memory sizes for SqueezeNet, ResNet, and ResNeXt. Increasing memory generally reduces delay and prediction time, but higher allocation can eventually raise cost without meaningful performance gains.
- Experimental setup: Warm experiments measure client-observed latency, prediction time, and execution cost across Lambda memory sizes for SqueezeNet, ResNet, and ResNeXt.Figures 1–3 report these measures for the three models.
- Latency and prediction time: Prediction time follows a similar pattern to latency because prediction time contributes to total latency.
- Latency and prediction time: Increasing memory size decreases total delay and prediction time, likely because Lambda allocates CPU, disk, and I/O resources proportionally to memory.Observed maximum memory remained 85MB for SqueezeNet, 229MB for ResNet, and 429MB for ResNeXt.
- Cost: Higher memory allocation does not necessarily increase total cost because shorter execution time can offset the higher per-100ms price.Execution cost is multiplied by 10^3 in the figures for display.
- Cost and resource selection: From 1024MB to 1536MB, SqueezeNet shows no considerable performance improvement while execution cost increases; similar observations apply to ResNet and ResNeXt.Allocating more resources than the function needs may impose additional customer costs.
C. Cold Results
Cold-start latency decreases with larger Lambda memory allocations, but its pattern differs from warm-start behavior because container launching and bootstrapping dominate the overhead.
- The cold experiments report average client-observed latency and average prediction time across varying Lambda memory sizes.
- Cold-start latency decreases as Lambda memory size increases, but it does not follow the warm-start pattern.The same cold-experiment pattern is reported for SqueezeNet, ResNet, and ResNeXt.
- Container launching and bootstrapping dominate the cold-start overhead.
D. Scalability evaluations
The scalability evaluation increases concurrent request demand and finds that latency and prediction time decrease with memory, with especially large memory sizes typically staying under an acceptable response time.
- The evaluation sends 10 parallel HTTP requests and increases the request rate by 10 requests per second for 10 seconds.Figure 7 illustrates the JMeter configuration used for this load pattern.
- The scalability experiment cannot distinguish warm starts from cold starts because request routing to existing or newly launched containers is unknown.
- Scalability results show that latency and prediction time decrease as function memory increases.This result is reported for SqueezeNet, with similar experiments conducted for ResNet and ResNeXt.
- At large memory sizes, latency is typically under an acceptable user-expected response time.
E. Discussions and limitations
The evaluation finds serverless prediction serving feasible, but cold starts, resource limits, and uncertain cost-performance trade-offs constrain deployment.
- Overall feasibility: The initial evaluation results show the feasibility of using serverless computing for prediction serving of trained neural networks.
- Latency and SLA limitations: Warm-request latency is within a reasonable range, particularly above 1024MB memory, whereas cold requests can be significantly slower.Cold-start overhead must be resolved for applications with stringent SLAs.
- Resource limitations: 512MB of ephemeral disk capacity limits serverless platforms when serving neural network models larger than 500MB.
- Cost-performance trade-offs: Increased execution cost does not always correlate with better performance across the evaluated configurations.
IV. RELATED WORK
Related work spans serverless applications, model-serving frameworks, and prior studies of serverless execution, while this work targets deep-learning model inference under broader experimental conditions.
- Serverless applications: Prior serverless applications include chatbots, mobile backends, data storage, and security analytics services.
- Prior serverless inference studies: Earlier serverless model-inference experiments omitted function-only execution timing and did not examine memory, request-rate changes, or cold and warm starts.
- Model-serving frameworks: Existing model-serving frameworks target specific frameworks or optimize throughput and latency through caching, batching, and adaptive model selection.These designs do not necessarily minimize operational costs under rapidly changing or unpredictable demand.
- Position of this work: This study evaluates serving deep-learning models in a more complex serverless function than prior small-code-snippet execution analyses.
V. CONCLUSION AND FUTURE WORK
The paper concludes that warm serverless inference can provide acceptable latency, but cold starts, CPU-only stateless execution, and limited state support remain important constraints.
- Conclusions: Warm serverless function executions are within an acceptable latency range, while cold starts add significant overhead and can risk SLA adherence.
- Conclusions: Lack of GPU access and stateless functions restrict each execution to CPU resources without runtime-maintained state between invocations.
- Future work: Future evaluations will cover other deep-learning frameworks, serverless providers, and model types while studying cost-memory trade-offs.
- Future work: The paper proposes declarative workload requirements, including minimum warm-container time and GPU access, to support more stateful workloads.
- Future work: Hybrid use of optimized virtual machines and serverless resources could support both occasional prediction requests and sustained high-throughput workloads with peaks.