Source-linked AI summary
Towards Demystifying Serverless Machine Learning Training
Jiawei Jiang, Shaoduo Gan, Yue Liu, Fanlin Wang, Gustavo Alonso, Ana Klimovic, Ankit Singla, Wentao Wu, Ce Zhang
TL;DR
Distributed ML training on FaaS has shown unclear advantages over IaaS, motivating a systematic comparison. The paper studies the design space, implements LambdaML, and develops an analytical model; it finds that FaaS pays off mainly for communication-efficient, quickly converging models and is generally not significantly cheaper.
Problem
The problem is determining when FaaS can outperform VM-based IaaS for distributed ML training despite inconclusive prior evidence.
Method
The paper systematically compares FaaS and IaaS design choices, implements LambdaML, and develops an analytical model of runtime tradeoffs.
Results
FaaS training pays off mainly for models with reduced communication that converge quickly, while other workloads can perform significantly worse.
Takeaways & Limitations
FaaS may provide substantial speed advantages, but it is generally not significantly cheaper than IaaS.
Takeaways & Limitations
Current FaaS infrastructures impose statelessness, limited computation resources and duration, and no customized scaling or scheduling strategies.
Abstract
from arXiv · showhide
The appeal of serverless (FaaS) has triggered a growing interest on how to use it in data-intensive applications such as ETL, query processing, or machine learning (ML). Several systems exist for training large-scale ML models on top of serverless infrastructures (e.g., AWS Lambda) but with inconclusive results in terms of their performance and relative advantage over "serverful" infrastructures (IaaS). In this paper we present a systematic, comparative study of distributed ML training over FaaS and IaaS. We present a design space covering design choices such as optimization algorithms and synchronization protocols, and implement a platform, LambdaML, that enables a fair comparison between FaaS and IaaS. We present experimental results using LambdaML, and further develop an analytic model to capture cost/performance tradeoffs that must be considered when opting for a serverless infrastructure. Our results indicate that ML training pays off in serverless only for models with efficient (i.e., reduced) communication and that quickly converge. In general, FaaS can be much faster but it is never significantly cheaper than IaaS.
1 INTRODUCTION
The paper studies when FaaS can outperform IaaS for distributed ML training, addressing inconclusive prior comparisons. It combines a fair empirical evaluation with an analytical cost/performance model and finds that FaaS advantages are conditional.
- Research question: The study asks when FaaS can outperform VM-based IaaS for distributed ML training.This question motivates the systematic comparison across algorithms, workloads, models, and infrastructure choices.
- Motivation: FaaS versus IaaS performance comparisons remain inconclusive because earlier studies often used unoptimized or non-equivalent implementations.Some previous work reported up to two orders of magnitude improvement for FaaS, but comparisons were frequently not on equal ground.
- Approach: The authors systematically compare FaaS and IaaS across optimization algorithms, training workloads, ML models, and infrastructure choices.The comparison uses the same algorithms where possible and evaluates time or dollar cost to converge to the same loss.
- Approach: The analytical model characterizes FaaS–IaaS runtime tradeoffs and supports speculation about future system configurations.The model captures factors including worker startup, communication, computation, convergence, bandwidth, and latency.
- Approach: LambdaML is a prototype platform designed to support a fair comparison between FaaS and IaaS-based distributed ML training.The system is carefully optimized and can select design points faster than previous FaaS systems.
- Findings: FaaS can be faster only when workloads have communication-efficient behavior and quickly converge, while faster FaaS generally has comparable dollar cost to IaaS.Some realistic workloads benefit, but others perform significantly worse under FaaS.
2 PRELIMINARIES
The preliminaries define distributed ML training and organize system design around communication choices, while contrasting IaaS resource management with FaaS elasticity and constraints.
- ML foundations: The ML training objective is to find a model w that minimizes a loss function over the training dataset.The dataset consists of examples represented by feature vectors and labels.
- Distributed ML: Distributed ML training repeatedly scans data, computes quantities such as gradients, updates the model, and stops when the procedure converges.An epoch is one pass over the entire dataset, though its relationship to iterations depends on the optimization algorithm.
- Design space: Distributed ML systems differ in communication channel, communication pattern, and synchronization protocol.Channels may use message passing, disk-based storage, or in-memory key-value stores; patterns include Gather, AllReduce, and ScatterReduce.
- FaaS vs. IaaS: IaaS requires users to rent or reserve VM clusters and pay for reserved resources, whereas FaaS allocates and auto-scales resources on demand.FaaS charges for actual resource usage but does not currently support customized scaling and scheduling strategies.
- FaaS vs. IaaS: FaaS functions are stateless and constrained in computation resources and duration, requiring external mechanisms for intermediate training state.For example, AWS Lambda functions can use up to 3GB of memory and must finish within 15 minutes.
3 LAMBDAML
LambdaML is a prototype FaaS-based ML system built on Amazon Lambda for studying the tradeoffs of training ML models over serverless infrastructure.
- LambdaML: LambdaML implements FaaS-based distributed ML training on top of Amazon Lambda.The prototype is used to study training tradeoffs over serverless infrastructure.
3.1 System Overview
LambdaML organizes distributed training around stateless workers, external storage, communication, aggregation, model updates, and synchronization. Its design addresses the lack of direct communication between FaaS functions.
- System Overview: LambdaML uses a communication channel, communication pattern, and synchronization protocol to coordinate distributed optimization across stateless functions.These components aggregate local statistics and govern the iterative training process.
- Communication: FaaS data aggregation uses external storage because stateless functions cannot communicate directly.The broader system design therefore treats storage as the channel for exchanging intermediate state.
- System Overview: Users specify data location, resources, optimization algorithms, and hyperparameters, after which AWS allocates serverless worker instances.Training data is partitioned in S3, and each worker maintains a local model copy.
- Job Execution: Each LambdaML training iteration loads data, computes local statistics, sends them through a communication channel, aggregates them, and updates worker models.Workers repeat the process when permitted by the synchronization protocol.
3.2 Implementation of LambdaML
LambdaML explores distributed ML training across four design dimensions: optimization algorithm, communication channel, communication pattern, and synchronization protocol. Its FaaS implementations use external storage because stateless functions cannot directly communicate.
- Distributed optimization algorithms: LambdaML studies distributed optimization algorithms including gradient averaging, model averaging, and ADMM.Executors train on partitioned data and exchange or update model information at synchronization points; ADMM instead solves local subproblems before exchanging models.
- Communication channel: FaaS training requires a storage component for stateless functions to exchange intermediate state and aggregate distributed updates.Possible channels include S3, ElastiCache services, DynamoDB, or a customized parameter server.
- Communication channel: S3 is always available and cheaper, whereas ElastiCache offers faster access but requires startup time.The channel choice therefore creates a cost/performance tradeoff between startup delay, access speed, and price.
- Communication pattern: AllReduce has one leader aggregate updates, while ScatterReduce partitions aggregation across all executors to reduce leader bottlenecks.With n executors, each executor aggregates one of n update partitions in ScatterReduce.
- Synchronization protocol: Synchronous training uses merging and updating phases, whereas asynchronous executors independently read, update, and write a shared global model.The asynchronous protocol does not wait for other executors’ speeds.
3.3 Other Implementation Details
LambdaML addresses the 15-minute lifetime limit of Lambda functions with hierarchical invocation, using a starter function to launch workers for partitioned training data.
- Lambda lifetime and invocation: 15 minutes is the maximum execution time for a Lambda function, constraining long-running ML training.LambdaML uses hierarchical invocation to schedule executions around this lifetime limit.
- Lambda lifetime and invocation: A starter Lambda is triggered after data reaches storage and then launches n worker Lambdas, one for each data partition.The workers serve as executors for distributed training.
4 EVALUATION OF LAMBDAML
The evaluation compares LambdaML’s algorithm, communication-channel, hybrid, communication-pattern, and synchronization choices across varied datasets and models. Results show that performance depends strongly on communication cost, convergence behavior, and workload characteristics.
- Experiment settings: LambdaML evaluates statistical efficiency through validation loss and system efficiency through iteration or epoch execution time.The experiments use Higgs, RCV1, and Cifar10, alongside linear, neural-network, and clustering models.
- Distributed optimization algorithms: ADMM converges fastest for logistic regression and SVM on Higgs, while GA-SGD is slowest because per-batch gradient transmission is costly.Model averaging reduces communication frequency relative to gradient averaging, and ADMM can reduce it further.
- Distributed optimization algorithms: For MobileNet, ADMM is unsuitable and model averaging converges unstably, leaving GA-SGD as the viable choice because it converges steadily to lower loss.The algorithm that minimizes communication for linear models is therefore not universally appropriate across workloads.
- Hybrid solutions: Pure FaaS can match hybrid parameter-server designs for many workloads, while Lambda-to-VM communication is constrained by serialization, deserialization, and parameter locking.Adding parameter servers does not significantly improve performance because serialization in Lambda, rather than bandwidth, is the bottleneck.
- Communication channels: Memcached improves communication over S3 by 7× for logistic regression and 7.7× for KMeans on clusters of up to 50 workers.Despite lower latency, Memcached’s startup delay can make it slower overall than always-on S3 for short-running workloads.
- Communication channels: DynamoDB reduces communication time by roughly 20% for logistic regression on Higgs but cannot handle messages above 400KB, limiting use for larger models.The result illustrates that faster communication does not necessarily make a channel practical across model sizes.
- Communication patterns: ScatterReduce is slightly slower than AllReduce for logistic regression, has similar cost for MobileNet, and is advantageous when AllReduce’s leader becomes a bottleneck.ScatterReduce adds partitioning overhead when communication is not the bottleneck.
- Synchronization protocols: Synchronous training converges steadily, whereas asynchronous training runs faster per iteration but suffers unstable convergence from inconsistent local model parameters.Fast executors may read stale parameters when stragglers are present.
5 FAAS VS. IAAS FOR ML TRAINING
LambdaML enables a fair end-to-end comparison of FaaS and IaaS ML training across algorithms, systems, models, and configurations. FaaS can be faster for communication-efficient workloads, but its advantage depends on communication, convergence, startup, and model characteristics rather than extending uniformly across workloads.
- Experimental setup: LambdaML compares FaaS and IaaS using competing systems, tuned configurations, and multiple ML models and datasets.The evaluation includes Distributed PyTorch and LambdaML, with models including LR, SVM, KMeans, MN, and ResNet50 across several datasets.
- End-to-end performance: Communication limits FaaS for deep learning: PyTorch benefits from faster VM communication, while GPU acceleration makes PyTorch-GPU fastest.A 3GB FaaS memory limit also causes an out-of-memory failure at batch size 64, demonstrating a current infrastructure limitation for large models.
- End-to-end performance: FaaS is significantly faster but not significantly cheaper on some communication-efficient workloads, while other workloads make it slower and more expensive.Runtime and cost depend on the workload and configuration; adding workers eventually increases cost while runtime plateaus.
- End-to-end performance: For LR and SVM, some FaaS configurations beat every tested IaaS configuration in runtime without being significantly cheaper; for KMeans, cost favors IaaS while runtime favors FaaS.For MN, an IaaS T4 configuration is 8× faster and 9.5× cheaper than the best FaaS execution.
- Analytical model: FaaS outperforms IaaS when its algorithm scales well and converges in few rounds, while the analytical model characterizes rather than fully predicts runtime.The model approximates measured runtime reasonably well, but prediction requires estimating the number of epochs needed by each algorithm.
6 RELATED WORK
Prior work spans distributed ML, serverless data processing, and serverless ML training, with increasing attention to training models on FaaS infrastructures.
- Distributed ML commonly uses data parallelism, with workers training on local partitions and synchronizing periodically through parameter servers or MPI.
- Serverless data-processing research examines large-scale workloads such as shuffling and linear algebra on platforms including AWS Lambda, Google Cloud Functions, and Azure Functions.
- Recent serverless ML research focuses on model training, including end-to-end workflows and neural-network training on AWS Lambda.
7 CONCLUSION
The paper systematically studies the FaaS–IaaS tradeoff for distributed ML training through a design-space analysis, LambdaML, experiments, and an analytic model. Serverless training pays off only for quickly converging models with reduced communication; FaaS can be much faster but is never significantly cheaper than IaaS.
- The study covers optimization algorithms, communication channels and patterns, and synchronization protocols, then implements LambdaML for systematic FaaS–IaaS comparison.
- Serverless ML training pays off only for models with efficient communication and fast convergence.
- FaaS can be much faster than IaaS but is never significantly cheaper.