Source-linked AI summary
SeBS: A Serverless Benchmark Suite for Function-as-a-Service Computing
Marcin Copik, Grzegorz Kwasniewski, Maciej Besta, Michal Podstawski, Torsten Hoefler
TL;DR
Serverless research lacks standardized, reproducible benchmarking across rapidly changing and opaque FaaS platforms. SeBS introduces an abstract, multi-provider benchmark suite with representative workloads, metrics, models, and implementation infrastructure. Across AWS, Azure, and Google Cloud, results show provider-dependent performance and behavior, including AWS speed advantages, Azure variance, and workload-specific suitability.
Problem
FaaS benchmarking lacks standardized, generalizable evaluation despite platform-specific overheads and black-box behavior complicating comparison.
Method
SeBS combines an abstract FaaS model, representative workloads, metrics, performance models, and implementation infrastructure for evaluating multiple providers.
Results
Across AWS, Azure, and Google Cloud, AWS is considerably faster in almost all scenarios, Azure has high variance, and performance and behavior differ across providers.
Takeaways & Limitations
SeBS provides a standardized methodology for evaluating performance, cost, reliability, and portability across FaaS platforms and workloads.
Takeaways & Limitations
Commercial FaaS platforms constrain user configuration and resource consumption, with Azure additionally bundling functions into function apps.
Abstract
from arXiv · showhide
Function-as-a-Service (FaaS) is one of the most promising directions for the future of cloud services, and serverless functions have immediately become a new middleware for building scalable and cost-efficient microservices and applications. However, the quickly moving technology hinders reproducibility, and the lack of a standardized benchmarking suite leads to ad-hoc solutions and microbenchmarks being used in serverless research, further complicating metaanalysis and comparison of research solutions. To address this challenge, we propose the Serverless Benchmark Suite: the first benchmark for FaaS computing that systematically covers a wide spectrum of cloud resources and applications. Our benchmark consists of the specification of representative workloads, the accompanying implementation and evaluation infrastructure, and the evaluation methodology that facilitates reproducibility and enables interpretability. We demonstrate that the abstract model of a FaaS execution environment ensures the applicability of our benchmark to multiple commercial providers such as AWS, Azure, and Google Cloud. Our work facilities experimental evaluation of serverless systems, and delivers a standardized, reliable and evolving evaluation methodology of performance, efficiency, scalability and reliability of middleware FaaS platforms.
1 Introduction
FaaS offers fine-grained, pay-as-you-go computing for modular applications, but its rapid evolution and platform opacity hinder reproducible, generalizable evaluation. SeBS addresses this gap with standardized workloads, metrics, models, and tooling evaluated across major providers.
- FaaS uses stateless functions for fine-grained computing and billing, avoiding infrastructure management and charges for unused services.
- Serverless evaluation is hindered by vendor lock-in, missing standardized development tools, cold-start overheads, high computation costs, and black-box platforms.
- SeBS defines comparison baselines, supports general performance, cost, and reliability insights, and provides public implementation.
- SeBS combines literature-based benchmark specifications, an abstract FaaS model, metrics, performance models, and an extensible implementation kit.
- The suite is evaluated on AWS, Microsoft Azure, and Google Cloud Platform using workloads representing diverse real-world applications.
- SeBS provides continuous comparison of FaaS performance, reliability, and cost-effectiveness through automated, open-source tooling for deployment and invocation.
2 Platform Model
The platform model abstracts FaaS components so benchmark design can generalize across providers despite proprietary implementation details. It represents triggers, execution environments, persistent and ephemeral storage, and invocation overheads.
- The benchmarking model abstracts key FaaS components to generalize provider-specific or unknown details in black-box platforms.
- Triggers: Triggers initiate function lifetimes through HTTP requests, schedules, events, or larger FaaS workflows.
- Execution environment: Execution environments isolate tenants using containers or microVMs, with containers potentially adding overheads of up to 20× over native execution.
- Storage: Persistent storage provides scalable, high-bandwidth retrieval with high throughput and latency, while ephemeral storage targets lower-latency data exchange.
- Invocation system: Invocation includes endpoint handling, scheduling, server communication, load balancing, caching, and possible cold-start startup latency.
3 Serverless Model Analysis
SeBS analyzes FaaS workloads, costs, I/O behavior, vendor portability, hardware heterogeneity, and microarchitectural effects. The analysis emphasizes workload-specific trade-offs and limitations in commercial platforms.
- SeBS selects workloads while investigating limitations that can throttle migration to serverless environments.
- FaaS offers burst parallelism and benefits for irregular workloads, but stragglers and programming difficulty can limit high performance.
- Computing Cost: FaaS is more cost-effective for infrequent workloads, whereas communication overheads can make machine-learning training more expensive than VM-based solutions.
- I/O performance: Network and disk behavior affect startup and execution, with function co-allocation decreasing AWS throughput by up to 20×.
- Vendor Lock-In: Provider-specific configuration and service interfaces complicate development, so SeBS supplies an adaptation layer for deployment, invocation, and storage management.
- Heterogeneous Environments: Commercial FaaS platforms restrict hardware and resource configuration, while SeBS includes tasks that can benefit from specialized hardware.
- FaaS platforms’ limitations: Azure Functions bundle multiple functions into apps, enabling less frequent cold starts and increased locality while preserving isolation and security.
4 Benchmark Specification
SeBS specifies representative FaaS workloads and benchmarking principles designed to support relevance, usability, reproducibility, interpretability, and extensibility. Its application set spans diverse resource profiles, including web, multimedia, utility, inference, and scientific workloads.
- 4.2 Applications: The benchmark specification classifies common FaaS workloads into six major categories.The supplied passages describe web applications, multimedia, utilities, inference, and scientific workloads; the collection is designed to span different performance profiles.
- 4.1 Benchmark Design Principles: SeBS follows benchmark-design principles covering relevance, usability, reproducibility, interpretability, and extensibility.The methodology uses representative workloads, automated evaluation, confidence intervals, and an abstract FaaS model independent of specific commercial systems.
- 4.2 Applications: SeBS uses local evaluation to characterize computing, memory, and external-resource requirements before selecting benchmarks.This evaluation supports representativeness checks and choosing workloads according to required resource consumption.
- 4.2 Applications: The application set includes low-resource web functions such as dynamic HTML generation and URL-based storage upload.These workloads have low CPU and memory requirements.
- 4.2 Applications: Scientific workloads include BFS, PageRank, and Minimum Spanning Tree, all data-intensive but differing in workload characteristics.BFS may exhibit severe work imbalance across iterations, unlike PageRank, while the three algorithms represent traversal, centrality, and graph-optimization problems.
5 Benchmark Implementation
SeBS implements a modular toolkit for characterizing applications, deploying and invoking benchmarks, and measuring performance across local and cloud environments. It combines function wrappers, provider APIs, abstract interfaces, and postprocessing to support experiments across FaaS platforms.
- 5.1 Metrics and 5.2 Deployment: The toolkit automatically deploys and invokes benchmarks while collecting local and cloud execution metrics.Local metrics include time, CPU utilization, and memory; cloud measurements include benchmark, provider, and client time plus memory and billing information.
- 5.1 Metrics: SeBS measures execution time at benchmark, provider, and client levels to separate function work from platform and end-to-end overheads.Client-side latency includes scheduling and deployment effects, while provider measurements add language and sandbox overheads.
- 5 Benchmark Implementation: SeBS supports application characterization, cloud deployment, and modeling of cloud performance and overheads.The platform is modular and supports adding new benchmarks, metrics, and platforms.
- 5.2 Implementation: Provider-specific wrappers and abstract interfaces standardize function inputs, triggers, storage, metrics, and experiment postprocessing.The toolkit supports SDK and HTTP triggers, provider log querying, unchanged benchmark outputs, and automatic integration of new experiments and triggers.
- 5.2 Implementation: The implementation uses containers, language workers, persistent storage, hardware performance counters, provider APIs, and cURL-based client measurements.PAPI is used for low-level characteristics, while provider APIs query execution time, billing, and memory when available.
6 Evaluation
SeBS evaluation shows substantial provider- and workload-dependent variation in performance, cold-start overhead, reliability, availability, and cost efficiency. AWS generally provides the strongest performance, while portability and predictable resource-cost relationships remain limited.
- Performance: AWS Lambda provides the best warm-invocation performance across benchmarks, while Google is comparable only for graph-bfs and storage-bound workloads show the largest slowdowns.Execution time decreases with additional resources until reaching a performance plateau.
- Performance: I/O-bound benchmarks have wider latency distributions and more outliers, whereas compute-intensive applications are more consistent but can produce stragglers.Python and Node.js runtime differences do not explain most performance variation.
- Cold starts: Cold execution of image-recognition can take up to ten times longer than warm execution, while compression longer than 10 seconds experiences negligible cold-start impact.Azure reports lower cold-start overheads, particularly for large deployment packages.
- Cold starts: Higher memory allocations reduce Lambda cold-start overheads but adversely affect Google Functions except for image-recognition, so stronger allocations are not portable solutions.The authors suggest container-pool competition may explain Google’s contrasting behavior; Azure benefits functions with expensive cold initialization.
- Reliability and availability: GCP exhibits unexpected cold startups and memory-limit failures, while Azure shows significant concurrent-invocation deviations and bottlenecks for Python functions.GCP image-recognition reached up to 80% errors at 4096 MB under 50 concurrent invocations; memory-related failure frequencies were 4% and 5.2% for reported cases.
- Conclusions: Overall, FaaS performance is unstable across providers, identical software configurations are not portable, and serverless execution can incur significant performance losses compared with VM-based deployment.The evaluation also identifies AWS as consistently highest-performing and I/O-bound workloads as a poor fit for serverless.
- Cost analysis: Memory usage is not necessarily proportional to CPU and I/O allocation, and pricing that rounds resources upward encourages memory over-allocation and cloud-resource underutilization.Azure’s dynamic memory allocation generates higher costs, while performance gains can be cost-effective for image-recognition but not compression.
7 Related Work
SeBS differs from earlier cloud and serverless benchmarks by systematically evaluating diverse real-world FaaS applications and analyzing performance, consistency, efficiency, overheads, and resource consumption.
- SeBS uses diverse real-world FaaS applications rather than microbenchmarks limited to basic CPU, memory, and I/O requirements.
- SeBS extends evaluation beyond function latency and throughput to consistency, efficiency, initialization overheads, container eviction probabilities, and local resource-consumption metrics.
- SeBS provides insights into reliability and economics on black-box commercial cloud systems, unlike related work focused on function composition or white-box open-source platforms.
- Other benchmark suites target domains including cloud and HPC, graph analytics, Java applications, machine learning, and dense linear algebra.
8 Conclusions
The paper presents SeBS as a portable benchmark suite for developing, evaluating, and analyzing FaaS applications. Evaluations across AWS, Azure, and Google Cloud reveal substantial provider and workload differences, while the open-source suite supports characterization and optimization of serverless middleware.
- SeBS combines an abstract, portable FaaS benchmarking model with a specification and implementation covering diverse metrics, applications, and performance characteristics.
- Across AWS, Microsoft Azure, and Google Cloud Platform, AWS is considerably faster in almost all scenarios, Azure has high variance, and provider behavior is inconsistent.
- The evaluation provides insights into performance overheads, portability, cost-efficiency, container eviction, and invocation overhead.
- The open-source benchmark suite characterizes serverless middleware to help users build and optimize applications using FaaS as a cloud execution backend.