Source-linked AI summary
Patterns in the Chaos - a Study of Performance Variation and Predictability in Public IaaS Clouds
Philipp Leitner, Juergen Cito
TL;DR
Public-cloud benchmarking has produced extensive but difficult-to-compare evidence about performance predictability. This paper combines a principled literature review and 15 hypotheses with real-life experiments, finding provider differences, less practical hardware heterogeneity than expected, and a dramatic impact from multi-tenancy.
Problem
Existing cloud studies report extensive performance results, but incomplete reporting and limited comparison make meaningful and portable knowledge difficult to extract.
Method
The paper systematically reviews and codifies public-IaaS research, formulates 15 hypotheses, and validates them with real-life benchmark data from multiple providers.
Results
The study finds substantial provider-performance differences, less practical hardware heterogeneity than earlier research suggested, and a dramatic impact of multi-tenancy on performance and predictability.
Takeaways & Limitations
Practitioners should not assume EC2 findings apply to other providers and should benchmark instance types for their specific use cases.
Takeaways & Limitations
The experiments were conducted over a relatively short period, so longer-term external influences on results were not controlled.
Abstract
from arXiv · showhide
Benchmarking the performance of public cloud providers is a common research topic. Previous research has already extensively evaluated the performance of different cloud platforms for different use cases, and under different constraints and experiment setups. In this paper, we present a principled, large-scale literature review to collect and codify existing research regarding the predictability of performance in public Infrastructure-as-a-Service (IaaS) clouds. We formulate 15 hypotheses relating to the nature of performance variations in IaaS systems, to the factors of influence of performance variations, and how to compare different instance types. In a second step, we conduct extensive real-life experimentation on Amazon EC2 and Google Compute Engine to empirically validate those hypotheses. At the time of our research, performance in EC2 was substantially less predictable than in GCE. Further, we show that hardware heterogeneity is in practice less prevalent than anticipated by earlier research, while multi-tenancy has a dramatic impact on performance and predictability.
1. INTRODUCTION
Cloud benchmarking is widespread, but inconsistent reporting and limited comparison make performance findings difficult to interpret and transfer. This study addresses that gap by reviewing existing evidence, formulating hypotheses, and validating them with large-scale measurements.
- Existing cloud studies evaluate providers across varied use cases, constraints, and experimental setups.
- Incomplete parameter and raw-data reporting makes differing results difficult to attribute to provider changes, unreported factors, or technical inaccuracies.
- Despite many reported data points, extracting meaningful and portable knowledge from cloud-performance research remains difficult.
- The study formulates 15 hypotheses about performance predictability, variation factors, and cloud-instance selection.
- The authors collect 53918 measurements from four IaaS providers using five micro- and application-level benchmarks across multiple configurations.
2. PRELIMINARIES
IaaS instances are configurable virtual machines whose performance and cost depend on selected regions, images, and instance types. Shared physical resources can create noisy-neighbour slowdowns, while instance types may be served by heterogeneous hardware.
- Users specify a region, base image, and instance type when requesting an IaaS virtual machine.
- Instance types define an instance’s price and available computing resources.
- IaaS clouds commonly group instance types into families for comparable use cases, such as general-purpose, CPU-optimized, and I/O-optimized workloads.
- Identical instance types may be served by different underlying hardware, creating hardware heterogeneity.
- Multiple customers typically share physical machines, so shared network or disk I/O can produce noisy-neighbour slowdowns.
3. RELATED WORK
Related work provides taxonomies, benchmarking methodologies, metrics, and measurement frameworks, but this study extends that foundation by codifying and validating reported performance findings.
- Li et al. systematically collected and classified IaaS benchmarking research, producing provider-evaluation methods, metric catalogues, and benchmark models.
- Unlike the present study, Li et al.’s work did not codify or validate the results reported in the reviewed research.
- Prior systems such as Expertus, Cloud-Bench, CloudCrawler, and Cloud Workbench support defining and executing benchmarks across cloud systems.
- The authors use Cloud Workbench for quantitative data collection, while noting that other benchmark-execution systems could also have been used.
- CloudHarmony collected performance data broadly and over multiple years, whereas this study differs in its emphasis on other research objectives.
4. HYPOTHESES
The paper derives 15 hypotheses from a systematic literature review about performance variation, predictability, temporal and geographical factors, and instance-type selection. The hypotheses distinguish between instance-level variability, workload type, tenancy, and cost efficiency.
- Review procedure: The literature review seeds citation-based search with seminal studies, filters candidate papers by public-cloud benchmarking and numerical results, and iterates until no further relevant studies are found.
- Review procedure: Open coding of the reviewed research produces 15 hypotheses grouped into four categories.
- H1: Performance Predictability: Performance can vary substantially between cloud instances using the same configuration.
- H1: Performance Predictability: CPU-bound applications are expected to depend strongly on the served CPU model and vary primarily because of hardware heterogeneity.
- H1: Performance Predictability: I/O-bound applications are expected to depend strongly on co-located tenants and may experience noisy-neighbour effects.
- H1: Performance Predictability: CPU-bound performance may be multi-modal, so knowing the hardware can substantially improve predictability; for I/O-bound workloads, hardware knowledge is less informative.
- H2: Intra-Instance Variability: I/O-bound performance is expected to vary within a single instance, while dedicated-CPU workloads are expected to remain comparatively stable over time.
- H2: Intra-Instance Variability: Bursting instance types are expected to show within-instance variability because shared processors use credit systems and remain exposed to noisy neighbours.
5. EXPERIMENTAL RESULTS
Experiments across four IaaS providers evaluate performance variability, predictability, hardware effects, temporal and regional factors, and instance-type selection. Results show strong provider-, benchmark-, and configuration-specific differences, with EC2 and Azure generally less predictable for several workloads than GCE and SoftLayer.
- Performance predictability: 63 of 82 configurations had cRSD above 5%, with EC2 and Azure generally less predictable than GCE and SoftLayer.EC2 t1.micro instances in us-east-1 exceeded 20% cRSD for every benchmark.
- Hardware heterogeneity: Hardware heterogeneity was not universal: GCE, EC2, and SoftLayer generally used consistent hardware, while Azure exposed wider CPU-model variation.More than 97% of EC2 m1.small eu-west-1 runs used the same CPU model.
- Hardware heterogeneity: Controlling for CPU model reduced relative standard deviations in CPU-bound benchmarks, supporting the hardware-heterogeneity hypothesis.The same control also reduced variability in IO-bound results, but the supplied passage does not quantify that reduction.
- Within-instance variability: Identically configured EC2 m3.large instances differed substantially: instance 9097 had nearly twice instance 14704’s mean IO speed, with iRSDs of 12.25% and 30.12%.Neither time series showed an obvious performance trend over time.
- Within-instance variability: CPU variability was very low on non-bursting instances, whereas EC2 and Azure showed fluctuating IO performance and GCE and SoftLayer were remarkably predictable.A shared-CPU t1.micro showed 34.82% iRSD and slowdown as its credit account depleted.
- Temporal and geographical factors: Time-of-day and day-of-week hypotheses lacked support, but region differences were significant in 26 of 33 configurations for performance and 23 of 33 for variance.Visual comparisons indicated the analyzed European region was preferable in most cases, without establishing a universally preferable region.
- Instance type selection: Cost-efficiency and predictability did not follow a provider-independent instance-size rule; the best choice depended on provider, benchmark, and use case.Larger instances were less cost-efficient only for Azure and SoftLayer, while specialized EC2 types showed slightly better cost-performance for related workloads.
6. IMPLICATIONS
The study finds substantial provider-specific differences in performance predictability and challenges several established assumptions about hardware heterogeneity, multi-tenancy, temporal effects, and instance selection.
- Provider differences: EC2 and Azure largely matched expectations, whereas GCE and SoftLayer were substantially more stable and predictable than anticipated.The authors argue that benchmarking research focused heavily on EC2 should be validated across more providers.
- Hardware heterogeneity: Hardware heterogeneity is less prevalent than earlier studies reported; in EC2 it mainly affects micro and small instances in North America.More expensive EC2 instance types were served with a fixed CPU model, while Azure retained hardware heterogeneity across all instance types.
- Multi-tenancy: Multi-tenancy strongly affects predictability, but its impact varies substantially by provider.IO-bound benchmarks were highly unpredictable on EC2 and Azure but comparatively predictable on GCE and SoftLayer; the authors speculate that infrastructure utilization may explain this difference.
- Temporal and geographical factors: Regions affect performance across providers, while the day of the week and time of day show little measurable impact.Regional differences were less pronounced in GCE and Azure than in EC2.
- Instance selection: No reliable rules of thumb identify the most cost-efficient instance type for a use case.Provider pricing strategies differ, so explicit benchmarking is generally required, including for specialized instance types.
7. THREATS TO VALIDITY
The study’s validity is constrained by literature-search coverage, a relatively short experimental period, and selection from a much larger space of providers and configurations.
- Construct validity: Citation-graph seeding may miss relevant papers that are disconnected from the selected seed publications.The authors checked ten arbitrarily selected relevant ACM Digital Library papers and found them included in the study set.
- Internal validity: The experiments covered only a relatively short period, leaving open the possibility that temporary external factors influenced the results.The authors call for longer-term research to control for effects such as exceptionally high seasonal load in EC2.
- External validity: Selecting a subset of providers, instance types, regions, and configurations limits direct generalization to untested settings.The authors argue that coverage of EC2, Azure, GCE, and SoftLayer is sufficient to support their conclusions despite this boundary.
8. CONCLUSIONS AND OUTLOOK
The study combines structured analysis with real-world cloud measurements and concludes that provider differences, reduced hardware heterogeneity, and uneven multi-tenancy make cloud performance a moving target.
- Conclusions: The study validated 15 literature-derived hypotheses using real-life data from EC2, GCE, Azure, and SoftLayer.Its structured analysis focused on performance variation and predictability in public IaaS providers.
- Conclusions: Provider performance differs substantially, so results obtained on EC2 should not automatically be assumed applicable to other clouds.The conclusion also reports that hardware heterogeneity is less practically important than earlier research suggested and that multi-tenancy varies by provider.
- Outlook: Regions significantly affect performance and predictability, whereas the study found no conclusive causal link with day or time.The authors also found that hard-and-fast rules for cost-efficient instance selection are infeasible.
- Outlook: Performance stability was studied only within a three-day window, leaving longer-term instance stability unaddressed.Longer measurements could help practitioners determine how often instance performance should be re-evaluated.
Companion Website
The authors provide an online companion containing the benchmark code and all study data, along with additional plots and analysis scripts.
- Companion Website: The online companion makes the benchmark code and all data used in the study available.It also includes additional plots and scripts for analysis and data cleaning.