Source-linked AI summary
Scientific Workflow Applications on Amazon EC2
Gideon Juve, Ewa Deelman, Karan Vahi, Gaurang Mehta, Bruce Berriman, Benjamin P. Berman, Phil Maechling
TL;DR
The paper asks whether commercial clouds can deliver suitable scientific-workflow performance at reasonable cost. It compares three workflows on Amazon EC2 and an HPC cluster, analyzes cloud costs, and finds comparable performance with similar resources while cloud storage can reduce transfer costs.
Problem
It is unclear whether commercial clouds provide the performance required by scientific applications at a reasonable price.
Method
The study runs three diverse scientific workflows on Amazon EC2, compares them with NCSA’s Abe cluster, and analyzes execution, storage, and transfer costs.
Results
Given equivalent resources, EC2 performance was close to Abe’s, although Abe’s high-speed interconnect and parallel file system benefited I/O-intensive workloads.
Takeaways & Limitations
Cloud workflow costs can be reduced by storing input data in the cloud rather than repeatedly transferring it from outside.
Takeaways & Limitations
The reported runtimes exclude EC2 VM installation and boot time, Abe queue waiting time, and input-transfer time.
Abstract
from arXiv · showhide
The proliferation of commercial cloud computing providers has generated significant interest in the scientific computing community. Much recent research has attempted to determine the benefits and drawbacks of cloud computing for scientific applications. Although clouds have many attractive features, such as virtualization, on-demand provisioning, and "pay as you go" usage-based pricing, it is not clear whether they are able to deliver the performance required for scientific applications at a reasonable price. In this paper we examine the performance and cost of clouds from the perspective of scientific workflow applications. We use three characteristic workflows to compare the performance of a commercial cloud with that of a typical HPC system, and we analyze the various costs associated with running those workflows in the cloud. We find that the performance of clouds is not unreasonable given the hardware resources provided, and that performance comparable to HPC systems can be achieved given similar resources. We also find that the cost of running workflows on a commercial cloud can be reduced by storing data in the cloud rather than transferring it from outside.
1. Introduction
Cloud computing offers virtualized, on-demand resources and usage-based pricing, but this paper evaluates whether commercial clouds can provide suitable performance and cost for scientific workflows.
- Clouds combine virtual-machine customization, grid-like scalability and resource sharing, and software-as-a-service stability and economy.
- Direct resource allocation reduces scheduling overhead for loosely coupled workflows, while elasticity lets workflow systems expand or shrink resources as needs change.
- The paper extends prior simulation and experimental-cloud studies by evaluating diverse workflows on Amazon EC2.
- The study compares EC2 performance with NCSA’s Abe cluster and analyzes the costs of running the experiments in the cloud.
2. Applications
The evaluation uses three workflows from astronomy, seismology, and bioinformatics to span different application domains and resource requirements.
- The workflows are Montage, an astronomy application; Broadband, a seismology application; and Epigenomics, a bioinformatics application.
- The three applications were selected to cover a wide range of application domains and resource requirements.
- Table 1 compares application resource usage across I/O, memory, and CPU categories.
- Montage produces 7.9 GB of output data and is I/O-bound because it spends more than 95% of its time waiting for I/O operations.
3. Execution Environment
The experiments create comparable EC2 and Abe execution environments for loosely coupled workflows, using distinct compute resources and storage configurations.
- 3. Execution Environment: Experiments ran workflows on Amazon EC2 and NCSA’s Abe cluster, selected as a commercial cloud and a typical HPC alternative.
- 3. Execution Environment: Workflow tasks communicate through files and dependencies rather than direct network communication, so storage configuration affects execution.
- 3. Execution Environment: The comparison used single-node experiments and both local-disk configurations, while Abe also used its parallel file system.
- 3. Execution Environment: Abe.local and Abe.lustre use equivalent hardware but differ in I/O handling through local and Lustre partitions.
- 3.2 Software: EC2 worker nodes ran workflow tasks from virtual-machine images, coordinated by an external submit host through Pegasus, DAGMan, and Condor.
- 3.3 Storage: Executables and input data were pre-staged; EC2 stored executables in VM images, intermediate and output data on local partitions, and inputs on EBS.
4. Performance Comparison
Runtime varied substantially across EC2 and Abe configurations, with the best setting depending on each workflow’s resource demands. I/O-bound Montage benefited strongly from parallel storage, while virtualization penalties were relatively small for Montage and Broadband but higher for CPU-bound Epigenomics.
- Workflow runtime was measured as makespan, excluding VM boot, queue waiting, and input/output transfer time.The reported metric covers wall-clock time from first task submission to last task completion.
- m1.small had the worst runtime by a large margin for every application because of its relatively low capabilities.
- Montage: More than twice faster on abe.lustre than abe.local, Montage shows a substantial parallel-file-system advantage for this I/O-intensive workflow.The c1.xlarge versus abe.local comparison suggests less than 8% virtualization overhead.
- Broadband: Only about 1% separated c1.xlarge and abe.local for Broadband, suggesting a relatively low virtualization penalty.Broadband’s lower I/O requirements make the abe.local versus abe.lustre difference less significant than for Montage.
- Epigenomics: About 10% virtualization overhead was estimated for CPU-bound Epigenomics, while additional processor cores reduced runtime.The abe.lustre and abe.local runtimes differed by only about 2%, consistent with Epigenomics’ relatively low I/O.
5. Cost Analysis
The cloud cost analysis separates resource, storage, and transfer charges. Resource usage dominated execution costs, while storage and transfer costs depended on how data and workflows were managed.
- Cost categories: Cloud costs comprise resource, storage, and transfer charges for workflow execution and data management.Resource charges cover EC2 instances; storage charges cover VM images in S3 and input data in EBS; transfer charges cover data and logs moving between the submit host and EC2.
- Resource cost: $0.10/hr to $0.80/hr were the hourly EC2 rates, with partial hours rounded up to full hours.The rates varied by instance type, including $0.20/hr for c1.medium and $0.80/hr for c1.xlarge.
- Resource cost: c1.medium had the lowest execution cost for all three applications despite not having the lowest hourly rate.The instance type did not achieve the best performance in any experiment, but it was the most cost-effective.
- Storage cost: $0.22 per month was the fixed cost for storing the 32-bit and 64-bit VM images, while variable image-access costs totaled approximately $0.03.Input-data storage also incurred 3.18 million I/O operations costing $0.30 in total.
- Transfer cost: $0.10 per GB inbound and $0.17 per GB outbound were the EC2 transfer charges.The transfer analysis counted workflow input, output, and logging data; Condor communication costs were estimated below $0.01 per workflow but excluded.
6. Discussion
EC2 performance was generally reasonable relative to the resources provisioned, but I/O-intensive workflows faced disadvantages from the absence of high-performance parallel file systems. Cost decisions required balancing instance performance against storage and transfer choices.
- 6.1 Performance: Comparable resources produced comparable performance between EC2 and Abe in the experiments.The EC2 c1.xlarge type, nearly equivalent to abe.local, delivered nearly the same performance.
- 6.1 Performance: Montage remained at a significant disadvantage on EC2 because it lacked a high-performance parallel file system.Commodity networking also limited the expected benefit of shifting I/O to a networked parallel file system.
- 6.1 Performance: Virtualization overhead was expected to decrease or disappear as virtualization technologies improve.The passage characterizes the remaining overhead as small.
- 6.2 Cost: c1.medium was the most cost-effective resource type because m1.small’s lower hourly rate was outweighed by its poor performance.The discussion frames EC2 provisioning as a performance–cost tradeoff rather than a choice based only on hourly price.
- 6.2 Cost: Storing input data is more cost-effective when it is reused often and accessed frequently, whereas transfer is more cost-effective for one-time use.The choice also depends on storage duration and access frequency.
- 6.2 Cost: $149.55 was the total cost of all reported experiments, including learning EC2, creating VM images, and running workflows.This total includes all charges associated with the experiments.
7. Conclusion
The paper evaluates scientific workflows on Amazon EC2 against NCSA’s Abe cluster, finding that EC2 can approach comparable performance with equivalent resources while offering distinct cost tradeoffs. Its main performance gap concerned I/O-intensive workloads, where Abe’s interconnect and parallel file system helped substantially.
- 7. Conclusion: The study ran diverse scientific workflows on Amazon EC2 and compared their performance and cost with NCSA’s Abe cluster.The conclusion summarizes both the experimental comparison and the cloud cost analysis.
- 7. Conclusion: EC2 performance was reasonable given available resources and approached Abe performance when resources were equivalent.The measured virtualization overhead was between 1% and 10% for the tested applications.
- 7. Conclusion: Abe’s high-speed interconnect and parallel file system substantially improved performance for the I/O-intensive application.Additional Abe tests using local disk indicated that equivalent-resource performance could be close to EC2’s.
- 7. Conclusion: Resource acquisition was the primary workflow cost, while storage costs were relatively small by comparison.The conclusion also reports that storing data in the cloud can reduce workflow costs compared with transferring it from outside.