Source-linked AI summary
Scientific Computing in the Cloud
J. J. Rehr, J. P. Gardner, M. Prange, L. Svec, F. Vila
TL;DR
The paper asks whether cloud computers can provide a practical alternative for high-performance scientific computing without requiring researchers to maintain specialized hardware. It ports FEFF to Amazon EC2, packages user-deployable environments, and benchmarks serial and parallel performance; preliminary results show EC2 parallel scaling similar to a physical cluster and suggest AWS is reasonably adaptable, while further production and communication testing remains planned.
Problem
The paper investigates whether virtualized cloud resources can support high-performance scientific computing while reducing the need to purchase and maintain specialized HPC hardware.
Method
The authors port FEFF and JFEFF to AWS machine images, develop cluster-management tools, and benchmark serial and MPI-parallel FEFF on EC2.
Results
EC2 parallel scaling was similar to that of a conventional physical cluster, and preliminary results found AWS EC2 reasonably adaptable for accessible high-performance scientific computation.
Takeaways & Limitations
Cloud deployment can provide FEFF users with accessible virtual compute clusters and storage without requiring them to purchase or maintain HPC hardware.
Abstract
from arXiv · showhide
We investigate the feasibility of high performance scientific computation using cloud computers as an alternative to traditional computational tools. The availability of these large, virtualized pools of compute resources raises the possibility of a new compute paradigm for scientific research with many advantages. For research groups, cloud computing provides convenient access to reliable, high performance clusters and storage, without the need to purchase and maintain sophisticated hardware. For developers, virtualization allows scientific codes to be optimized and pre-installed on machine images, facilitating control over the computational environment. Preliminary tests are presented for serial and parallelized versions of the widely used x-ray spectroscopy and electronic structure code FEFF on the Amazon Elastic Compute Cloud, including CPU and network performance.
I. Introduction
The study investigates whether AWS cloud services can support high-performance scientific computing by porting FEFF and benchmarking its serial and parallel performance on EC2.
- I. Introduction: The project evaluates AWS cloud computing for high-performance scientific computing through FEFF benchmarks and user-deployable virtual clusters.FEFF computes x-ray spectra and electronic structure for complex systems on large atomic clusters.
II. Achievements
The work combines performance benchmarking of FEFF on EC2 with development of an environment for users to run FEFF versions in their own cloud clusters.
- II. Achievements: The project has two main efforts: benchmarking FEFF performance on EC2 and developing user-deployable EC2 compute clusters.Both efforts required understanding the AWS infrastructure.
A. Understanding the EC2 and S3 infrastructures
EC2 hosts reusable machine images whose instances can be launched elastically, while S3 stores images and files; command-line, API, library, and graphical tools support their management.
- A. Understanding the EC2 and S3 infrastructures: EC2 hosts customizable Amazon Machine Images, which users can store in S3 and launch in arbitrary numbers as computational demand changes.This elasticity scales computational resources to instantaneous needs.
- A. Understanding the EC2 and S3 infrastructures: AMI, API, and S3 tools bundle images, control EC2 instances, and manage stored files through Python, Ruby, Java, Bash, and related interfaces.The available language implementations let developers tailor their implementations.
- A. Understanding the EC2 and S3 infrastructures: Elasticfox and S3fox provide browser-based interfaces for viewing and managing EC2 instances and transferring S3 files.They offer an alternative for new AWS users.
B. Implementation of FEFF and JFEFF on EC2
The authors packaged FEFF84 and its JFEFF graphical interface in an AWS machine image, enabling remote execution while revealing slower GUI response over the network.
- B. Implementation of FEFF and JFEFF on EC2: A Fedora 8 AWS machine image packages command-line FEFF84 and the JFEFF Java GUI for execution on EC2.The image was enhanced with X11 and a Java runtime so JFEFF could function remotely.
- B. Implementation of FEFF and JFEFF on EC2: The EC2 deployment's notable limitation is relatively slow JFEFF GUI response over a network.The screenshot shows JFEFF, the FEFF console, and the Elasticfox control screen.
C. Testing FEFF84 serial performance
The study benchmarks FEFF84 on EC2 and develops tools and images intended to make cloud-based scientific computing usable for both novice and experienced HPC users.
- C. Testing FEFF84 serial performance: FEFF84 was tested on EC2 instances with different computing power using typical full multiple scattering calculations across atomic cluster sizes.The Boron Nitride calculations ranged from 29 to 157 atoms, with about 100 atoms typically sufficient for converged spectra.
- C. Testing FEFF84 serial performance: Figure 2 compares serial runtime against cluster size for different AMI instance types running typical FEFF84 calculations.
- D. Development of strategies and tools for parallel cluster computing: Python tools launch configured EC2 Linux images for parallel computation, while improved scripts start, configure, monitor, and terminate MPI clusters.
E. Testing FEFF84 parallel performance
The authors evaluate FEFF84MPI scalability on EC2 against a local physical cluster and identify further work needed for production cloud-HPC assessment.
- E. Testing FEFF84 parallel performance: Similar FEFF84MPI scaling on EC2 and the local 1.8 GHz AMD Opteron cluster indicates that EC2 virtualization did not degrade parallel performance.
- E. Testing FEFF84 parallel performance: The preliminary study used the developed cluster-management tools to assess FEFF84MPI scalability on EC2.
- Remaining challenges and goals: Future high-performance AMIs will use 64-bit systems, compilers, and numerical libraries to support a representative production environment for economic assessment.
- Remaining challenges and goals: Further testing will optimize interprocess communication, which is expected to matter more for many other codes than for the moderately coupled FEFF program.
- E. Testing FEFF84 parallel performance: The report concludes that AWS EC2 is reasonably adaptive for making high-performance scientific computation available to the scientific community, despite the preliminary scope.