Source-linked AI summary
High-Performance Cloud Computing: A View of Scientific Applications
Christian Vecchiola, Suraj Pandey, Rajkumar Buyya
TL;DR
Scientific computing needs large, flexible infrastructures, but traditional HPC and Grid environments can constrain software and access. The paper presents Aneka as a service-oriented Cloud platform and demonstrates it through gene-expression classification and fMRI workflow experiments. The preliminary results show faster execution with increased Cloud resources, alongside a cost–performance trade-off.
Problem
Scientific computing requires large-scale resources, while Grid environments may lack required software in application runtime environments and Cloud use remains limited.
Method
The paper uses Aneka across private and public Clouds to support multiple programming paradigms and evaluates gene-expression classification and fMRI workflow execution.
Results
The preliminary experiments demonstrate Cloud-based gene-expression classification and scientific-workflow execution, with larger workflows completing faster on EC2 as resources increase.
Takeaways & Limitations
Cloud resources can support scientific applications, but effective use requires evaluating the trade-off between execution cost and performance.
Abstract
from arXiv · showhide
Scientific computing often requires the availability of a massive number of computers for performing large scale experiments. Traditionally, these needs have been addressed by using high-performance computing solutions and installed facilities such as clusters and super computers, which are difficult to setup, maintain, and operate. Cloud computing provides scientists with a completely new model of utilizing the computing infrastructure. Compute resources, storage resources, as well as applications, can be dynamically provisioned (and integrated within the existing infrastructure) on a pay per use basis. These resources can be released when they are no more needed. Such services are often offered within the context of a Service Level Agreement (SLA), which ensure the desired Quality of Service (QoS). Aneka, an enterprise Cloud computing solution, harnesses the power of compute resources by relying on private and public Clouds and delivers to users the desired QoS. Its flexible and service based infrastructure supports multiple programming paradigms that make Aneka address a variety of different scenarios: from finance applications to computational science. As examples of scientific computing in the Cloud, we present a preliminary case study on using Aneka for the classification of gene expression data and the execution of fMRI brain imaging workflow.
I. INTRODUCTION
Scientific computing has relied on difficult-to-access HPC and Grid infrastructures, while Cloud computing offers dynamically scalable, customizable, pay-per-use resources. The paper introduces Aneka and examines Cloud-based scientific computing through gene-expression classification and scientific-workflow case studies.
- Scientific experiments require many computing resources to execute large-scale models within a reasonable time.
- Scientific Grids provide large distributed infrastructures but can be difficult to access because of proposal-based allocation and runtime-environment constraints.
- Cloud computing uses virtualization to provide customizable services across the computing stack, with resources scaled according to application requirements and user budgets.
- Cloud resources can be provisioned and released on demand through a pay-per-use model, avoiding capacity planning for experiments.
- Aneka is presented as a Cloud platform for developing applications that can scale on demand, alongside other infrastructure-, platform-, and application-level solutions.
- The paper presents gene-expression classification and scientific-workflow execution on Amazon EC2 as case studies of Cloud-based scientific computing.
B. Cloud Computing Reference Model
The Cloud computing reference model presents a layered stack that moves from physical resources toward services and applications. Physical infrastructure supplies the computing power, while middleware manages it and can use virtualization for isolation and quality of service.
- Figure 1 presents Cloud computing as a layered view of IT infrastructure, services, and applications that shifts perspective from system to end user.
- The lowest layer contains physical resources such as clusters, datacenters, and spare desktop machines.
- Commercial Clouds typically use datacenters with hundreds or thousands of machines, whereas private Clouds may combine heterogeneous resources including idle desktop capacity.
- Core middleware manages the physical infrastructure, provides application runtime environments, and exploits resources efficiently.
- Virtualization can support application isolation, quality of service, and sandboxing within the middleware layer.
C. Cloud Computing Services Offering
Cloud services are organized into Infrastructure as a Service, Platform as a Service, and Software as a Service. The paper contrasts these offerings and positions Aneka as a pure PaaS for distributed Cloud applications.
- Cloud services are classified into Infrastructure as a Service, Platform as a Service, and Software as a Service.
- Infrastructure as a Service: IaaS delivers virtual or physical infrastructure, including compute, memory, storage, and operating-system resources, typically billed on a pay-per-use basis.
- Platform as a Service: PaaS provides application or development platforms with frameworks and APIs for creating Cloud applications.
- Platform as a Service: Google AppEngine and Microsoft Azure provide application models and services for developing scalable Cloud applications.
- Platform as a Service: Aneka is a pure PaaS that uses .NET to support distributed-application development and can be deployed across physical or virtual infrastructures.
- Software as a Service: SaaS provides integrated hosted applications while limiting end-user customization of the underlying service.
III. ANEKA
Aneka is a service-oriented platform for developing and managing distributed Cloud applications across heterogeneous private or public resources. Its extensible architecture supports multiple programming models and on-demand integration with public Clouds.
- Aneka provides APIs and programming abstractions for transparently exploiting heterogeneous desktop, server, and datacenter resources on demand.
- Aneka combines application-development APIs with administrative tools for monitoring and controlling public or private Cloud deployments.
- Its flexible service-oriented architecture supports scenarios ranging from financial applications and teaching to large-scale scientific experiments.
- Aneka supports different application programming patterns through an extensible set of programming models.
- Aneka can integrate with Amazon EC2, acting as a middleman between user applications and public-Cloud resources.
- The platform includes a software development kit, management kit, and configurable service-based container as core building blocks.
A. Aneka Clouds
Aneka Clouds use container-based, service-oriented infrastructure deployed across physical or virtual resources. They support private, public, and hybrid configurations with elastic provisioning and customizable node roles.
- Container-based architecture: Aneka Clouds consist of networked software containers hosted on physical or virtual resources.Each container exposes services that customize the runtime environment for applications.
- Container-based architecture: Fabric, foundation, execution, persistence, and security services collectively provide the container’s runtime and management capabilities.Fabric services interact with nodes, while foundation services provide core infrastructure features.
- Deployment configurations: Aneka supports private, public, and hybrid deployments, with hybrid systems extending local infrastructure using additional virtual resources.Public-cloud deployments may use platforms such as Amazon EC2, while private deployments can use physical or virtual infrastructure.
- Elasticity: Nodes can be provisioned or released on demand, either manually or through scheduler-driven elastic and autonomic management.Releasing virtual resources also requires terminating the virtual machine hosting the container.
- Elasticity: Membership Services track cloud nodes and support dynamic discovery of services, configurations, and operating-system capabilities.Scheduling services can use this registry to locate nodes able to execute a given programming model.
- Deployment configurations: A classic deployment assigns scheduling services to a limited number of nodes while configuring most containers as compute resources.This creates a master-slave topology, although Aneka supports other configurations.
B. Aneka Application Model
The Aneka Application Model defines common abstractions and runtime requirements for integrating distributed applications with Aneka services. It supports multiple programming-model implementations while retaining shared management functions.
- Model foundations: The application model defines abstractions and runtime requirements that specific programming models must satisfy for seamless Aneka integration.These requirements describe how distributed applications are structured and executed within the cloud.
- Model foundations: An Aneka application is a deployable collection of execution units whose scheduling and execution depend on its programming model.Configuration and security operate at the application level rather than for isolated tasks.
- Model foundations: Aneka executes user-code units only within distributed applications rather than as standalone tasks.This application-level organization provides a common execution context across supported programming models.
- Programming-model implementation: Implementing a programming model requires defining its abstractions, providing execution services, and implementing a coordinating client component.These components are common to different programming-model implementations.
- Programming-model implementation: The model comparison describes each programming model through its application type, execution units, execution services, and user and system perspectives.The comparison demonstrates the flexibility of the Aneka Application Model.
C. Accouting, Pricing, and Integration with Public Clouds
Aneka combines accounting and pricing services with extensible scheduling to manage resource usage and costs, particularly when integrating public-cloud resources and meeting SLAs.
- Accounting and pricing: Accounting and pricing collectively implement Aneka’s mechanism for charging application resource usage.These functions are especially relevant in public and hybrid clouds, where resource control and usage pricing are required.
- Accounting and pricing: The accounting service records usage statistics by user and application and tracks execution-unit time and execution history.Pricing strategies use these data to determine user charges.
- Public-cloud integration: Hybrid-cloud virtual resources are provisioned to meet the Service Level Agreement signed with users, while their costs are included in billing.This makes resource provisioning and pricing interdependent in hybrid deployments.
- Public-cloud integration: Aneka provides an object model through which third parties can integrate scheduling algorithms for coordinating cloud resources.Different policies can determine when the cloud’s node set should grow or shrink.
IV. CASE STUDIES
The paper presents two scientific-computing case studies implemented on Amazon EC2: gene-expression classification and an fMRI brain-imaging workflow compared with traditional Grids.
- Scientific-computing applications: Both case studies were implemented on the Amazon EC2 infrastructure.The studies examine scientific computing using an Aneka Cloud.
- Scientific-computing applications: The first case study uses Aneka to classify gene-expression datasets.It illustrates a scientific-computing application executed in the Cloud.
- Scientific-computing applications: The second case study executes an fMRI brain-imaging workflow and compares its performance with the same experiment on traditional Grids.The case studies also include cost analysis.
A. Classification of Gene Expression Data
The case study applies CoXCS, a feature-partitioned co-evolutionary classifier, to gene expression data using Aneka on Amazon EC2. Cloud execution enables distributed training, but the tested configuration shows that extra cores can increase runtime without reducing cost.
- Gene expression profiling measures thousands of genes simultaneously, producing datasets whose cancer-diagnosis classification requires computerized techniques.
- CoXCS trains independent classifier populations on different feature-space partitions and combines selected classifiers after fixed iterations.
- CoXCS accuracy in the test phase is reported as better than that of the original XCS and classic classifiers.
- Cloud-CoXCS distributes the evolution of independent classifier populations across Aneka tasks to address CoXCS’s long computation time.
- 32% more time was required on average by c1.medium than m1.small because single-threaded tasks could not benefit from two cores and competed for shared resources.
- The BRCA and Prostate experiments completed in less than one and two hours respectively, yet both setups incurred the same cost under the provider’s pricing granularity.
B. Functional Magnetic Resonance Imaging Workflows
The fMRI workflow processes image data through spatial normalization and is both data- and compute-intensive. Experiments on EC2 show that scaling resources substantially reduces makespan, with higher but more efficient per-machine cost.
- Workflow: fMRI analysis processes MRI images to determine which brain regions respond to a stimulus.The workflow includes spatial normalization to reduce anatomical variability.
- Workflow: The spatial-normalization stage is modeled as a data- and compute-intensive workflow.Only this stage of the full fMRI process is represented as a workflow.
- Workload: A 20-image workflow processes 640MB of input, while 40 images can produce more than 20GB of processed data.Individual input images are approximately 16MB, and process outputs range from 20MB to 40MB.
- Experimental setup: The experiment used EC2 for computing, S3 for input storage, and compared workflow execution with Grid’5000 using makespan and execution cost.Grid’5000 execution cost was assumed to be zero.
- Results: For large workflows, EC2 reduces makespan because data transfer between virtual nodes is shorter than transfer across multiple Grid’5000 sites.The difference is not significant for two subjects but becomes substantial for 20 subjects as file-transfer times accumulate.
- Results: 391 minutes to 107 minutes: increasing EC2 nodes from 2 to 20 significantly decreases workflow makespan.The overall cost rose from $5.2 to $14.28, while per-machine cost for 20 subjects fell to $0.714 versus $2.6 using two nodes.
- Conclusion: Cloud resources can reduce execution time substantially, while the increase in cost remains subdued for large high-performance applications.The authors conclude that on-demand access and scalability benefit these applications.
V. OBSERVATIONS AND THOUGHTS
Cloud computing enables infrastructure to scale dynamically and charges users according to actual needs, but public-cloud use introduces legal, security, privacy, and data-governance challenges.
- Benefits: Dynamic scaling and pay-per-use infrastructure are major advantages of Cloud computing for scientific institutions.These capabilities reduce the need to provision infrastructure statically for peak loads.
- Challenges: Public-cloud deployments raise legal and security concerns because systems, applications, and personal data may reside in third-party datacenters.Datacenter location can affect applicable law and the confidentiality of stored data.
- Challenges: Government-funded scientific projects may face significant restrictions on the use and management of sensitive data.These restrictions make cloud data handling relevant even for scientific computing.
VI. CONCLUSIONS AND FUTURE DIRECTIONS
Scientific cloud computing remains at an early stage despite established Grid infrastructures. The paper presents Aneka case studies showing scalable cloud execution, while emphasizing careful cost–performance evaluation and the need for further study.
- State of the field: Scientific Cloud computing is still at an early stage, although science Grids already provide large-scale infrastructures and established tools.The paper contrasts the maturity of Grids with emerging cloud adoption in scientific computing.
- Opportunities: Clouds offer scientific institutions customizable runtime environments and configurable Quality of Service for experiments.Service Level Agreements can support choices about experimental accuracy or the data partition analyzed.
- Aneka: Aneka supports scientific applications by leveraging public and private Clouds and scaling resources on demand according to application requirements.The demonstrated cases include gene-expression classification on Amazon EC2 and scientific workflow execution on EC2.
- Implications: Cloud experiments require careful evaluation of the trade-off between cost and performance.Aneka is presented as a platform that mediates access to cloud resources and supports resource utilization and pricing.