Source-linked AI summary
Compute and Storage Clouds Using Wide Area High Performance Networks
Robert L. Grossman, Yunhong Gu, Michael Sabala, Wanzhi Zhang
TL;DR
Large datasets make data movement and conventional database infrastructure limiting, especially when distributed clusters are connected by relatively small bandwidth. The paper introduces Sector and Sphere for wide-area high-performance networks, and reports that distributed-data computing can achieve approximately the same efficiency as local-data computing.
Problem
Large datasets make data movement costly, while prior data-cloud systems assumed relatively small bandwidth between distributed clusters.
Method
The paper develops the Sector storage cloud and Sphere compute cloud, using specialized transport protocols for wide-area high-performance networks and applying them to data analysis.
Results
Computing with distributed data achieved approximately the same efficiency as computing with local data in the described applications.
Takeaways & Limitations
A cloud architecture can support storage and parallel analysis of large distributed datasets over wide-area high-performance networks.
Abstract
from arXiv · showhide
We describe a cloud based infrastructure that we have developed that is optimized for wide area, high performance networks and designed to support data mining applications. The infrastructure consists of a storage cloud called Sector and a compute cloud called Sphere. We describe two applications that we have built using the cloud and some experimental studies.
1 Introduction
Cloud infrastructures layer storage, data, and compute services to support cloud-based applications. This paper targets large terabyte-scale data management and analysis over geographically distributed nodes connected by 10+ Gbps networks.
- Cloud model: Clouds provide resources or services over the Internet, commonly layering compute over data over storage services.These layers form a computing platform for developing cloud-based applications.
- Motivation: Data clouds become more attractive than conventional databases as datasets grow beyond a few hundred terabytes.Google’s GFS is cited as managing petabytes of data.
- Motivation: Replication supports locality-aware performance, backup, archiving, failure handling, and parallel computation.Replicas may be placed within racks, across racks, and across data centers, with automatic replacement after failures.
- Motivation: Cloud storage lets data wait for computation, avoiding repeated movement to available nodes that can consume a significant share of processing time.This contrasts with grid environments that move data to nodes when a sufficiently large pool of nodes is available.
- Contribution: The proposed infrastructure supports ingestion, management, analysis, and distribution of large terabyte-scale datasets over geographically distributed storage nodes linked by a 10+ Gbps wide-area backplane.The design is optimized for high-performance wide-area networks and assumes an OptIPuter-style architecture.
- Contribution: The paper presents Sector as a storage cloud, Sphere as a compute cloud, and applications using both systems.The paper also includes experimental studies and organizes these topics across its later sections.
2 Related Work
Prior data-cloud systems commonly move computation to persistently stored data but assume relatively limited intercluster bandwidth. The paper contrasts this with Sector and Sphere, which target wide-area high-performance networks and support flexible computation and efficient data transfer.
- Existing systems: Existing data-mining platforms range from single workstations to clusters, distributed clusters, grids, and service-oriented systems.The cited systems reflect progressively distributed execution environments.
- Existing systems: Cluster and grid data-mining systems generally treat processors as scarce shared resources and move data to them when processors become available.This supercomputing or Teragrid model can spend substantial time transporting data.
- Existing systems: Data-center models instead persistently store data and run computations over it when required, so the data waits for the task or query.S3, GFS, and HDFS support this model.
- Novelty: Earlier data-cloud work assumed relatively small bandwidth between distributed data clusters, whereas Sector targets 10 Gbps wide-area networks and uses UDT.The contrast defines the paper’s network-oriented novelty.
- Novelty: MapReduce extracts, shuffles, and reduces data, while Sphere permits arbitrary user-defined operations and uses specialized protocols during wide-area shuffling.Sphere replaces both the map and reduce operations with user-defined computation.
3 Sector Storage Cloud
Sector separates routing from storage to locate file metadata, manage replicated files, and transfer data across geographically distributed nodes. Its access model supports community-controlled writes and broadly open reads, while specialized protocols handle wide-area transport.
- Architecture: Sector has routing and storage layers, with services such as Sphere implemented over the storage layer.The routing layer locates metadata, while the storage layer manages files and replicas.
- Routing: Sector’s routing layer maps names to metadata locations and currently uses the Chord P2P routing protocol.Future versions are intended to support specialized protocols for uniform and non-uniform wide-area clouds.
- Transport: UDT provides rate-based transport for large data flows, sharing bandwidth fairly among large flows while backing off for TCP traffic.This transport is used for Sector data transfer over high-performance networks.
- Access protocol: A typical access session looks up an entity through the routing layer, receives one or more locations, establishes data connections, and then uses UDT for requests and responses.Clients may connect to any known Sector server before receiving the relevant locations.
- Access control: Sector serves communities in which authorized users can write data, while community members or the public can generally read it unless restrictions are imposed.This access model differs from systems targeted exclusively at organizations or virtual organizations.
- Operations: Sector automatically updates membership, creates replicas for archival storage and distribution, and supports parallel computation.Data providers added to access-control lists can upload files, and clients can access stored data through known server nodes.
4 Sphere
Sphere provides distributed computing over persistent Sector data, processing records in parallel while avoiding data movement when possible. Its stream-oriented interface simplifies expressing and automatically distributing common data-intensive loops.
- Sphere overview: Sphere performs distributed computations over persistent Sector data, locating resources, balancing load, and providing fault tolerance while avoiding data movement when possible.Sphere implicitly parallelizes processing by locating computing nodes and managing required data movement.
- Stream processing: Sphere divides distributed data into chunks and applies the same processing function independently to each record in parallel.After each processing stage, data may be transferred between nodes before another processing function is applied.
- Programming interface: The stream model replaces serial loops with a concise sphere.run(data, process) operation.The example expresses the processing function and data source without explicitly implementing distributed execution.
- Programming interface: Sphere automatically parallelizes and distributes loops that developers otherwise spend substantial effort implementing with MPI or PVM.The interface targets common data-intensive loops while limiting the supported operation types.
- Design context: Sphere uses a simplified data-parallel and master/worker model, related to stream-processing approaches also used by MapReduce and GPGPU.The paper characterizes stream processing as a special-purpose parallel model with prior successful uses.
5 Sector/Sphere Applications
The Sector and Sphere applications demonstrate wide-area data distribution, TCP-traffic analysis, and distributed sorting on high-performance testbeds. Experiments report large-scale SDSS usage, scalable clustering, and faster Sphere sorting than Hadoop in the stated benchmark.
- Experimental setup: The Teraflow Testbed connects geographically distributed sites with 10 Gbps networks and uses small clusters of dual-core Opteron servers.The testbed includes sites in the United States, Japan, and Korea.
- SDSS distribution: Sector distributed Sloan Digital Sky Survey releases totaling 27TB, with files ranging from 5GB to 100GB.The releases comprise 13TB of DR5 data and 14TB of DR6 data.
- SDSS distribution: 200TB of data was transferred to end users through about 5000 system accesses during the last 18 months of reported SDSS distribution.Sector had distributed the SDSS since July 2006.
- Performance measures: Sector’s wide-area performance is measured by LLPR, the wide-area transfer performance divided by local-area performance on identical hardware.An LLPR of 1.0 is the maximum possible value; higher values indicate better performance.
- TCP traffic analysis: Angle analyzes anonymized TCP packet data from geographically distributed sites to identify anomalous or suspicious behavior.Sector manages collected packet data, while Sphere services process it; each day involves approximately 575 pcap files, 7.6GB, and 97 million packets.
- TCP traffic analysis: Sphere clustering scales with record counts from 500 to 100,000,000 records, according to Table 2.The clustering workload computes models from distributed pcap data using k-means.
- Hadoop versus Sphere: Sphere sorted the benchmark data approximately 2 to 3 times faster than Hadoop, while using one core per node versus Hadoop’s four.The comparison sorts a 10GB file on each of six nodes; file generation was 68 seconds per Sphere node versus 212 seconds per Hadoop node.
6 Summary and Conclusion
The paper presents Sector and Sphere as a high-performance storage and compute cloud architecture for distributed data, using specialized transport protocols over wide-area networks. Two applications and experiments show that distributed-data computing can achieve approximately the efficiency of local-data computing.
- Large data sets are stored in Sector and analyzed in parallel through the Sphere compute cloud.The architecture combines long-term storage with computing cycles available when required.
- Sector and Sphere use specialized protocols such as UDT to use wide-area high-bandwidth network capacity efficiently.
- Two applications and experiments show that distributed-data computing can be approximately as efficient as computing with local data.