PaperlayerView exact revision on arXiv

Source-linked AI summary

The Anatomy of Big Data Computing

Raghavendra Kune, Pramodkumar Konugurthi, Arun Agarwal, Raghavendra Rao Chillarige, Rajkumar Buyya

arXiv:1509.01331v1cs.DC

TL;DR

The paper addresses how rapidly growing, multidimensional data can support knowledge discovery and decision making through suitable organization, processing, storage, and analytics. It synthesizes Big Data computing’s characteristics, technologies, cloud integration, layered architecture, applications, and open challenges. Its conclusion is a comprehensive classification and architecture framework for understanding Big Data computing and identifying gaps for future work.

  • Problem

    Rapidly growing data creates challenges for organization and processing in scientific and business decision making, requiring large-scale data-intensive computing infrastructure.

  • Method

    The paper characterizes and categorizes Big Data computing, surveys underpinning technologies and tools, and presents Big Data Clouds, their types, access networks, layered architecture, and components.

  • Results

    The paper delivers a comprehensive classification framework and architecture-oriented study that compares existing systems, identifies shortcomings and gaps, and describes Big Data computing over Clouds.

  • Takeaways & Limitations

    Big Data Clouds combine Big Data and cloud computing to deliver analytics as a service over large-scale cloud infrastructure for enterprise, scientific, and engineering applications.

  • Takeaways & Limitations

    The paper identifies unresolved needs for distributed mining, statistical-significance tools, historical-plus-real-time analytics, domain-specific models, and effective indexing and processing.

Abstract

from arXiv · show

Advances in information technology and its widespread growth in several areas of business, engineering, medical and scientific studies are resulting in information/data explosion. Knowledge discovery and decision making from such rapidly growing voluminous data is a challenging task in terms of data organization and processing, which is an emerging trend known as Big Data Computing; a new paradigm which combines large scale compute, new data intensive techniques and mathematical models to build data analytics. Big Data computing demands a huge storage and computing for data curation and processing that could be delivered from on-premise or clouds infrastructures. This paper discusses the evolution of Big Data computing, differences between traditional data warehousing and Big Data, taxonomy of Big Data computing and underpinning technologies, integrated platform of Big Data and Clouds known as Big Data Clouds, layered architecture and components of Big Data Cloud and finally discusses open technical challenges and future directions.

1 Introduction

Big Data computing emerges as a paradigm for mining rapidly growing, multidimensional data to support scientific discovery, business analytics, and decision making on large-scale infrastructure. The paper introduces its applications, relationship with cloud computing, and planned coverage of technologies, architectures, and challenges.

  • 1 Introduction: Big Data computing mines multidimensional information for scientific discovery and business analytics over large-scale infrastructure.It supports data management, analysis, validation, visualization, and dissemination while preserving data value.
  • 1 Introduction: Digital data growth, semiconductor advances, and cloud services are increasing the feasibility of preserving and processing large data volumes.The cited IDC report predicted a 40-fold increase in digital data from 2012 to 2020.
  • 1 Introduction: Big Data technologies address data-warehousing weaknesses by organizing and querying streaming, heterogeneous data from sources such as click logs, sensors, mobile devices, emails, and videos.These sources support analytics that extract information for enterprise use.
  • 1 Introduction: Applications span scientific exploration, healthcare, governance, and financial or business analytics.Examples include sensor analysis, disease-spread prediction, real-time traffic analysis, fraud detection, sentiment analysis, and predictive pricing.
  • 1 Introduction: Cloud computing delivers compute, storage, and software through pay-as-you-go services, while Big Data Clouds integrate these technologies into an analytics platform.The paper presents this integration alongside comparisons with warehousing, taxonomies, and a cloud architecture framework.

2 Big Data Characteristics – Traditional Data Vs Big Data paradigms

Big Data extends traditional warehousing by supporting large, diverse, rapidly changing data on distributed scale-out infrastructure, with analytics over integrated data platforms. The section also contrasts their consistency models and abstraction layers.

  • 2 Big Data Characteristics – Traditional Data Vs Big Data paradigms: Traditional databases target smaller volumes of structured data with predictable updates, whereas Big Data handles diverse formats through batch and stream processing.Big Data architectures use distributed storage and compute rather than relying primarily on a single server.
  • 2 Big Data Characteristics – Traditional Data Vs Big Data paradigms: Traditional warehousing uses subject-oriented, integrated, time-variant, and nonvolatile data for analysis, including OLAP and knowledge-discovery operations.The described operations include slice-and-dice, drill down, drill up, pivoting, classification, prediction, and visualization.
  • 2 Big Data Characteristics – Traditional Data Vs Big Data paradigms: Big Data integrates analytics over unstructured data with scale-out compute and data platforms, while traditional systems separate operational and historical data.Big Data tools support decision-support software, distributed storage and compute nodes, and large-scale infrastructure.
  • 2.3 CAP Theorem – ACID and BASE: Traditional databases follow ACID properties, whereas distributed Big Data systems follow BASE properties that accommodate eventual consistency and availability.BASE includes basically available and soft-state behavior, with data becoming eventually consistent after input stops.
  • 2.3 CAP Theorem – ACID and BASE: CAP states that a networked shared-data system can provide only two of consistency, availability, and partition tolerance.The paper defines consistency across nodes, response availability, and continued operation despite node or message failures.
  • 2.4 Big Data – Abstraction Layers: Big Data uses four abstraction layers—physical, data, computing, and analytics—rather than traditional physical, logical, and view/user layers.The physical and data layers distribute, partition, index, and retrieve heterogeneous data; the computing layer provides modeling, query, and programming abstractions.

3 Big Data Taxonomy

Big Data taxonomy organizes data-intensive computing around data dimensions, analytics techniques, storage and processing technologies, and programming models. These elements support analysis of rapidly growing, diverse data beyond traditional relational approaches.

  • Taxonomy overview: Big Data addresses investigations over rapidly growing data, using expanded storage and processing capabilities to analyze full or near-real-time data sets.The paper contrasts this with traditional systems that commonly rely on structured transactional and summarized data.
  • Data dimensions: The taxonomy includes data dimensions such as Volume, Velocity, Variety, Veracity, and Value, which characterize the scale, speed, forms, quality, and usefulness of data.The paper presents the first four dimensions as 4V’s and identifies Value/Valor as an additional dimension.
  • Analytics techniques: Big Data analytics combines statistical models, data mining, and computing technologies to extract information from varied data sets.Examples include text analytics, in-memory analytics, predictive analysis, and graph analytics.
  • Underlying technologies: Big Data technologies are grouped into file systems, open-source computing frameworks, and analytics tools for organizing, processing, and interpreting data.The paper discusses distributed file-system requirements, Hadoop and Spark, and NoSQL key-value stores.
  • Programming and scheduling models: Programming models include data-intensive, stream, batch, high-performance, query, and column-oriented processing, with MapReduce separating filtering and sorting from aggregation.Scheduling mechanisms include query schedulers and data-aware schedulers such as Capacity and Fair schedulers.

4 Big Data in Clouds: An Integrated Big Data and Cloud Platform

Big Data Clouds integrate distributed data, computing, and analytics services on elastically scalable infrastructure. The paper classifies deployment models, describes enterprise benefits, and maps the platform into infrastructure, platform, and analytics services and layers.

  • Integrated platform: Big Data in clouds provides a data-intensive platform for building analytics quickly and deploying them over elastically scalable infrastructure.The platform integrates data management, computing, programming models, schedulers, security, and analytics services.
  • Deployment models: Public, private, hybrid, and multi-provider access-network models distinguish Big Data Cloud deployments by infrastructure location, control, federation, and service delivery.Hybrid clouds federate public and private resources for scalability, disaster recovery, and high availability.
  • Cloud characteristics: Big Data Clouds combine large-scale distributed compute and storage with metadata-based access, dynamic virtual file systems, and resource selection.These features provide transparent access to compute and data resources while supporting dynamically created virtual machines and data containers.
  • Enterprise services and benefits: Big Data Clouds are presented as enabling enterprises to build applications, reduce development costs through pay-as-you-go services, fuse information, and respond more rapidly to demand.The paper also describes infrastructure, platform, and analytics services delivered under SLA and QoS parameters.
  • Layered architecture: The reference architecture organizes Big Data Cloud services into infrastructure, fabric, platform, and analytics layers.Infrastructure supplies compute, storage, and networking; fabric supplies access and aggregation tools; platform supports processing and scheduling; analytics exposes usage and publishing services.

a) Infrastructure layer

The infrastructure layer manages physical and virtual compute, storage, and data resources for Big Data Cloud services. Virtualization supports elastic, on-demand provisioning but can introduce performance degradation.

  • Layer role: The infrastructure layer provides compute, storage, data, and networking resources through resource and interface sublayers.Its resource layer handles physical and cloud resources.
  • Virtualization trade-off: Virtual environments improve resource utilization and enable on-demand provisioning and scalability, but enforced virtualization technologies can cause performance degradation.The paper states that physical and virtual resource layers function similarly while differing in these operational properties.
  • Physical resources: Physical resources are non-virtualized compute and storage delivered through local data centers or in-house infrastructure.They are accessed through standard protocols and networking interfaces.
  • Virtualized and cloud resources: Cloud resources provide compute, storage, and application services from cloud providers, including on-demand machine instances and storage repositories.Data clouds may provide storage together with the data required for computation.

ii. Interface Layer

The interface layer exposes open, interoperable standards and protocols for accessing compute, storage, data, and application services across heterogeneous cloud infrastructures. Its components cover networking, virtual-machine management, data management, and application services.

  • Interface role: The interface layer provides open-standard, protocol-based interfaces for interoperable access across heterogeneous hardware, storage infrastructure, and cloud providers.Migration and cross-provider access are identified as major challenges.
  • Network Interface: Network interfaces connect physical devices, compute instances, and storage through standard networking interfaces and protocols.Examples include terminal or web-console access and NFS-based storage access.
  • CCMI: CCMI provides interoperable interfaces for on-demand creation and management of virtual machines across public cloud providers.Its scope is virtual-machine lifecycle management.
  • CS/DMI: CS/DMI lets applications create, retrieve, update, and delete cloud data while discovering storage capabilities and managing containers, metadata, security, and monitoring information.The interface exposes underlying storage and data-service capabilities to clients.
  • CASI: CASI exposes published applications as web services and uses application virtualization to deliver needed applications from cloud providers.It is based on standard web protocols.

b) Big Data Platform layer

The Big Data Platform layer is a middleware stack connecting infrastructure resources to applications through resource management, scheduling, programming models, and analytics tools.

  • Platform middleware: The middleware layer comprises foundation, runtime, programming modeling, and SDK sublayers.The foundation manages resources, storage, data, security, and virtual appliances; runtime handles scheduling and jobs.
  • Foundation layer: Foundation services manage physical resources, including utilization monitoring and hardware profiling for compute, storage, RAM, bandwidth, and load.These services track resource availability and retrieve hardware information.
  • Foundation layer: Data management supports structured, unstructured, and semistructured formats, querying, sharing, migration, synchronization, and data discovery.Discovery can use query mechanisms or metadata searches, while transfer tools support movement into cloud Big Data workloads.
  • Foundation layer: Appliances provide self-configuration by integrating hardware and software components, with machine images and domain-specific Big Data appliance repositories.Prebuilt machine templates support on-demand system creation and automated management.
  • Runtime layer: Runtime scheduling includes thread, task, Map Reduce, data-aware, and compute-and-bandwidth-aware mechanisms for assigning workloads to resources.Data-aware scheduling locates computation near replicated data, while Map Reduce maps computation to data nodes and consolidates results.
  • Runtime layer: Parallel data extraction transfers data from storage clouds to compute nodes, while schedulers map jobs and job-management tools monitor executions.Scheduling may select suitable nodes or clone virtual machines using heuristic and data-aware approaches.
  • Programming and application layers: Programming models address threads, tasks, data-intensive computation, and NoSQL queries, while SDKs expose Java, C, C++, and C# APIs.The application layer provides statistical, graph, business, text, and data analytics tools through SDKs, APIs, and management interfaces.
  • Users: Platform users include developers, data scientists, and end users who design, develop, or consume Big Data analytics applications.The described roles span general-purpose application design, analytics development, and analytics use.

5 Gap Analysis and Future Directions

The paper organizes Big Data research into four segments—Depository, Devise, Domain, and Determine—and identifies storage, platforms, domain tools, and analytics as future-work areas.

  • 4D research taxonomy: The 4D taxonomy comprises Depository for storage, Devise for platforms and programming models, Domain for domain-specific tools, and Determine for analytics.The segments structure the paper’s discussion of Big Data research directions.
  • Devise: Devise focuses on distributed, in-memory, and stream computing, query languages, synchronization, and scheduling methodologies.The segment covers programming models and platforms for varied data stores and distributed computation.
  • Domain: Domain concerns platforms and tools tailored to scientific, engineering, and business applications, including weather, earth observation, business intelligence, and social networking analytics.These examples are presented as domain-specific research areas in the taxonomy.
  • Depository: Depository addresses persistent storage and retrieval of structured and unstructured data across geographically dispersed locations.It includes migration toward container-based object storage with location transparency, availability, and fault tolerance.
  • Gap analysis: The paper presents its Depository discussion alongside a gap-analysis and future-directions table.Table 3 is identified as covering Depository gaps and future directions.

2. File system

The file-system discussion spans storage organization, computation, access control, platform services, and domain analytics, emphasizing unresolved needs for scalable Big Data workloads.

  • File system: N-tier storage can place frequently used data on Flash or SSD tiers and migrate it toward disks or cloud storage for backup and archival.The hierarchy moves data across storage tiers according to use and preservation needs.
  • File system: In-memory computing with object-storage file systems is needed for effective computation, long-term preservation, metadata queries, and object-based analysis.The passage links in-memory systems with querying cloud storage pools and object-based data operations.
  • File system: High-availability storage uses redundant components and access paths to restore services after failures, minimizing rather than eliminating downtime.The architecture provides a second critical component or path to data.
  • File system: Big Data Clouds require multidimensional indexing, object-based retrieval, and access policies that protect sensitive data while preserving usable analytical values.Security policies specify which users can access original data and support realistic values for analytics and modeling.
  • Devise: Big Data platforms include Hadoop, Spark, Amazon EMR, Dryad, HDInsight, Aneka, and Map Reduce, while programming-model gaps remain.The paper identifies debugging and profiling needs for Map Reduce and calls for domain-specific languages.
  • Devise: Unstructured-data processing needs improved document, text, graph, indexing, schema-less, and multi-node query mechanisms.The paper also calls for database systems bridging traditional databases and key-value databases.
  • Devise: Compute-intensive workloads require combined Thread, Task, and Map Reduce models, alongside QoS-aware scheduling and HPC in-memory or stream computing.The proposed scheduling parameters include time, budget, and accuracy.
  • Domain and analytics: Domain analytics need specialized indexing, metadata, visualization, NoSQL selection, cloud migration, and analytical models for ranking relevant data.These tools target unstructured, domain-specific data and aim to support dashboards, migration, and more focused analysis.

6 Summary and Conclusions

The paper characterizes Big Data computing as a broad analytics platform and develops classification and architecture frameworks for understanding its technologies, systems, and services. It also identifies continuing research needs for next-generation infrastructure.

  • Big Data computing addresses knowledge discovery and decision making through large-scale multidimensional analytics, combining data organization, computing workloads, and statistical and machine learning tools.
  • The paper presents a taxonomy covering Big Data characteristics, technologies, tools, security mechanisms, data organization, and scheduling approaches.
  • Big Data Clouds integrate Big Data and Cloud computing to deliver analytics computing as a service over large-scale clouds, including access networks as emerging platform services.
  • A layered architecture maps components and relevant technologies, while comparisons of existing systems classify the tools and services available to users.
  • Further research is needed in data organization, decision making, domain-specific tools, and platform tools for next-generation Big Data infrastructure.
Loading 1509.01331v1…