Source-linked AI summary
Big Data Systems Meet Machine Learning Challenges: Towards Big Data Science as a Service
Radwa Elshawi, Sherif Sakr
TL;DR
Big Data Science must analyze growing volumes of data, but the tools and platforms face challenges in scalability, data sharing, and model development. This paper surveys the software stack for offering big-data science as a cloud commodity service. It concludes that data access, interoperability, disk-based execution, and lifecycle management remain important open challenges.
Problem
Big-data analytics must scale to massive datasets and meet rising analysis needs, while data sharing, interoperability, and model development remain challenging.
Source: §3 · p. 3, §4 · pp. 5–6Method
Results
The paper’s survey identifies data sharing, interoperability across distributed platforms, memory-bounded execution, and model lifecycle management as open challenges.
Source: §4 · pp. 5–6Takeaways & Limitations
Cloud storage and elastic computing resources can support faster and more scalable data-science tasks, while effective exploitation of massive, complex data remains essential.
Source: §4 · p. 5Most statistical and scientific-computing tools rely on in-memory processing, risking scalability when data exceed available memory and potentially requiring costly cloud memory.
Source: §4 · p. 6
Abstract
from arXiv · showhide
Recently, we have been witnessing huge advancements in the scale of data we routinely generate and collect in pretty much everything we do, as well as our ability to exploit modern technologies to process, analyze and understand this data. The intersection of these trends is what is called, nowadays, as Big Data Science. Cloud computing represents a practical and cost-effective solution for supporting Big Data storage, processing and for sophisticated analytics applications. We analyze in details the building blocks of the software stack for supporting big data science as a commodity service for data scientists. We provide various insights about the latest ongoing developments and open challenges in this domain.
1. BIG DATA SCIENCE
Big Data Science combines the expanding scale and diversity of data with methods for extracting knowledge and value. The section frames this shift alongside advances in data-intensive computing and describes the article’s focus on a cloud-based service stack and its challenges.
1. BIG DATA SCIENCE
- Big data is characterized by Volume, Velocity, and Variety: data scale, incoming speed and frequency, and diversity of formats, types, sources, and structures.The passage also notes that the term arose in response to the tremendous increase in digital data.Source: §1 · p. 1
- The section describes a shift toward data-intensive science, requiring computing systems that can manage, analyze, and visualize large-scale data.Hadoop and newer platforms for SQL, graph, and stream processing are presented as responses to increasingly complex data-science jobs.Source: §1 · p. 1
- Big data science combines statistics, artificial intelligence, and computer science with fields including databases, machine learning, and distributed systems to turn available data into value.Its stated goal is to convert data into meaningful information.Source: §1 · p. 1
- Advances in storage, memory, processing, and distributed and parallel computing have made it feasible to process massive datasets and extract patterns and knowledge.The article examines software-stack building blocks for delivering Big Data Science as a commodity service and discusses developments and open challenges.Source: §1 · p. 2
2. CLOUD COMPUTING
Cloud computing offers elastic, pay-as-you-go infrastructure for big data storage, processing, and analytics. The section outlines cloud service models and describes how cloud services make large-scale data work accessible on demand.
2. CLOUD COMPUTING
- Cloud computing offers elastic resources, pay-as-you-go consumption, and centralized datacenters intended to reduce resource-management costs.The passage reports that 53% of enterprises had deployed or planned to deploy Big Data Analytics applications in the cloud.Source: §2 · p. 2
- Infrastructure as a Service dynamically allocates storage, processing, networks, and other computing resources to build virtualized systems.The passage lists Amazon EC2, Google Compute Engine, and Rackspace as examples.Source: §2 · p. 2
- Platform as a Service supplies a software platform and lets users control deployed applications and possibly their hosting configurations without deploying cloud resources.The passage names Azure Platform, Google App Engine, Engine Yard, AppFog, and Heroku as examples.Source: §2 · p. 2
- Software as a Service delivers cloud-hosted services through thin clients such as web browsers, without requiring consumers to manage cloud resources.Salesforce.com, Oracle, and Zoho are given as examples.Source: §2 · p. 2
- Cloud services combine with big-data storage and processing offerings to let small companies and individual data scientists analyze terabytes on demand.Examples include cloud storage services and Amazon EMR, which supports large-data analysis without requiring users to manage the challenging operational aspects of analytics infrastructure.Source: §2 · p. 2
3. BIG DATA SCIENCE FRAMEWORKS
Big-data science tools address the need to analyze massive datasets and span cloud services, programming environments, analytics systems, and machine-learning frameworks. The section surveys these approaches and highlights how they support scalable analysis and model development.
3. BIG DATA SCIENCE FRAMEWORKS
- Big-data science frameworks are needed to analyze massive amounts of data and extract information, insights, patterns, and knowledge.The section surveys cloud services, programming tools, machine-learning libraries, and declarative frameworks as parts of this landscape.Source: §3 · p. 3
- R provides a flexible analytics language and extensive libraries for data preparation, statistical analysis, machine learning, and visualization, with connections to systems such as Python.Most R packages were designed for in-memory, interactive use, limiting their fit to data that exceed available memory; distributed systems such as RHIPE, Segue, and SparkR address large-scale execution.Source: §3 · p. 3
- H2O, Mahout, and Spark MLlib provide parallel or distributed machine-learning libraries for tasks including classification, regression, clustering, and collaborative filtering.MLlib is described as suited to iterative learning and offers a high-level API for end-to-end pipelines.Source: §3 · p. 3
- Declarative frameworks such as Samsara, MLbase, and SystemML let users express machine-learning or linear-algebra work at a higher level and compile or optimize execution plans.MLbase also selects and dynamically adapts learning algorithms, while SystemML generates plans for single-node and distributed operations.Source: §3 · pp. 3–4
- AzureML provides a managed, web-based environment for building and automating analytics workflows, including scalable machine-learning methods and custom scripts.Cumulon instead supports matrix-based cloud programs organized as jobs and independent tasks on Hadoop.Source: §3 · p. 4
- Google and Amazon offer SaaS machine-learning services with pretrained or user-created models and API-based prediction, while Watson Analytics accepts natural-language analytics tasks.The services integrate with their respective cloud storage and data systems; Watson also suggests data-cleaning and visualization choices.Source: §3 · p. 4
- BigML presents predictive models as interactive decision trees and also offers a PaaS prediction service; SaaS tools ideally scale resources as analytical workloads grow.The passage also names Kognitio and Hunk among SaaS frameworks.Source: §3 · p. 4
- KeystoneML optimizes machine-learning pipelines at both operator and end-to-end levels, while TensorFlow maps dataflow computations across mobile and large distributed platforms.F2 separates data management from execution and makes runtime partitioning and processing decisions, allowing operations to run on available data.Source: §3 · pp. 4–5
4. DISCUSSION AND OPEN CHALLENGES
The discussion identifies data sharing, interoperability, distributed execution, and model development as open challenges for Big Data Science. It argues that addressing them matters because data-driven applications depend on large, diverse data and scalable analysis.
4. DISCUSSION AND OPEN CHALLENGES
- Big Data Science is increasingly used across business and research, where cloud data storage and elastic computing resources can support faster and more scalable data-science tasks.The section links these applications to the need for effective analysis and exploitation of massive, complex data.Source: §4 · p. 5
- Data-driven applications depend on large, diverse datasets, but data are often siloed across people and organizations, making collaboration and sharing an open challenge.Open-data initiatives and cloud-based data markets are described as approaches to making data available and encouraging innovation.Source: §4 · p. 5
- Existing Hadoop- and Spark-style platforms assume single-cluster management and homogeneous connectivity, making them suboptimal or infeasible for analytics across distributed datasets.Interoperability also requires standard formats and models; some scenarios need local analysis combined with model exchange and fusion.Source: §4 · p. 5
- Most statistical and scientific-computing tools are memory-bounded, risking scalability or costly cloud memory use when data do not fit in main memory.The section calls for efficient distributed and parallel disk-based execution for complex analysis jobs.Source: §4 · p. 6
- Finding and training an adequate predictive model is difficult and time-consuming, with no single algorithm suited to every dataset and changing data.Existing automation efforts are described as focused mainly on single-node implementations and treating model training as a black box, limiting usefulness for large datasets; the section calls for better lifecycle and pipeline management.Source: §4 · p. 6