PaperlayerView exact revision on arXiv

Source-linked AI summary

Data Science: A Comprehensive Overview

Longbing Cao

arXiv:2007.03606v1cs.CY

TL;DR

Big data and advanced analytics create challenges and opportunities that existing disciplines do not fully accommodate, while data science remains an early-stage and debated field. The paper surveys data science across research, innovation, analytics, economy, profession, education, and future directions, presenting a comprehensive field overview. It highlights data science’s connection to research and innovation, the growth of data-driven enterprise, and a shift toward more implicit, actionable analytics.

  • Problem

    New data-intensive challenges and opportunities are not adequately addressed by existing disciplinary knowledge and capabilities, while data science’s scope and role remain unclear.

  • Method

    The paper provides a comprehensive survey and overview of data science’s evolution, concepts, analytics, innovation, economy, profession, education, and future directions.

  • Results

    The review identifies stronger historical and disciplinary foundations for data science, data analysis, and data analytics than for big data, alongside business-led big-data growth and research-linked data science.

  • Takeaways & Limitations

    Data science and analytics are increasingly recognized as driving forces for next-generation innovation, economy, and education.

  • Takeaways & Limitations

    The analytics evolution shown in Figure 4 is not a linear organizational path; teams may iterate back and forth and use multiple components in parallel.

Abstract

from arXiv · show

The twenty-first century has ushered in the age of big data and data economy, in which data DNA, which carries important knowledge, insights and potential, has become an intrinsic constituent of all data-based organisms. An appropriate understanding of data DNA and its organisms relies on the new field of data science and its keystone, analytics. Although it is widely debated whether big data is only hype and buzz, and data science is still in a very early phase, significant challenges and opportunities are emerging or have been inspired by the research, innovation, business, profession, and education of data science. This paper provides a comprehensive survey and tutorial of the fundamental aspects of data science: the evolution from data analysis to data science, the data science concepts, a big picture of the era of data science, the major challenges and directions in data innovation, the nature of data analytics, new industrialization and service opportunities in the data economy, the profession and competency of data education, and the future of data science. This article is the first in the field to draw a comprehensive big picture, in addition to offering rich observations, lessons and thinking about data science and analytics.

1. INTRODUCTION

The rise of big data creates major challenges and opportunities across science, engineering, business, and society, while data science remains an emerging and debated field. This article surveys its evolution, domains, analytics, innovation, economy, profession, education, and future directions.

  • 1. INTRODUCTION: Big data growth creates substantial challenges alongside innovation and economic opportunities, attracting private, governmental, and academic attention.Examples include data-centric projects and strategic initiatives involving Google, Facebook, IBM, the United Nations, the EU, and China.
  • 1. INTRODUCTION: Recognizing big data’s value is reshaping traditional and non-traditional data-oriented scientific and engineering fields.The shift is associated with understanding, exploring, and utilizing data across computing, informatics, statistics, social science, business, and management.
  • 1. INTRODUCTION: Fundamental questions concern what data science is, whether it differs from information science or established disciplines, and how it relates to data-intensive discovery.These questions arise in research and disciplinary development, while data economy and competency receive less attention in the cited conference discussions.
  • 1. INTRODUCTION: Data science addresses new or more complex challenges that existing theories, systems, tools, and capabilities do not adequately accommodate.The field is nevertheless at an early stage, generating hype, bewilderment, and competing interpretations.
  • 1. INTRODUCTION: The article presents a comprehensive survey of data science spanning research, innovation, economy, profession, disciplinary development, and education.It draws on practical data innovation experience, education and training opportunities, and reflection on critical issues, future directions, and strategic opportunities.
  • 1. INTRODUCTION: The review covers the progression from data analysis to data science, the era of data science, deep analytics, data-driven innovation, industrialization, services, professional competency, education, and future directions.The paper also organizes and explains closely connected terms including data analysis, data analytics, advanced analytics, big data, and several analytics types.

2. FROM DATA ANALYSIS TO DATA SCIENCE

The paper traces data science from early data analysis and descriptive analytics through data mining, knowledge discovery, and interdisciplinary synthesis. It also examines changing interest in related terms and defines data science as a field that transforms data and its environments into insights and decisions.

  • From Data Analysis to Data Science: Data analysis developed from empirical science and exploratory processing toward using data to suggest suitable hypotheses for testing.The paper locates this evolution in statistics and mathematics beginning in 1962.
  • From Data Analysis to Data Science: Data mining and knowledge discovery extended data analysis toward discovering hidden and interesting knowledge from data.These developments helped establish data-driven discovery as a central theme across subsequent conferences and disciplines.
  • From Data Analysis to Data Science: Data analytics combines data mining, knowledge discovery, machine learning, original data analysis, and descriptive analytics as a multidisciplinary science.It supports exploratory, predictive, and confirmatory purposes and has expanded into domain-specific forms such as business, risk, social, and web analytics.
  • Online Search Interest Trends: Google search trends indicate that data science, data analysis, and data analytics have richer histories and stronger disciplinary foundations than big data.Search interest in data science and analytics increased consistently in the later period examined, while big data rose rapidly after 2012 and then showed less movement.
  • What Is Data Science: The paper defines data science as an interdisciplinary field synthesizing statistics, informatics, computing, communication, management, and sociology to transform data into insights and decisions.Its scope includes data environments and follows data-to-knowledge-to-wisdom thinking and methodology.
  • What Is Data Science: Data products include discoveries, predictions, services, recommendations, models, tools, and systems, with ultimate value in knowledge, intelligence, wisdom, and decisions.The paper connects these products to data-driven and data-enabled transformations across academic, industrial, governmental, and socioeconomic settings.

3. THE ERA OF DATA SCIENCE

The era of data science is marked by ubiquitous, rapidly expanding quantification, coordinated government initiatives, cross-disciplinary development, and growing educational and economic activity. These developments span datafication, public policy, disciplinary transformation, course formation, and data-driven enterprise innovation.

  • Datafication and Quantification: Datafication and quantification increasingly occur across times, places, bodies, forms, and sources, producing data objects at incremental to exponential speed.Examples include information systems, sensors, IoT, mobile and social applications, wearables, and quantified-self services.
  • Data Initiatives by Governments: Governments are introducing big-data initiatives to support research, innovation, policy, industrialization, and economic development.Examples include Australia’s strategy and coordination center, Canada’s stewardship framework, China’s coordination mechanisms and strategic plans, and the United Nations’ public-good vision.
  • Data Science Disciplinary Development: Data science is reshaping traditional and non-traditional disciplines through cross-disciplinary, data-driven discovery and specialized X-informatics fields.Examples include astroinformatics, bioinformatics, health informatics, medical informatics, and social informatics.
  • Data Science Disciplinary Development: About 500 related courses were identified, with 78% offered in the US, 72% at Master’s level, and only 7% at bachelor level.Approximately 40% focus on business and social science, while 43% encompass “Analytics” and 18% “Data Science”.
  • Data Science Disciplinary Development: More than 85% of courses cover broad big-data, analytics, and data-science topics, while project-management and communication training remains very rare.Many other courses concentrate on specific technical skills and technologies such as machine learning, visualization, cloud computing, and predictive analytics.
  • New Data Economy and Industry Transformation: Data science supports a data product-based, data technology-driven economy in which organizations invest in infrastructure, talent, and teams for innovation and productivity.The data science and analytics professional community is also growing incredibly quickly.

4. DATA ANALYTICS: A KEYSTONE OF DATA SCIENCE

Data analytics spans the data life cycle from historical description to future prediction and prescriptive decision support. Its evolution moves from explicit, hypothesis-driven analysis toward implicit deep analytics that extracts hidden insights and supports higher-value action.

  • 4.1. Data-to-Insight-to-Decision Whole-of-Life Analytics: Analytics covers the whole data-to-decision life cycle, progressing from historical and present-data understanding to prediction and actionable decisions.The stages address what happened, what is happening, what will happen, and what best action to take.
  • 4.2. Explicit-to-Implicit Analytics Evolution: Explicit analytics uses visible, domain-driven, hypothesis-based methods, whereas implicit analytics seeks hidden knowledge, intelligence, and value from data and context.The two dimensions track visibility, automation, complexity, intelligence, and value as analytics becomes more advanced.
  • 4.2. Explicit-to-Implicit Analytics Evolution: Typical analytics components include reporting, statistical analysis, alerting, forecasting, predictive modeling, optimization, prescriptive analytics, and actionable knowledge delivery.These components can be combined across analytics tasks, and an approach may also be used for non-analytical purposes.
  • 4.2. Explicit-to-Implicit Analytics Evolution: The evolution is not a linear organizational path: analytics teams often iterate back and forth and use multiple components in parallel.Figure 4 represents an evolution of the analytics family rather than a prescribed sequence within a specific organization.
  • 4.3. Descriptive-to-Predictive-to-Prescriptive Analytics: The shift reduces routine explicit-analytics effort through automation while increasing effort in implicit analytics and actionable knowledge delivery.Moving to higher stages is associated with greater knowledge, intelligence, and organizational value.

5. DATA INNOVATION: CHALLENGES AND OPPORTUNITIES

Data innovation faces interconnected challenges in understanding complex domain data, building adequate foundations and systems, and supporting decisions, infrastructure, and social responsibilities. The paper presents these issues as a complex-system map requiring systematic and interdisciplinary approaches.

  • 5.1. Data Science Conceptual Map: The paper organizes data-science challenges into five areas spanning domain understanding, mathematical and statistical foundations, X-analytics and engineering, and social issues.Figure 6 presents this conceptual map as a complex system addressing big-data complexities.
  • 5.2. Challenges in Data and Business Understanding: Existing theories and techniques do not adequately represent or manage domain-specific X-complexities and X-intelligence embedded in data and business problems.The challenge includes understanding their forms, levels, extent, interactions, and integration into data-science processes.
  • 5.3. Challenges in Data-to-Decision and Actions: Data-to-decision research must develop theories and systems for insight-to-decision transformation, decision generation, action generation, governance, and management.The paper states that existing technologies cannot manage these requirements.
  • 5.4. X-Analytics and Data/Knowledge Engineering: Big-data systems must address scale, velocity, variety, real-time processing, storage, quality problems, and cross-organizational or cross-cultural data settings.The listed quality issues include noise, uncertainty, missing values, imbalance, and newly emerging multi-context problems.
  • 5.4. X-Analytics and Data/Knowledge Engineering: Deep analytics seeks unknown knowledge and intelligence through new theories and algorithms that combine data-driven and model-based problem solving.Related challenges include simulation, experimental design, high-performance processing, architectures, and distributed team interoperation.
  • 5.5. Interdisciplinary Directions: Addressing these challenges requires synergy across data representation, preprocessing, distributed systems, high-performance computing, management, infrastructure, and related disciplines.The paper explicitly calls for systematic and interdisciplinary methodologies.

6. DATA ECONOMY: DATA INDUSTRIALIZATION AND SERVICES

Data science and analytics are transforming data into products, services, business models, and decision-support capabilities across the economy. The emerging data industry spans design, content, software, infrastructure, services, and education, with global and lifecycle-oriented delivery.

  • 6.1. Data Industry: Data science and big-data analytics create new data products, businesses, services, and industrialization opportunities while contributing to economic innovation, competition, and productivity.The paper describes these opportunities as previously impossible prospects enabled by the data economy.
  • 6.1. Data Industry: The data industry’s six core areas are data and analytics design, content, software, infrastructure, services, and education.Together they define major drivers of new data business and related transformation.
  • 6.1. Data Industry: Data businesses cover product and service design, content distribution, software and platforms, storage and computing infrastructure, consulting and operational services, and professional education.Education addresses corporate competency, training, courses, workshops, and shortages of qualified professionals.
  • 6.2. Data Services: Data services support storage, understanding, processing, optimization, value creation, transport, communication, servicing, and decision support across data-intensive sectors.Examples include finance, government, telecommunications, healthcare, manufacturing, marketing, surveillance, and web analytics.
  • 6.2. Data Services: Actionable knowledge delivery can produce significant savings and efficiency improvements, while data services enable cross-media, cross-source, and cross-organization innovation.The paper distinguishes these services from traditional physical material- or energy-oriented services.
  • 6.3. Global Data Services: Data-driven services increasingly operate online, mobile, socially, globally, and continuously, supporting real-time processing and full descriptive-to-prescriptive decision cycles.The paper presents data-driven production and decision making as core functions for complex organizational decisions and strategic planning.
  • 6.3. Global Data Services: Global data services require shared analytics objectives, governance, security, privacy, accountability, and cross-organizational data matching and sharing.These requirements are highlighted for multinational companies and whole-of-government settings.

7. DATA EDUCATION: CAPABILITIES AND COMPETENCY

Data education responds to gaps in data and analytics capabilities by developing multidisciplinary professionals who can move from business and data understanding to actionable decisions. The section presents data scientists as technically sophisticated, experimental, collaborative practitioners requiring formal training and domain awareness.

  • Capabilities and competency: Data innovation and the data economy depend on stronger analytics capabilities, organizational maturity, education, training, and social-issue management.The required competencies include thinking, managing, computing, mining, communicating, delivering, and acting with data.
  • Education and training: Data science curricula and certification efforts are expanding across general and domain-specific Masters and PhD programs.Examples include a PhD in Analytics and a Master’s degree in supply-chain-management predictive analytics.
  • Capabilities and competency: Their responsibilities span domain understanding, ethical and data assessment, analytical planning, engineering, modeling, deployment, communication, and lifecycle management.The workflow converts business and data into information, insight, and decision-making actions while addressing privacy, security, and veracity.
  • Professional profile: Compared with BI professionals, data scientists are more technically trained, use more sophisticated and diversified toolkits, conduct more experiments, and spend nearly twice as much time manipulating big data.The cited survey reports differences in disciplinary backgrounds, tool sophistication, experimentation, interaction, and big-data manipulation.

8. THE FUTURE OF DATA SCIENCE

The paper frames data science’s future as the construction of scientific, technological, educational, and collaborative foundations for addressing currently invisible data challenges. Its agenda includes autonomous data intelligence, richer representations and infrastructure, new analytical systems, and cross-domain training and cooperation.

  • The future of data science: Data science is expected to develop systematic scientific foundations, disciplinary structures, theoretical systems, technologies, and engineering tools as an independent science.This future extends beyond statistics toward foundational scientific problems and grand challenges.
  • Intelligent data systems: Future research should design data brains that mimic human recognition, understanding, analysis, learning, inference, reasoning, and action.The proposed systems connect environmental and data understanding with decisions and actions.
  • Foundations and challenges: The field should investigate invisible data characteristics, complexities, intelligence, and value, including what remains unknown about the unknown.This agenda is intended to clarify data science’s capabilities, limitations, and future directions.
  • Data infrastructure: New data representations, storage, access, and management mechanisms should preserve richer real-world characteristics while supporting scalable, transparent, flexible, interpretable, personalized, and real-time analytics.The proposed infrastructure spans memory, disk, and cloud-based mechanisms.
  • Analytical capabilities: Future analytical capabilities should create original mathematical, statistical, and analytical theories, algorithms, and models to disclose unknown knowledge.The agenda also includes intelligent systems and collaborative platforms for automated or human-data-cooperative exploration.
  • Education and collaboration: Progress requires training data scientists with data literacy, thinking, competency, communication, curiosity, and cognitive intelligence, alongside cross-domain and trans-disciplinary collaboration.The paper connects this educational agenda to complex data-science problem-solving and emerging data-driven applications and economies.

9. CONCLUSIONS

The conclusion presents data science, big data, and advanced analytics as emerging forces for innovation, economy, and education. It argues that strategic discussion and coordinated efforts across government, industry, academia, and private institutions are important while the field remains young.

  • Conclusions: Data science, big data, and advanced analytics are increasingly recognized as driving forces for next-generation innovation, economy, and education.The paper emphasizes that these fields remain at an early stage of development.
  • Conclusions: Strategic discussion of the field’s big picture, trends, challenges, future directions, and prospects is presented as critical for healthy development.The article shares an overview of data science’s conceptualization, development, and observations.
  • Conclusions: The evolving data world connects daily life, work, learning, economy, and entertainment, while institutions increasingly seek to convert data into decision-making.Government, industry, academia, and private institutions are promoting data-science and analytics research and development.
Loading 2007.03606v1…