Big Data: Challenges, Opportunities and Realities

INTRODUCTION

• Digitization, sensors, social platforms, and connected devices generate data continuously, with types and volumes surpassing existing storage capabilities.The chapter notes that these data may carry more information than before.Source: INTRODUCTION · p. 2
• Growing volumes of unstructured data, such as reviews and tweets, make it almost impossible to process them with existing storage techniques and plain queries.The chapter frames big data analytics and new storage and processing approaches as responses to these limitations.Source: INTRODUCTION · p. 3

BACKGROUND AND NEED FOR BIG DATA ANALYTICS

• Big data concerns storing, processing, and comprehending structured and unstructured data for prediction with acceptable time delay.The chapter links this challenge to the need to retrieve large volumes of data within a desirable time lag.Source: BACKGROUND AND NEED FOR BIG DATA… · p. 3

Grid Computing

• Grid computing distributes problems across networked computers that share computing and memory resources, harnessing otherwise unused capacity.The chapter describes commercial and social models, while noting financial, social, legal, regulatory, and ownership concerns.Source: Grid Computing · p. 4

Cloud Computing

• Cloud computing places processing, memory, and storage on a service provider's infrastructure, accessed remotely through a thin client.It offers dynamic scaling and pay-as-you-go use, but shares grid computing drawbacks and depends on Internet access.Source: Cloud Computing · pp. 4–5

Dimensions of Big Data

• The 3Vs characterize big data through Volume, Velocity, and Variety, with volume and classification depending on data type and time.The chapter describes velocity as a generation rate that can require real- or near-real-time processing, and variety as including structured, semi-structured, and unstructured data.Source: Definition · p. 6, Dimensions of Big Data · pp. 6–7

Definition

• Big data has no specific definition, and what counts as big can change as storage technologies advance.The chapter says that beyond defining big data, it is necessary to understand how to use it for decision-making.Source: Definition · p. 6

Dimensions of Big Data

• Additional dimensions include Veracity, Variability, Complexity, and Low-Value density, reflecting unreliability, changing flow rates, diverse semantics, and limited value in raw data.The chapter says raw data must be analyzed to discover value, such as predicting customer behavior from website logs.Source: Dimensions of Big Data · p. 7

Sources of Big Data

• Digitized content and improved instrumentation generate growing datasets, including astronomy surveys and frequent smart-meter readings.The Sloan Digital Sky Survey collected 25 TB across its first two surveys and 100 TB during its third; smart meters record use every 15 minutes rather than monthly.Source: Sources of Big Data · p. 7
• IoT sensors produce data in sectors including agriculture, industry, medical care, and smart cities.Examples across the listed sectors include electronic medical records, gene sequences, sensor readings, and fleet or smart-meter data.Source: Sources of Big Data · p. 8
• The chapter also identifies social media, financial news, astronomy observations, and telecommunications records as sector-specific data sources.The listed uses include trading decisions, monitoring astronomical bodies, customer-behavior analysis, and churn management.Source: Sources of Big Data · p. 8

BIG DATA VALUE CHAIN

• The Big Data Value Chain comprises seven phases, from data generation and collection through transmission, preprocessing, storage, analysis, and decision-making.The framework adapts the idea of a value chain to activities that create value from available data.Source: BIG DATA VALUE CHAIN · p. 8
• Data collection gathers inputs from possible sources, while transmission moves collected data to storage and processing infrastructure.Collection methods include logs, sensors, web crawlers, and network-monitoring software; transmission includes inter- and intra-data-center movement.Source: BIG DATA VALUE CHAIN · p. 9
• Preprocessing integrates data from multiple sources and addresses redundancy, noise, and inconsistency to improve data quality for analysis.Integration can use federation, warehousing, or virtual databases; cleaning checks accuracy, completeness, and consistency.Source: BIG DATA VALUE CHAIN · pp. 9–10
• Data-reduction methods such as filtering and compression can reduce storage costs and inconsistency, but increase computational cost.The chapter recommends a cost-benefit analysis before using these techniques.Source: BIG DATA VALUE CHAIN · p. 10
• Distributed storage must balance consistency, availability, and partition tolerance, since the chapter says all three cannot be met simultaneously.It lists GFS, Dynamo, BigTable, Cassandra, CouchDB, and Dryad among storage mechanisms.Source: BIG DATA VALUE CHAIN · p. 10
• Analysis planning selects metrics, architecture, algorithms, and tools, matching real-time or offline processing to the problem's timing needs.Real-time examples include retail and telecom fraud; Hadoop is identified as a commonly used architecture for offline analysis, with visualization supporting pattern identification.Source: BIG DATA VALUE CHAIN · pp. 10–11

• Analyzing large volumes and varied data economically requires computing architecture that supports storage and heavy server processing.The section surveys technologies adopted for big-data analysis.Source: TOOLS FOR COLLECTING, PREPROCESSING… · p. 11
• Data are increasingly explored with intelligent systems to derive insights that may not have been envisioned when the data were collected.The chapter describes systems that initiate and refine questions without expert intervention as capable of discovering new information.Source: TOOLS FOR COLLECTING, PREPROCESSING… · p. 12
• Big data analytics spans applications including astronomy, healthcare, and telecommunications, while raising security and privacy concerns.The chapter connects these advances and limitations to data science, an interdisciplinary field drawing on psychology, statistics, economics, social science, network science, and computer science.Source: TOOLS FOR COLLECTING, PREPROCESSING… · p. 12
• Digital devices and applications generate logs, text, voice, images, and video, prompting techniques for representing unstructured data and deriving insights.Source: TOOLS FOR COLLECTING, PREPROCESSING… · p. 12

Not only Structured Query Languages (NoSQL)

• RDBMS stores structured data in relational databases and schemas, commonly using SQL; warehouses organize data through dimensional or normalized approaches.The dimensional approach uses fact and dimension tables, while the normalized approach divides data into entities across tables.Source: Not only Structured Query Languages… · p. 13
• RDBMS limitations in scaling large data volumes and handling semi-structured or unstructured data led to the concept of NoSQL.Source: Not only Structured Query Languages… · p. 13
• NoSQL databases manage unstructured data with flexible schemas, support several store types, and allow data scalability.They use a looser consistency model than traditional databases and separate data management from storage; examples include HBase, MangoDB, and Dynamo.Source: Not only Structured Query Languages… · p. 13

Hadoop Distributed File System (HDFS)

• HDFS is a fault-tolerant, scalable distributed storage system that divides Hadoop-cluster data into pieces stored across servers.Each server holds a small chunk of the overall dataset.Source: Hadoop Distributed File System (HDFS) · p. 13

Hadoop MapReduce

• MapReduce divides workloads into smaller units, maps data blocks to sorted key-value pairs, then reduces and merges the results for output to HDFS.The mapper output is shuffled to the reduce phase.Source: Hadoop MapReduce · p. 13

Limitations of Hadoop

• Hadoop’s listed limitations include at least three HDFS data copies, limited SQL support, inefficient execution, difficulty expressing complex transformations, and required distributed-development skills.Source: Limitations of Hadoop · p. 15

SOFTWARE TOOLS FOR HANDLING BIG DATA

• Big-data tools include R for statistical and graphical analysis and Python libraries for preprocessing, computing, modeling, and machine learning.R faces memory-management, speed, and efficiency challenges; Python’s NumPy supports multidimensional arrays, while Scikit provides modeling and preprocessing algorithms.Source: SOFTWARE TOOLS FOR HANDLING BIG DATA · p. 15
• Scala is the language used to implement Spark, an in-memory cluster-computing framework with APIs for Java, Scala, Python, and R.Spark supports SQL, streaming, graph processing, machine learning, and MapReduce; it uses Hadoop for storage and distributes data across machines through RDDs.Source: SOFTWARE TOOLS FOR HANDLING BIG DATA · p. 16
• Hive provides SQL-like HiveQL for querying and managing datasets in distributed storage, while Pig uses Pig Latin to create MapReduce programs.Hive permits custom mapper and reducer code; Pig supports user-defined functions and substantial parallelization.Source: SOFTWARE TOOLS FOR HANDLING BIG DATA · p. 16
• The chapter also lists MongoDB, BlinkDB, Tachyon, Cassandra, CouchDB, Clojure, Tableau, and Splunk among other big-data frameworks and tools.Amazon EC2 provides cloud computing capacity on a pay-as-you-go basis.Source: SOFTWARE TOOLS FOR HANDLING BIG DATA · p. 16

Healthcare

• 60% fewer imaging procedures and 30% fewer hospitalizations followed the Rizzoli Orthopedic Institute’s analysis of patient symptoms and clinical variation.The passage also describes integrating clinical, public-health, and behavioral data as supporting robust treatment systems that can reduce costs and improve treatment quality.Source: Healthcare · p. 18
• After Haiti’s January 2010 earthquake, analysis of tweets helped track cholera spread; sensor data are also monitored for adverse-event prediction and safety monitoring.Source: Healthcare · p. 18
• Artemis analyzes intensive-care sensor data to detect the onset of medical complications, especially in neonatal care.The passage also notes that real-time analysis of many claims requests can minimize fraud.Source: Healthcare · p. 18

Telecommunication

• Telecom providers use customer demographics, preferences, household structure, and usage data to target personalized offers; Ufone used call patterns to customize plans.The passage says targeted marketing can improve mobile-service adoption, reduce churn, and increase revenue.Source: Telecommunication · p. 18
• Telecom providers analyze real-time data to minimize fraud losses, while network analytics can help resolve network problems within minutes and improve service quality.Mobileum was developing a real-time fraud-detection system using predictive analytics and machine learning.Source: Telecommunication · p. 19

Financial Firms

• Capital markets use big data for regulatory preparation, fraud mitigation, and pre-trade analytics, where timely value makes real-time processing important.Examples include sentiment analysis, predictive analytics, and data tagging to identify trades.Source: Financial Firms · p. 19

Retail

• Retailers use market-basket analysis for product placement and pricing, while Stage Stores used analytics to forecast garment orders by size and region.Online retailers also use clickstream behavior for real-time product recommendations.Source: Retail · p. 19

Law Enforcement

• The RTM Dx app measures spatial correlations between crime locations and environmental features for police departments in several states.The passage identifies the app as developed by Rutgers University researchers.Source: Law Enforcement · pp. 19–20

Marketing

• Marketing analytics evaluates performance and consumer behavior to inform ad placement, dynamic pricing, and personalized product offers.Source: Marketing · p. 20

New Product Development

• Analysis of earlier product releases informed Microsoft’s Ribbon UI design, while integrating social-media and CRM data can inform new-product planning.The chapter also states that analysis and planning during development can minimize product risk, increase customer lifetime value, and promote brand engagement.Source: New Product Development · p. 20

Banking

• Banking uses demographic, behavioral, financial, and sentiment data with CRM records to target customer segments for cross-selling.Source: Banking · p. 20

Energy and Utilities

• Smart-meter readings taken hourly can reveal energy-consumption patterns, electricity theft, and price fluctuations through real-time analysis.Source: Energy and Utilities · p. 20

Insurance

• Insurers combine driving patterns with statistical driver-risk information to develop personalized insurance plans.Inputs described include miles and routes driven, time of day, and abrupt braking.Source: Insurance · pp. 20–21

Education

• Computerized course modules enable real-time performance monitoring and feedback, while an intelligent-tutor study found fewer reading mistakes when students reread an old story rather than a new one.Source: Education · p. 21

Other sectors

• Big-data analytics may also benefit construction and material sciences as organizations develop analytics skills.Source: Other sectors · p. 21

TECHNOLOGICAL GROWTH AND TECHNOLOGICAL LIMITATIONS

• Big-data technologies and algorithms have expanded processing options, but storage, timeliness, heterogeneous data, and database performance remain challenges.The chapter calls for systems combining relational and non-relational database benefits to provide flexibility.Source: TECHNOLOGICAL GROWTH AND TECHNOLOGICAL… · pp. 21–22
• Limited data access and unreliable or biased datasets constrain analysis; social-media samples may not represent populations, and network interactions do not necessarily reveal personal ties.The chapter also notes security, privacy, and intellectual-property risks associated with integrated IT architectures and personal information.Source: TECHNOLOGICAL GROWTH AND TECHNOLOGICAL… · pp. 22–23

• Research is needed for storing, retrieving, and processing images, audio, and video, including multidimensional analytics and array-based in-memory representations.The passage also identifies multidimensional extensions to HiveQL as an area to explore.Source: RESEARCH AREAS WITH TECHNOLOGICAL… · p. 23
• Data life-cycle research must address real-time value, filtering, completeness, reliability, sparse distributions, and the risk of biased conclusions from unbalanced datasets.Source: RESEARCH AREAS WITH TECHNOLOGICAL… · p. 23
• Alternative big-data computation paradigms under exploration include YarcData graph analytics and HPCC as a Hadoop alternative.Source: RESEARCH AREAS WITH TECHNOLOGICAL… · p. 23
• The chapter identifies a need for visualization tools for high-dimensional data to support decision analysis throughout analytics.Source: RESEARCH AREAS WITH TECHNOLOGICAL… · p. 24
• General machine-learning algorithms may not meet domain-specific requirements, so discipline-specific algorithms are needed to gain relevant insights.Source: RESEARCH AREAS WITH TECHNOLOGICAL… · p. 24
• Real-time processing algorithms must meet expected delay requirements as data generation accelerates.Source: RESEARCH AREAS WITH TECHNOLOGICAL… · p. 24
• Research into efficient storage devices could reduce reliance on HDFS and its software management layer.The passage notes that current storage-device purchasing and use cannot meet growing demand.Source: RESEARCH AREAS WITH TECHNOLOGICAL… · p. 24
• Big-data results carry social and cultural implications, including a tension between improved services and privacy incursions or surveillance of political activity.Source: RESEARCH AREAS WITH TECHNOLOGICAL… · p. 24

CONCLUSION

• The chapter reviews big-data processing, storage platforms, tools, languages, applications, technological limitations, and future research directions.Source: CONCLUSION · p. 24

KEY DEFINITIONS

• Big Data means structured and unstructured data too large for traditional database techniques, while Big Data Analytics obtains, stores, and analyzes such data for insights.Source: KEY DEFINITIONS · p. 29
• Cloud Computing provides remote Internet-based infrastructure on a pay-as-you-go basis, whereas Grid Computing distributes computing through users’ donated memory and processing resources.Source: KEY DEFINITIONS · p. 29
• The Data Value Chain covers activities from data generation through analysis; MapReduce processes large datasets across Hadoop clusters using the Hadoop distributed file system.Source: KEY DEFINITIONS · p. 29
• Hadoop is a Java-based framework for processing large amounts of data in a distributed computing environment, and high-dimensional data has more variables than samples.Source: KEY DEFINITIONS · p. 29