PaperlayerView exact revision on arXiv

Source-linked AI summary

Big Data For Development: Applications and Techniques

Anwaar Ali, Junaid Qadir, Raihan ur Rasool, Arjuna Sathiaseelan, Andrej Zwitter

arXiv:1602.07810v1cs.CY

TL;DR

BD4D is a nascent field, and harmonizing diverse data for effective analytics remains challenging. This paper reviews BD4D research and related materials, identifying its potential across development settings and the challenges that must be solved for success.

  • Problem

    BD4D is a nascent research field, and harmonizing unstructured data across formats remains challenging for effective analytics.

    Source: §I · p. 2, A. Technical Challenges · p. 11

  • Method

    The paper reviews literature, official documents, online projects, blogs, and technical reports related directly or indirectly to BD4D from a multidisciplinary perspective.

    Source: §I · p. 2

  • Results

    The review identifies BD4D potential across humanitarian emergencies, agriculture, poverty alleviation, food production, healthcare, and education, while highlighting associated challenges and pitfalls.

    Source: §VI · p. 12

  • Takeaways & Limitations

    For BD4D to succeed in human development, researchers must tackle and solve the identified challenges.

    Source: §VI · p. 12

    Policy analysis using big data faces privacy concerns, altered data context or semantics, and spatial and temporal constraints that can bias statistics and yield inefficient policies.

    Source: A. Technical Challenges · p. 11

Abstract

from arXiv · show

With the explosion of social media sites and proliferation of digital computing devices and Internet access, massive amounts of public data is being generated on a daily basis. Efficient techniques/ algorithms to analyze this massive amount of data can provide near real-time information about emerging trends and provide early warning in case of an imminent emergency (such as the outbreak of a viral disease). In addition, careful mining of these data can reveal many useful indicators of socioeconomic and political events, which can help in establishing effective public policies. The focus of this study is to review the application of big data analytics for the purpose of human development. The emerging ability to use big data techniques for development (BD4D) promises to revolutionalize healthcare, education, and agriculture; facilitate the alleviation of poverty; and help to deal with humanitarian crises and violent conflicts. Besides all the benefits, the large-scale deployment of BD4D is beset with several challenges due to the massive size, fast-changing and diverse nature of big data. The most pressing concerns relate to efficient data acquisition and sharing, establishing of context (e.g., geolocation and time) and veracity of a dataset, and ensuring appropriate privacy. In this study, we provide a review of existing BD4D work to study the impact of big data on the development of society. In addition to reviewing the important works, we also highlight important challenges and open issues.

I. INTRODUCTION

The paper reviews how big data can support human development, covering applications, analytics techniques, challenges, and open issues. It frames this work around the promise of BD4D and the difficulties of accessing and responsibly using data.

I. INTRODUCTION

  • Advances in distributed big-data processing and storage, together with AI and machine-learning analytics, can extract actionable knowledge from massive, diverse, unstructured data,.The introduction situates this opportunity amid exponentially growing data generation, including data from IoT sensors and wearable devices,.Source: §I · p. 1
  • Open data can ease BD4D's access problem by making public and private sector data searchable and machine-readable, while open-source tools such as Open Data Kit support mobile data collection.The Rio Operation Center illustrates applications in transport management, disaster relief, mass movement, and slum management,; a McKinsey report estimated open data's net worth at $3 trillion and identified seven sectors.Source: §I · p. 1
  • The UN Global Pulse initiative, launched in 2009, aims to use big data for human development through innovation centers and collaboration, including “data philanthropy” to anticipate humanitarian crises,.Mobile-phone and social-media data are discussed as potential tools against hunger, disaster, and poverty, alongside concerns about data access, privacy, and integration into UN systems.Source: §I · pp. 1–2
  • The paper reviews BD4D from a multidisciplinary perspective, identifying development opportunities, analytics techniques, challenges, potential harms, and directions for future work.It draws on research literature, official documents, projects, blogs, and technical reports, and asks how data can be accessed and used for development, which areas may benefit, and which analytics techniques apply.Source: §I · p. 2

II. BACKGROUND: BIG DATA TECHNIQUES · A. Machine Learning · 1) Supervised Learning:

This background introduces big-data analytics and machine learning as tools for turning rapidly generated, varied data into actionable information, then outlines supervised learning and several classification techniques.

II. BACKGROUND: BIG DATA TECHNIQUES

  • Big data’s volume, velocity, and variety motivate techniques for gathering, storing, processing, and analyzing data in support of humanitarian development.The section aims to provide background on relevant techniques and examples for understanding their development applications.Source: §II · p. 2

A. Machine Learning

  • Machine learning enables computational systems to learn task performance from data and underpins data mining and big-data applications across diverse fields.Applications include decision making, forecasting, and prediction in healthcare, science, engineering, business, and finance.Source: A. Machine Learning · p. 2

1) Supervised Learning:

  • Supervised learning generalizes from labeled examples to predict unseen cases, with continuous outputs treated as regression and discrete outputs as classification.Source: 1) Supervised Learning · p. 2
  • Naive Bayes classifiers assume feature independence given a class under Bayes’ Theorem and have been used for Internet traffic classification.Source: 1) Supervised Learning · p. 3
  • Decision trees predict quantitative or nominal target attributes and offer intuitive interpretations, though they do not always perform competitively.Interpretability is useful when network operators need to analyze and understand classification methods and results.Source: 1) Supervised Learning · p. 3
  • Support vector machines are practical and theoretically grounded supervised-learning methods whose training has a unique solution through concave-function optimization.Their approach is rooted in statistical learning theory.Source: 1) Supervised Learning · p. 3

2) Unsupervised Learning Techniques: … 6) Numeric Prediction:

These sections describe learning techniques for discovering structure, optimizing action, processing patterns, finding variable associations, and predicting numeric quantities. They also note applications of deep learning and association learning relevant to development and agriculture.

2) Unsupervised Learning Techniques:

3) Reinforcement Learning:

  • Reinforcement learning seeks an action-to-reward policy that maximizes average long-term reward through probabilistic rewards or punishments.Source: 3) Reinforcement Learning · p. 3

4) Deep Learning:

  • Deep learning uses complex architectures with multiple nonlinear processing layers composed of parallel processors called neurons,.Source: 4) Deep Learning · p. 3
  • Deep learning supports pattern recognition, image and natural-language processing, and applications from healthcare to fashion,.Source: 4) Deep Learning · p. 3

5) Association Rule Learning:

  • Association rule learning discovers relations among database variables; a weather-data example links sunny outlook and a game being played to a non-windy day.Such associations may help farmers plan activities for better crop production.Source: 5) Association Rule Learning · p. 3

6) Numeric Prediction:

  • Numeric prediction estimates a quantity rather than a discrete class, such as game duration or how long or how much rain will fall.Source: 6) Numeric Prediction · p. 3

B. Data Mining, Knowledge Discovery, and Data Science … D. Internet of Things

The sections introduce data-mining and knowledge-discovery concepts, then survey technologies and approaches for storing, predicting from, gathering, and connecting data. They describe potential development uses alongside challenges such as prediction uncertainty, NoSQL databases’ lack of inherent ACID support, and IoT interoperability.

B. Data Mining, Knowledge Discovery, and Data Science

  • Data mining discovers patterns or predicts from large datasets; knowledge discovery (KDD) extracts useful knowledge, while data science studies KDD techniques and processes.KDD outputs may include visual reports, predicted values, or models of data-generating systems.Source: B. Data Mining, Knowledge Discovery,… · p. 3

1) New Trend in Database Technology: NoSQL:

  • NoSQL databases store and access unstructured data and are distributed, scalable, fast, and flexible, but generally lack inherent ACID support and require manual implementation.Examples include Amazon Dynamo and Google Bigtable.Source: 1) New Trend in Database Technology:… · p. 4

2) Predictive Analytics:

  • Predictive analytics uses past data, data mining, and machine learning to make empirical predictions that can improve decisions probabilistically, though unexpected events may undermine them.The passage notes the philosophical problem of induction and cautions about “black swan” events.Source: 2) Predictive Analytics · p. 4

C. Crowdsourcing and Big Data

  • Crowdsourcing solicits work from the general public through an open call, rather than assigning it to a designated professional or organization.It can gather text messages, social-media updates, and blog posts for mapping disaster-struck regions and supporting search operations.Source: C. Crowdsourcing and Big Data · p. 4
  • Crowdsourced data helped during the 2010 Haiti earthquake, while social-media crowdsourcing offers disaster-relief opportunities and poses challenges.Source: C. Crowdsourcing and Big Data · p. 4

D. Internet of Things

  • 3$–11$ trillion USD is the potential economic value attributed to IoT if all challenges are overcome, according to a McKinsey Global Institute report.IoT connects sensors and actuators through networks to computing systems, producing data for actionable knowledge.Source: D. Internet of Things · p. 4
  • IoT interoperability could enable healthcare monitoring with wearable and hospital data, while real-time sensor data and predictive analytics could reduce industrial breakdowns and system downtime.Source: D. Internet of Things · p. 4

III. BIG DATA FOR DEVELOPMENT: DEVELOPMENT AREAS · A. Humanitarian Emergencies · 1) Natural Disasters:

The section surveys how big data for development can support humanitarian emergencies and other development areas, while outlining data-related challenges. Its natural-disaster discussion highlights crowdsourced crisis information and the need to verify its credibility.

III. BIG DATA FOR DEVELOPMENT: DEVELOPMENT AREAS

  • The development-area overview covers natural disasters and political crises, as well as agriculture, healthcare, education, and poverty and hunger alleviation.Source: §III · p. 4
  • Table I provides a tabulated summary of projects across development areas and tools for big data analytics.Source: §III · p. 4, Table I · p. 5

A. Humanitarian Emergencies

  • The humanitarian-emergencies section presents case studies of natural disasters and political crises and considers data acquisition, storage, and sharing during emergencies.Source: A. Humanitarian Emergencies · p. 4

1) Natural Disasters:

  • After Haiti’s 2010 earthquake, online crowdsourcing and AI/ML analysis of tweets and SMS helped create a real-time crisis map, as described by Meier.Meier calls the community members who contributed digital humanitarians.Source: 1) Natural Disasters · p. 5
  • Meier describes big crisis data as data generated during crises and disasters, from which machine-learning analysis can extract useful, actionable information.The section presents big crisis data as not inherently different from other big data, but distinguished by its crisis context.Source: 1) Natural Disasters · p. 5
  • The natural-disaster discussion warns that fake datasets or trends can be generated easily, making data credibility a key concern; AI/ML techniques are proposed for verification.The concern is linked to ubiquitous online connectivity and the spread of digital communication devices.Source: 1) Natural Disasters · pp. 5–6

2) Migrant Crisis: · B. Hunger, Food and Agriculture · C. Healthcare

The sections describe big-data applications for humanitarian response, agriculture, and healthcare, while highlighting the need for timely, coordinated data use and integration to support better-informed action.

2) Migrant Crisis:

  • Humanitarian organizations must deliver the right assistance to the right regions on time and coordinate across organizations; data is vital to both challenges.The Syrian unrest displaced people internally and abroad, creating a mass movement that challenged aid operations in war-torn areas.Source: 2) Migrant Crisis · p. 6
  • The Services Advisor map gives refugees real-time information on nearby organizations, services, operating times, and capacity, while ActivityInfo enables agencies to crowdsource data for coordination,,,,.The interactive map is accessible through desktop and mobile browsers and is intended to help refugees obtain assistance quickly and avoid queues.Source: 2) Migrant Crisis · p. 6
  • Big-data projects often respond only after crises take a heavy toll; predictive analytics using social and political indicators could instead help prevent emergencies, including the Syrian migrant crisis.The passage calls for models that use data from projects already started during the crisis.Source: 2) Migrant Crisis · p. 6

B. Hunger, Food and Agriculture

  • Skills, funding, and cultural or industrial norms contribute to uneven technology diffusion; Kshetri surveys these enablers and inhibitors and presents an agriculture case study.The survey highlights modern data sources such as social media and cell-phone data.Source: B. Hunger, Food and Agriculture · p. 6
  • Farmers in developing countries often lack information on soil, weather, planting, topography, and markets,,; sensor, satellite, and expert data can feed models whose findings reach them by cellular network.The proposed approach sends relevant information to individual farmers based on predictive models.Source: B. Hunger, Food and Agriculture · p. 6

C. Healthcare

  • Healthcare analytics can combine medical records and wearable data for early diagnosis and personalized app-based care; Google Flu Trends algorithms support epidemic warnings,, and genome sequencing data helps explain disease origins and dynamics.The cited study compares algorithms deployed on Google Flu Trends to detect early warning signs of potential epidemic outbreaks.Source: C. Healthcare · pp. 6–7
  • Fragmented patient records hinder a complete care picture, but integrating them could democratize health information and enable a countrywide Learning Health System,,.The passage continues by describing a system joining policymaking, medical care, engineering, and technology.Source: C. Healthcare · pp. 6–7
  • A worldwide Learning Health System could provide country health-status information and early warning of viral epidemics, then guide region-specific assistance, vaccines, and prevention.It would apply research and analysis to patients in near real time by combining policymaking, medical care, engineering, and technology.Source: C. Healthcare · p. 7

D. Education · IV. BIG DATA ANALYTICS FOR DEVELOPMENT

The education section describes learning analytics and educational data mining as ways to use digital learning data to understand student needs and improve teaching. It highlights individualized support for struggling students and data-informed teacher interventions.

D. Education

  • Education is shifting toward digital study materials and online or blended learning, which combines online teaching, learning, and assessment with conventional pedagogy.Source: D. Education · p. 7
  • Learning analytics combines data analysis and learning to support prediction, social-network and sentiment analysis, personalized learning, and curriculum design; educational data mining applies data-mining and machine-learning techniques to student-system interactions.These related fields use data from digital and online education to inform learning and teaching.Source: D. Education · p. 7
  • Student interactions in MOOCs and online tests can be mined to reveal insights that help optimize the learning experience; digital activity also leaves data trails that can be used to understand behavior.Source: D. Education · p. 7
  • Vendors offer user-friendly, data-driven products to connect research with educational practice, while the International Educational Data Mining Society provides a platform for developing and publishing learning and teaching techniques,.Source: D. Education · p. 7
  • Assessment products collect students’ answers, response times, and link clicks, then present visual reports that help teachers identify topic-specific difficulties and provide individualized support.Source: D. Education · p. 7
  • Comparing data across teachers can reveal which teachers work better with particular student types, enabling timely intervention when, for example, shy students struggle with one teacher but do better with another.Source: D. Education · p. 7
  • Three vendor projects illustrating cross-teacher data analysis are cited as references,, and.Source: D. Education · p. 7

A. Mobile Analytics · B. Living Analytics

Mobile-phone data can support social-network analysis, population planning, and epidemic response, while living analytics uses digital traces to study individual and collective behavior. Both sections emphasize that realizing these applications requires attention to privacy and secure data practices.

A. Mobile Analytics

  • Mobile analytics mines call volume, calling patterns, and location for development insights, but its use raises privacy and ethical concerns.Nokia’s Mobile Data Challenge used multimodal data volunteered by about 200 Swiss smartphone users, with anonymization, secure storage, and agreements to protect privacy and enable research access.Source: A. Mobile Analytics · pp. 7–8
  • Spatiotemporal cell-phone analysis used percolation theory to map anomalous behavior’s spread and decay, offering insight into emergencies and population mobility.The analysis could inform responses to political turmoil or epidemics and transportation planning.Source: A. Mobile Analytics · p. 8
  • Removing strong ties minimally affected a mobile-call-derived social network, while removing weak links collapsed it; intermediate ties were most useful for spreading information.The study proposed targeting intermediate-strength ties for campaigns such as medical awareness.Source: A. Mobile Analytics · p. 8
  • AlertImpact analysis of anonymized CDRs from Mexico’s H1N1 outbreak documented a 10% reduction in peak individual infections due to government mandates.Other mobile-analytics applications include CenCell census maps from anonymized CDRs and commuting matrices from mobility patterns.Source: A. Mobile Analytics · p. 8

B. Living Analytics

  • Living analytics combines computing, network, social, and statistical methods to study behavior, using digital-device data to address biases and gaps in surveys.GPS locations and email logs can provide dynamic visibility into behavior at individual and societal levels.Source: B. Living Analytics · p. 8
  • Computational social science requires secure, channelized data access and centrally supervised storage rather than distribution across users with varying security practices.The passage identifies these as approach and infrastructure barriers to the field’s development.Source: B. Living Analytics · p. 9
  • MIT reality mining used nine months of phone, location, and proximity data from 100 participants to model individual behavior and collective networks.Its findings were presented as useful for anticipating behavior and planning during crises.Source: B. Living Analytics · p. 9
  • Human-centered interfaces should collect behavior data while protecting privacy and comfort, using natural signals such as voice, facial expressions, or physiological conditions.The reviewed questions concern what data is collected, how and why it is collected, its time and location context, and how to respond appropriately.Source: B. Living Analytics · p. 9

C. Visual Analytics · 1) Data maps:

Visual analytics uses interactive visualization to help people reason through large datasets, while data maps combine spatial and statistical information to support analysis of regional patterns.

C. Visual Analytics

  • Interactive visual interfaces support analytical reasoning and can convey the essence of massive datasets quickly.Unlike automatic data-mining and machine-learning tools, this approach can help users explore data without requiring as much technical knowledge.Source: C. Visual Analytics · p. 9
  • Visual analytics can begin with little prior knowledge of the data or goals, which users refine through interaction.Information visualization can also show large amounts of quantitative data in limited space.Source: C. Visual Analytics · p. 9
  • Animated timelines of social-media trends can help track a viral outbreak’s origin and spread hubs, which network science can analyze or help prevent.The passage gives Twitter and Facebook as examples of social-media sites where viral trends may be observed.Source: C. Visual Analytics · p. 10
  • The section introduces four visualization approaches: data maps, time-series, space-time narratives, and relational graphics.Source: C. Visual Analytics · p. 10

1) Data maps:

  • Data maps combine cartography with statistics by displaying a selected variable across the spatial dimensions of a region.For example, a US map can use county colors to show cancer death rates.Source: 1) Data maps · p. 10
  • Adding county socioeconomic information to a data map can help investigate disease patterns and support healthcare and epidemiology efforts to prevent viral outbreaks.The map’s color scheme lets users locate counties with higher or lower cancer death rates.Source: 1) Data maps · p. 10

2) Time series: … A. Technical Challenges

Time-series and related graphics reveal temporal trends and spatial or relational patterns, while the paper identifies technical and ethical challenges to large-scale BD4D deployment.

2) Time series:

  • Time-series analysis reveals trends over specified periods and can give disaster managers advance information for preventive action.Time scales may range from seconds to centuries; the passage illustrates the use of time patterns to anticipate periods of dengue mosquito activity.Source: 2) Time series · p. 10

3) Space-time narrative:

  • Space-time graphics combine time and location to show where and when conditions such as pollutant levels peak.The passage illustrates this with carbon monoxide peaks during rush hours near large-road intersections.Source: 3) Space-time narrative · p. 10

4) Relational graphics:

  • Relational graphics can add time and other variables to reveal changing patterns across regions and help target resistant areas.The example tracks deaths per million against cigarette consumption to assess changes associated with an anti-smoking campaign.Source: 4) Relational graphics · p. 10

V. CHALLENGES, OPEN ISSUES AND FUTURE WORK; A. Technical Challenges

  • Big data is not a panacea, and technical and ethical challenges require further work; cloud computing and SDN help, but facilities must scale with data growth.The paper identifies processing and storage capacity scaling with daily data production as a technical challenge.Source: §V · p. 10, A. Technical Challenges · pp. 10–11

A. Technical Challenges

  • Without coordination among aid agencies using crowdsourced social-media data, duplicate responses can leave other problems or complementary actions unattended.Crisis locations inferred from users’ posts may also be inaccurate, so the passage calls for corroboration with ground surveys and aerial imagery.Source: A. Technical Challenges · p. 11
  • Personalized algorithms can polarize search results, while fraudulent information can spread through connected networks and trigger cascading failures.The passage notes deep-learning and context-aware computing as possible responses to polarization.Source: A. Technical Challenges · p. 11
  • Privacy, altered context or semantics, and uneven technology access can bias data and lead to faulty or inefficient policies.For student learning analytics, technology-use differences also require care so students who use the Internet less or differently are not penalized.Source: A. Technical Challenges · p. 11
  • Harmonizing varied data formats is difficult, while fragmentation can prevent collective analysis that could speed diagnosis and enable personalized treatment.The passages describe combining call-detail records, satellite imagery, and surveys, and integrating clinical test results from different specialists.Source: A. Technical Challenges · p. 11

B. Ethical

The section argues that BD4D deployment requires attention to ethical harms as well as technology challenges, with privacy and fragmentation among the major barriers and additional concerns involving digital divides, data governance, and expertise.

B. Ethical

  • Privacy and fragmentation are major impediments to large-scale BD4D deployment, alongside subtler ethical risks from technology abuse.Collected personal data must not be abused for personal or financial gain.Source: B. Ethical · pp. 11–12
  • The digital divide harms nations lacking infrastructure, economic affordability, and data-savvy faculty, while data control and monopolies hinder wide-scale deployment.Source: B. Ethical · p. 12
  • Open data governance must balance transparency with access and reuse rules, while protecting sensitive humanitarian locations from malicious exploitation.The passage cites the locations of humanitarian actors and internally displaced persons as sensitive information that should not become open.Source: B. Ethical · p. 12
  • Ethical BD4D analysis requires interdisciplinary expertise and collaboration, which the passage says should be encouraged and financially incentivized.The field's evolution itself poses a challenge because it draws on expertise from multiple disciplines.Source: B. Ethical · p. 12

VI. CONCLUSIONS

The paper reviews big data for development (BD4D), highlighting its potential across human-development settings and the challenges that must be solved for it to contribute to human development and global prosperity.

VI. CONCLUSIONS

  • The review highlights BD4D’s potential in humanitarian emergencies, agriculture, poverty alleviation, food production, healthcare, and education.It aims to make this potential accessible to a broad audience.Source: §VI · p. 12
  • The paper identifies challenges and pitfalls associated with BD4D as obstacles that researchers need to tackle and solve.Source: §VI · p. 12
  • The authors envision BD4D playing a major role in human development and global prosperity in the future, conditional on solving those challenges.Source: §VI · p. 12

Related papers

Loading 1602.07810v1…