Source-linked AI summary
Crisis Analytics: Big Data Driven Crisis Response
Junaid Qadir, Anwaar Ali, Raihan ur Rasool, Andrej Zwitter, Arjuna Sathiaseelan, Jon Crowcroft
TL;DR
Disaster response increasingly depends on making sense of large, fast-changing crisis datasets generated by affected communities and other sources. The article surveys big crisis data analytics, including mobile, open, crowdsourced, machine-learning, and visual approaches, while examining their promise and challenges. It presents these tools as useful for improving response but emphasizes that noise, coordination requirements, and the limits of technology constrain their role.
Problem
Disaster responders need to extract timely, useful information from large amounts of crisis-related data to understand fast-changing situations.
Method
The article surveys the history, technologies, applications, human-machine approaches, and challenges of big crisis data analytics.
Results
Big crisis data analytics can support humanitarian response through mobile data, open mapping, crowdsourcing, machine learning, and visual analytics.
Takeaways & Limitations
Big crisis data analytics is one component of a broader crisis-informatics ecosystem and cannot work in isolation or serve as a technological panacea.
Takeaways & Limitations
Crisis analytics must address false data, rumors, uncertainty, and other forms of noise that make relevant information harder to identify.
Abstract
from arXiv · showhide
Disasters have long been a scourge for humanity. With the advances in technology (in terms of computing, communications, and the ability to process and analyze big data), our ability to respond to disasters is at an inflection point. There is great optimism that big data tools can be leveraged to process the large amounts of crisis-related data (in the form of user generated data in addition to the traditional humanitarian data) to provide an insight into the fast-changing situation and help drive an effective disaster response. This article introduces the history and the future of big crisis data analytics, along with a discussion on its promise, challenges, and pitfalls.
I. INTRODUCTION
The 2010 Haiti earthquake response marked the emergence of digital humanitarianism, as technologies and distributed volunteers helped process crisis data and coordinate aid. This history motivates big crisis data analytics while linking it to earlier mapping and data-based crisis practices.
- I. INTRODUCTION: The Haiti earthquake response marked a new era of digital humanitarianism through rapid, technology-enabled humanitarian aid.The response used emerging digital tools and generated large amounts of crisis-related data.
- I. INTRODUCTION: Crowdsourcing and mapping helped digital humanitarians find actionable information within voluminous SMS and social media data.Ushahidi managed crowdsourcing for the Haiti earthquake response.
- I. INTRODUCTION: Crisis communications provide situational updates, medical needs, and aid-station locations that support both international assessment and response.Affected communities use calls, messages, emails, and tweets to report incidents and connect with others.
- I. INTRODUCTION: The Haiti response also produced distributed intelligence as volunteers and technical communities collaborated to interpret a large-scale calamity.This collaboration was described as the arrival of Disaster Response 2.0.
- I. INTRODUCTION: Earlier crisis analytics used spatial analysis and visualization, as illustrated by John Snow’s cholera mapping and Florence Nightingale’s healthcare analyses.These examples show that data-based crisis reasoning predates contemporary big-data technologies.
A. What is Big Data?
Big data refers to massive, largely unstructured datasets from multiple sources that exceed traditional processing capabilities because they are too large, too fast, or insufficiently structured. Crisis analytics draws on several sources, including digital traces and online user-generated activity.
- A. What is Big Data?: Big data comprises massive, largely unstructured datasets from multiple sources that traditional tools cannot process on a single state-of-the-art machine.The defining challenge may involve excessive size, speed, or lack of structure rather than volume alone.
- A. What is Big Data?: Figure 1 identifies six important sources of big crisis data within a broader crisis analytics taxonomy.The supplied passage names the taxonomy but does not enumerate all six sources.
- A. What is Big Data?: Crisis analytics uses data exhaust, including mobile call detail records, transaction records, and usage logs.Much data exhaust is privately owned and rarely shared publicly because of legal and privacy concerns.
- A. What is Big Data?: Online activity includes emails, SMS, blogs, comments, search queries, and social-network posts that can reveal crisis development.SMS is used mostly by affected communities on the ground, whereas Twitter is used mostly by international aid organizations.
C. Big Crisis Data Analytics
Big crisis data analytics combines data engineering and data-science techniques to extract insights from large crisis datasets. Its aim is efficient humanitarian response across thematic applications such as digital epidemiology and population-scale analysis.
- C. Big Crisis Data Analytics: Big data technologies comprise engineering for storing and processing data and analytics for mining insights with data-science methods.Engineering includes NoSQL, Hadoop, and Spark, while analytics combines computing, statistics, signal processing, data mining, machine learning, and visualization.
- C. Big Crisis Data Analytics: Big crisis data analytics applies AI, machine learning, data analytics, and digital platforms to improve humanitarian response across crises.The article identifies thematic applications including data-driven digital epidemiology using call records and social media.
A. Mobile Phones
Mobile, open, and visual technologies expand crisis-response capabilities by revealing population movements, combining distributed data, and enabling participatory geographic analysis. These tools support both large-scale coordination and near-real-time visual understanding of crisis sites.
- A. Mobile Phones: Anonymous mobile call records can reveal and predict population movements during disasters at relatively low cost.FlowMinder used mobile-operator data during the Haiti earthquake to track displacement and future trajectories.
- B. The Internet, Open Source, and Open Data: Open-source collaboration and open data enabled volunteers to combine satellite, mobile, health, and police information into shared crisis maps.OpenStreetMap became the de facto source of Haiti map data for most United Nations agencies after the earthquake.
- C. Neo-Geography and Visual Analytics: Visual analytics supports analytical reasoning through visual interfaces, while neo-geography uses participatory mapping and volunteered geographic content.OpenStreetMap exemplifies geographic data collected from volunteers rather than controlled exclusively by an elite group.
- C. Neo-Geography and Visual Analytics: Crisis mapping combines crowdsourced imagery with computer vision and geospatial techniques to provide an up-to-date virtual view of disaster sites.These visual products are intended to guide response efforts.
D. Leveraging the Wisdom and the Generosity of the Crowd
Problem solving in crisis response can draw on experts, crowds, or machines, with crowdsourcing and crowdcomputing enabling open, distributed contributions to time-critical tasks.
- Crowds provide diverse collective input that can complement expert knowledge in crisis problem solving.Experts offer experience but may suffer from biases, while diverse groups can benefit from the wisdom of crowds.
- Crowdsourcing uses open calls to outsource jobs traditionally performed by designated agents to large, undefined groups.It applies open-source principles beyond software and has been used for multiple disaster-response activities.
- Crowdcomputing applies crowds to complex problems, including time-critical crowdsearching supported by social networking, the Internet, and incentives.Microtasking platforms have emerged as a contemporary approach to crowdsearching.
E. Artificial Intelligence and Machine Learning
AI and machine learning support automated crisis problem solving, especially when exact algorithms are unavailable, across discovery and predictive analytics tasks.
- Machine learning is appropriate for non-trivial crisis problems when exact algorithms are not known, because it learns from data.AI and ML can formulate automated approaches to crisis-response problems.
- Big crisis data analytics includes discovery tasks such as clustering, outlier detection, and correlation analysis, plus prediction tasks such as classification, regression, and recommendation.
- Supervised learning infers a classifier from labeled examples, with support vector machines and random forests among common techniques.
2) Types of ML algorithms:
Crisis analytics uses unsupervised learning to find structure in unlabeled data and applies AI-based language and vision methods to humanitarian information.
- 2) Types of ML algorithms:: Unsupervised learning automatically discovers features and hidden structure without training data, helping reduce manual feature engineering and labeling.K-means and expectation-maximization are examples of unsupervised techniques.
- The paper identifies computational linguistics and computer vision as two important AI/ML application areas for crisis informatics.
- Computational linguistics supports automated social-media analysis through sentiment analysis, opinion mining, and crowdsourced translation.Natural-language processing remains non-trivial in crisis informatics; translation platforms were used in Haiti and Pakistan.
- Computer vision applies AI to aerial imagery from UAVs and satellites, supported by initiatives that broaden access to high-resolution commercial images.
4) Interfacing Human and Artificial Intelligence:
Effective crisis analytics should combine human judgment with machine speed while addressing technical, informational, volunteer, ethical, and privacy challenges.
- 4) Interfacing Human and Artificial Intelligence:: Human–machine collaboration can improve crisis analytics by combining human pattern recognition and unstructured-information coding with machine processing speed.AIDR uses crowds to generate classifiers, automatically tags tweets, and brings humans into the loop when confidence is low.
- Big crisis data extends the traditional four Vs with vagueness, virality, volunteers, validity, values, and visualization.These challenges concern language, misinformation, volunteer coordination, social-media bias, privacy and ethics, and crisis-data display.
- Crisis-data policies must govern access, reuse, linking, and ethical use because sensitive information can expose victims or others to harm.The paper specifically warns that requests containing personal information may be misused in conflict settings.
2) Ethical Big Crisis Data Analytics:
Ethical big crisis data analytics should follow established humanitarian principles grounded in international humanitarian law and guide work across humanitarian action.
- Big crisis data analytics communities should adopt humanitarian principles to guide their work.These principles define universal standards for humanitarian action and are widely accepted by humanitarian actors.
A. Caveat Emptor: Beware of the Big Noise
Crisis analytics must separate actionable information from overwhelming, unreliable, and potentially harmful data while recognizing that technology cannot replace coordinated human-led response.
- False, stale, biased, and intentionally manipulated data can obscure time-critical information during crises.Crowdsourced data may contain noise from pranks, cyber-attacks, rumors, stale reports, and sampling bias.
- Big crisis data analytics can improve emergency-response efficiency but remains one component of a broader crisis informatics ecosystem.The Haiti response benefited from crowdsourcing and crisis mapping, but radio and the Haiti diaspora were reported as more important.
- Real-time analytics should filter crisis information, prioritize urgent problems, broadcast warnings, and coordinate responders before further harm occurs.Timely action matters because stale information can be harmful, especially during cascading disasters.
B. Secure, Reliable, Disaster-Tolerant Crisis Analytics
Crisis analytics should be designed for adverse conditions, extend beyond hindsight toward prediction, and integrate diverse data sources to form a more complete picture of unfolding disasters.
- Crisis analytics systems require reliability, availability, security, redundancy, and diverse infrastructure to withstand severe disaster conditions.Relevant conditions include flooding, earthquakes, physical infrastructure damage, and power outages.
- Predictive crisis analytics can provide advance, understandable information and support personalized, context-aware alerts for affected or at-risk communities.Potential applications include earthquake notifications based on epicenter and user location, with additional social-network data integration.
- Most crisis-analytics research has focused on descriptive or diagnostic hindsight rather than forward-looking prediction.Descriptive analytics asks what happened or is happening, while diagnostic analytics asks why it happened.
- Multimodal analytics must reconcile text, images, speech, video, maps, crowdsourced data, and formal reports because each source offers only an incomplete view.Combining modalities is presented as a frontier for future crisis-analytics research.