Source-linked AI summary
'Beating the news' with EMBERS: Forecasting Civil Unrest using Open Source Indicators
Naren Ramakrishnan, Patrick Butler, Sathappan Muthiah, Nathan Self, Rupinder Khandpur, Parang Saraf, Wei Wang, Jose Cadena, Anil Vullikanti, Gizem Korkmaz, Chris Kuhlman, Achla Marathe, Liang Zhao, Ting Hua, Feng Chen, Chang-Tien Lu, Bert Huang, Aravind Srinivasan, Khoa Trinh, Lise Getoor, Graham Katz, Andy Doyle, Chris Ackermann, Ilya Zavorin, Jim Ford, Kristen Summers, Youssef Fayed, Jaime Arredondo, Dipak Gupta, David Mares
TL;DR
The paper asks whether open-source indicators can support prospective forecasting of civil unrest rather than only retrospective analysis. It presents EMBERS, a continuously operating multi-source forecasting system with structured alerts and fused models, and reports evaluations across 10 Latin American countries. The results show useful event-level and broader temporal forecasting performance, while the discussion identifies tunable precision–recall and quality–lead-time tradeoffs.
Problem
The paper addresses the need to determine whether precursors in open-source data can support prospective forecasting of civil unrest.
Method
EMBERS continuously ingests and analyzes open-source indicators, applies multiple data-specific models, fuses their predictions, and delivers structured alerts for civil unrest.
Results
Across 10 countries, EMBERS outperformed baserate methods and captured both specific event matches and significant protest upticks across countries.
Takeaways & Limitations
Prospective, continuously evaluated forecasting of civil unrest is feasible using heterogeneous open-source indicators and model fusion.
Takeaways & Limitations
The system’s future work includes modeling theory-based grievance-to-trigger processes and formalizing tradeoffs between precision–recall and quality–lead time.
Abstract
from arXiv · showhide
We describe the design, implementation, and evaluation of EMBERS, an automated, 24x7 continuous system for forecasting civil unrest across 10 countries of Latin America using open source indicators such as tweets, news sources, blogs, economic indicators, and other data sources. Unlike retrospective studies, EMBERS has been making forecasts into the future since Nov 2012 which have been (and continue to be) evaluated by an independent T&E team (MITRE). Of note, EMBERS has successfully forecast the uptick and downtick of incidents during the June 2013 protests in Brazil. We outline the system architecture of EMBERS, individual models that leverage specific data sources, and a fusion and suppression engine that supports trading off specific evaluation criteria. EMBERS also provides an audit trail interface that enables the investigation of why specific predictions were made along with the data utilized for forecasting. Through numerous evaluations, we demonstrate the superiority of EMBERS over baserate methods and its capability to forecast significant societal happenings.
1 Introduction
EMBERS addresses whether open-source signals can support continuous forecasting of civil unrest and other significant societal events. The paper presents its architecture, structured warnings, multi-model fusion, and evaluation across Latin America.
- EMBERS continuously monitors data sources, mines emerging trends, and processes them into forecasts of significant societal events such as protests.
- The system focuses on civil unrest, defined here as population-level protests against governments or larger organizations over policies, issues, or situations.
- EMBERS uses open-source feeds including tweets, web searches, news and blogs, economic indicators, Wikipedia, Internet traffic, and other sources.
- Unlike retrospective predictability studies, EMBERS has generated forecasts into the future since November 2012 for independent evaluation by MITRE.
- Its contributions include a modular 24x7 architecture, structured warnings, multi-model probabilistic fusion, and application across 10 countries.
2 What is Civil Unrest?
The paper treats civil unrest as a broad category of population-level political protest rather than private criminal activity. Latin America is selected because its unrest, media coverage, empirical research, and polling support model development and evaluation.
- Civil unrest encompasses population-level protests against government or larger organizations concerning policies, issues, or situations.
- The definition excludes acts undertaken for purely private criminal gain, while retaining illegal political protests as civil unrest.
- Latin America provides many unrest events for training, broad news coverage for ground truth, and empirical research and polling for contextual description.
3 Related Work
Related work spans event coding, civil-unrest characterization and forecasting, retrospective analysis of information preceding major events, and event extraction.
- Event-coding systems such as ICEWS and GDELT extract structured events from text and have supported predictive systems.
- Much civil-unrest research characterizes protest dynamics or distributions, while other studies retrospectively analyze signals preceding events such as the Arab Spring.
- EMBERS is situated within a growing literature on event extraction and prior analyses of Latin American protest signals in social media.
4 System Architecture
EMBERS is a modular platform that ingests heterogeneous open-source streams, enriches messages and locations, generates predictions, and delivers structured alerts continuously. Its architecture supports real-time processing and integration of multiple data sources and analytical modules.
- EMBERS continuously monitors open-source data streams and delivers structured alerts in real time for scoring by IARPA and MITRE.
- The architecture separates ingest, enrichment, prediction, and delivery, with flexible data transfer among the corresponding modules.
- The ingest layer handles Twitter, news and blogs, alerts, financial news, satellite and health indicators, Internet traffic, restaurant cancellations, surveys, and linked webpages.
- Twitter geocoding combines geotags, Twitter places, profile fields, and tweet text to infer geographic references.
- Text enrichment identifies language, tokenizes and lemmatizes content, extracts named entities, and normalizes date expressions.
- The PSL news geocoder weights locations, people, organizations, aliases, and population evidence to infer an article’s primary city, state, and country.
5 Prediction Models
EMBERS combines five prediction models that use distinct data sources and modeling assumptions to forecast structured civil-unrest events. The models include planned-protest detection, dynamic query expansion, cascades, volume-based prediction, and an MLE baseline.
- 5 Prediction Models: EMBERS evaluates five prediction models with different assumptions, data sources, and applicability scenarios.The models are presented as a set for forecasting civil unrest within the broader EMBERS architecture.
- 5.1 Planned Protest: The planned protest model detects organized future unrest from multilingual news, blogs, web pages, Twitter, and Facebook Events.It matches action-oriented expressions, identifies future dates, classifies event type and population, and geocodes locations.
- 5.1 Planned Protest: Basis RLP and TIMEN enrichment normalize language and temporal expressions, converting relative dates such as “mañana” into exact dates.The enrichment process supports later identification of when a planned event is expected to occur.
- 5.2 Volume-based Model: The volume-based model uses LASSO logistic regression to map protest-related volume features to country-level civil-unrest probabilities.Features include filtered tweet and news/blog counts, exchange rates, and TOR requests, with tweets requiring at least three keywords.
- 5.3 Dynamic Query Expansion: DQE dynamically expands a short protest-related seed query to discover emerging keywords and pertinent tweets rather than relying on a fixed vocabulary.This model is based exclusively on tweets and seeks changing conditions associated with protests.
- 5.4 Cascades Model: The cascades model tracks recruitment and cause popularization by characterizing information diffusion through activity cascades on directed Twitter networks.An activity cascade begins with a user post followed by related posts from followers within a short interval.
- 5.5 Baseline Model: The MLE baseline predicts future event schemas from their recent frequency in the GSR, using a three-month training interval, threshold two, and two-week lead time.An event schema combines location, event type, population, and day of the week.
6 Fusion and Suppression
The fusion and suppression engine converts model outputs into a final warning set by removing duplicates, completing missing fields, rewriting improbable predictions, and managing the recall-quality tradeoff.
- 6 Fusion and Suppression: The fusion and suppression engine generates the final set of warnings delivered to analysts.It performs several post-processing operations across outputs from the prediction models.
- Duplicate Detection and Warning Updating: Duplicate alerts sharing location, event type, population, and event date are discarded, while dates within two days trigger an update.The engine treats alerts differing only slightly in predicted dates as the same event.
- Filling Missing Values: Missing event type, population, or city-level location values are filled using their likelihood of appearance in the GSR.This operation supplies details that certain models cannot predict completely.
- Warning Rewriting: For improbable location-event-population combinations, the fusion model may rewrite the city to the historically most probable city within a specified radius.This assumes the broader region may still have been identified correctly when the city prediction is erroneous.
- Balancing the Recall-Quality Tradeoff: EMBERS trades recall for warning quality by suppressing alerts below a random-forest quality threshold or scoring alerts with a PSL engine.The PSL engine is trained on matched alerts and events to learn probabilities.
7 Audit Trail Interface
EMBERS archives system data and processing records so analysts can inspect which messages and analytic processes produced each warning through a web-based audit trail.
- 7 Audit Trail Interface: Archived data and processing records let EMBERS produce a complete audit trail for any generated warning.The data are stored in Amazon S3 and the processing chain is recorded in a NoSQL database.
- 7 Audit Trail Interface: The web dashboard enables analysts to search warnings, identify models and post-processing steps, and inspect the data sources used.The interface visualizes the evidence and processing path associated with an alert.
8 Evaluation Methodology
EMBERS evaluates structured alerts against independently organized GSR events using temporal and content-based matching, quality scoring, and aggregate precision, recall, lead-time, and probability measures.
- 8 Evaluation Methodology: Each alert records protest where, why, when, and who, forecast confidence, and the forecast date; GSR events record the corresponding realized details and reported date.The GSR is organized by the independent third party MITRE.
- 8.1 Matching Alerts to Events: The four dates are forecast date, predicted event date, actual event date, and reported date, which determine temporal matching and evaluation.Figure 6 illustrates matching when the forecast is issued before the event is reported.
- 8.1 Matching Alerts to Events: A warning qualifies as predicting an event when forecast date precedes reported date; lead time is reported date minus forecast date.The predicted-date versus actual-date difference measures forecast quality or accuracy.
- 8.2 Other Quality Aspects: Quality combines date, location, event-type, and population scores as QS = DS + LS + ES + PS.The quality score is designed to range from 0 to 4.
- 8.2 Other Quality Aspects: The date score is 1 for an exact event-date match and 0 when predicted and actual dates differ by more than seven days.The displayed formula truncates date error at seven days.
- 8.2 Other Quality Aspects: Location scoring compares country, state, and city matches using binary component scores, with an alternative distance-based formulation standardized through World Gazetteer coordinates.The distance formulation gives exact predictions score 1 and cities beyond 300 km score 0.33.
- 8.3 Inclusion Criteria: Allowable warning-event pairs require positive lead time, the same country, and predicted and actual dates within seven days; a 300 km location limit is optional and stringent.Inclusion criteria determine which pairs can be scored, whereas scoring criteria determine their score.
- 8.4 Matching Alerts to Events: Matching constructs a weighted bipartite graph whose allowable edges satisfy inclusion criteria and whose weights represent quality scores.A non-crossing matching is considered to prevent temporal criss-cross matches.
9 Evaluation Results
Across multiple evaluations, EMBERS produced higher-quality forecasts than baserate methods, while fusion, suppression, confidence scores, and matching constraints shaped performance and tradeoffs. The deployed system also tracked major protest upticks and improved over time.
- Cross-country performance: 3.11 quality, 8.8 days average lead time, 0.69 precision, and 0.82 recall were achieved across the 10 countries through model integration.Individual models showed selective country-level strengths, while fusion supported higher recall by combining limited warning sets.
- Fusion and suppression: The suppression engine shifted the quality-distribution mode from around 2.25 to around 3.2 by learning which warnings to suppress or issue.This tuning capability directly supported balancing recall against warning quality.
- Comparison with baserates: +0.4 quality improvement over baserate methods under lenient inclusion criteria increased to +1.0 under strict location-based criteria.The baserate model generated alerts from event occurrence rates during the preceding three months.
- Significant uprisings: Nearly 0.5 improvement over baserate models occurred during months of significant uprisings, while normal-activity months showed comparable performance.During Brazil’s June 2013 protests, EMBERS tracked the rise in protest counts accurately; later violence upticks in Brazil and Venezuela were also captured.
- Confidence and matching: Warning probability scores had a monotonic relationship with match likelihood, indicating that confidence captured mappings from warning attributes to event matches.The evaluation examined whether emitted probabilities were reliable indicators of matching likelihood.
- Lead time and quality: Quality scores initially decreased as lead time increased, then improved beyond a threshold because planned-protest sources supplied high-quality, long-lead warnings.Facebook event pages and other feeds contributed to this high-lead-time pattern.
- Confidence and matching: Non-crossing matching reduced the number of matches and overall quality, yet EMBERS maintained consistent improvement over baserate methods.The constraint preserves temporal order between warnings and events.
- Performance over time: Quality scores rose from just over 2 in the prior year to above 3.0 in recent months for the deployed EMBERS system.This result reflects performance over time rather than a single evaluation period.
10 Discussion
EMBERS is presented as an automated forecasting system whose evaluations across 10 countries covered both individual event matching and broader protest upticks. The discussion identifies future work on theory-based forecasting, tradeoff analysis, and analyst-facing narratives.
- Discussion: EMBERS evaluations across 10 countries addressed both specific-event warning matches and significant cross-country upticks.The paper characterizes these as capabilities “in the small” and “in the large.”
- Evaluation scope: Quality scores in the model-comparison tables range from 0 to 4, with 4 representing the most accurate forecasts.The table uses country abbreviations and marks countries without warnings using a dash notation.
- Future work: Future work targets theory-based modeling of grievances and triggers, statistical analysis of precision-recall and quality-lead-time tradeoffs, and automated alert narratives.The proposed system is intended to remain tunable for analysts with different preferences along these tradeoff boundaries.