Source-linked AI summary
Down for Failure: Active Power Status Monitoring
Niloofar Bayat, Kunal Mahajan, Sam Denton, Vishal Misra, Dan Rubenstein
TL;DR
Direct power monitoring systems can be costly to install and maintain, and the paper explores IP probing as an alternative for identifying regional outages. The method dynamically selects residential IPs whose availability correlates with regional power and uses their probes to assess regional outage status. During massive power outages such as those caused by Hurricane Florence, detections matched power company reports 90% of the time, with false positive and false negative rates averaging below 10%.
Problem
Direct power monitoring systems can be costly to install and maintain, and the paper explores IP probing as an alternative for identifying regional outages.
Source: §1 · p. 1Method
Results
During massive power outages such as those caused by Hurricane Florence, detections matched power company reports 90% of the time, with false positive and false negative rates averaging below 10%.
Source: §1 · p. 2Takeaways & Limitations
IP probing can serve as a complementary technique for power outage detection, with the paper describing an outage-centric model adjusted against power company data.
Source: §6 · p. 11The evaluation focused on major weather events; applying the method to minor events requires further fine-tuning to reduce false-positive and false-negative rates.
Source: §6.1 · p. 11
Abstract
from arXiv · showhide
Despite society's strong dependence on electricity, power outages remain prevalent. Standard methods for directly measuring power availability are complex, often inaccurate, and are prone to attack. This paper explores an alternative approach to identifying power outages through intelligent monitoring of IP address availability. In finding these outages, we explore the trade-off between the accuracy of detection and false alarms. We begin by experimentally demonstrating that static, residential Internet connections serve as good indicators of power, as they are mostly active unless power fails and rarely have battery backups. We construct metrics that dynamically score the reliability of each residential IP, where a higher score indicates a higher correlation between that IP's availability and its regional power. We monitor specifically selected subsets of residential IPs and evaluate the accuracy with which they can indicate current county power status. Using data gathered during the power outages caused by Hurricane Florence, we demonstrate that we can track power outages at different granularities, state and county, in both sparse and dense regions. By comparing our detection with the reports gathered from power utility companies, we achieve an average detection accuracy of $90\%$, where we also show some of our false alarms and missed outage events could be due to imperfect ground truth data. Therefore, our method can be used as a complementary technique of power outage detection.
1 Introduction
The paper explores IP probing as an alternative for identifying regional power outages, distinguishing power failures from other causes of Internet unresponsiveness. It evaluates residential IP indicators against utility reports, including during Hurricane Florence.
1 Introduction
- The paper frames IP probing as an alternative to direct power monitoring, which can rely on costly wired communications that are not widely implemented.The introduction notes that outages have economic, social, physical, and psychological impacts.Source: §1 · p. 1
- The method distinguishes outages due to power failure from general Internet outages and can assess conditions online as outages occur.Prior Internet-based probing methods focused on general outages regardless of cause.Source: §1 · p. 1
- The study compares static residential IP behavior with utility reports across more than 1,500 U.S. counties to assess their suitability as power indicators.The introduction identifies static residential IPs as especially susceptible to power outages.Source: §1 · p. 1
- The approach identifies residential IPs whose up/down status correlates with regional power and dynamically selects the most correlated addresses for a watchlist.The watchlist is updated over time.Source: §1 · p. 2
- The paper presents its contributions as designing a process to identify IPs whose availability is correlated with regional power.The paper also describes scanning designed to address concerns about ICMP flood rates.Source: §1 · p. 2
2 Design Challenges
The design must find IPs that track regional power, separate power-related failures from other causes of unresponsiveness, and scan without excessive network traffic. Practical constraints include unreliable address classes, unused address blocks, and limited probing capacity.
2 Design Challenges
- The design challenge is to select IPs that are up when regional power is available and down when it is not, while distinguishing power failures from other causes of unresponsiveness.Other causes include BGP misconfigurations, DDoS attacks, and ISP outages.Source: §2 · p. 2
Which IP addresses to monitor
- Residential IP indicators can be unreliable when dynamically assigned, backed by data-center or business power, or historically unresponsive despite power being available.The approach seeks to avoid such classes, build sampling histories, and weight regional assessments toward more reliable indicators.Source: Which IP addresses to monitor · p. 2
Minimizing Internet background radiation
- Unused address blocks are uninformative, so the system excludes them to reduce unwanted traffic and Internet background radiation.If unused blocks later become assigned, the stochastic selection process can cover them over time.Source: Minimizing Internet background radiation · p. 2
Impacting Internet service
- Scanning must provide useful outage information without overloading a region or a particular subset of IP addresses.Increasing probe rates may improve detection accuracy but can cause unintended anomalous behavior.Source: Impacting Internet service · pp. 2–3
Bandwidth limitations
- Probe-source capacity limits can constrain monitoring and require available bandwidth to be partitioned among regions.Source: Bandwidth limitations · p. 3
3 Methodology and Design
The methodology periodically scans IPs to infer regional power availability, first forming a residential watchlist and then refining it into a smaller reliable watchlist that is continually updated.
3 Methodology and Design
- The pipeline periodically scans IPs, maps identified residential addresses to geographic regions, and refines a monitored residential watchlist into a smaller, dynamically adjusted reliable watchlist.The reliable watchlist contains IPs judged to be the best indicators of power availability; its formation, updates, and probe interpretation are described as part of the pipeline.Source: §3 · p. 3
3.1 Residential IP determination
The paper identifies residential IPs using ISP, port, and security-method characteristics, then filters out indicators associated with non-residential connections. The resulting address blocks form the residential watchlist, with nonresponding unassigned IPs removed in a later step.
3.1 Residential IP determination
- Residential IP selection uses ISP names, port numbers, and security methods as indicators, filtering out ports and methods associated with non-residential connections.Shodan supplies host IPs categorized by ISP; ports 110, 123, 161, and 5060 are avoided, as are HTTP, FTP, SSH, and telnet security methods.Source: §3.1 · p. 3
- The filtering process yields approximately 200,000,000 U.S. residential IP addresses as the basis of the residential watchlist.Because the watchlist consists of CIDR blocks, unassigned IPs that do not respond to ICMP pings are removed during reliable-watchlist formation.Source: §3.1 · p. 3
3.2 Geolocational mapping of IP addresses
IP addresses are mapped to geographic regions so the system can monitor addresses assigned to each region. County assignments are derived from mapped coordinates.
3.2 Geolocational mapping of IP addresses
- The authors use MaxMind City database coordinates to map residential IP addresses to geographic locations.Source: §3.2 · p. 3
- MaxMind is reported as up to 86% accurate for mapping IP addresses within a 50-kilometer radius.Source: §3.2 · p. 3
- An FCC API assigns each IP address to a county based on its coordinates.Source: §3.2 · p. 3
3.3 The Residential Watchlist
The residential watchlist is screened to exclude blacklisted addresses and monitored with periodic ICMP probes. The list is updated as exclusions and newly identified residential IP blocks arise.
3.3 The Residential Watchlist
- The watchlist excludes reserved IP addresses and IP addresses or blocks submitted through the opt-out process.Source: §3.3 · p. 3
- ZMap probes the remaining residential watchlist every six hours with ICMP pings to record general availability.Source: §3.3 · p. 3
- The system updates the blacklist as opt-out requests arrive and adds newly identified blocks that may contain residential IP addresses.Source: §3.3 · p. 3
3.4 Maintaining and Using the Reliable Watchlist: DDII method
DDII maintains a dynamically adjusted reliable watchlist by scoring residential IPs, selecting a smaller subset for faster regional scans, and using scan outcomes to assess power status. Regions with too few expected responses are not tracked, and scan data from suspected outages are excluded from score updates.
3.4 Maintaining and Using the Reliable Watchlist: DDII method
- DDII scores residential IP addresses from probe histories and uses those scores to maintain a smaller, dynamically adjusted reliable watchlist.It also adjusts IP scores and regional probing periods based on outage-status results.Source: §3.4 · p. 4
3.4.1 IP Address Scoring
- Each IP's score is an exponential weighted moving average of Boolean probe responses, initialized at 0.5 with α = 0.01.The initial-score bias fades over subsequent samples, while recent history receives more weight than older history or only the most recent few probes.Source: §3.4.1 · p. 4
3.4.2 Slow-Period Full Residential Watchlist Scans
- Low-rate scans probe the full residential watchlist every six hours and initially assume power is available when interpreting nonresponses.This assumption is intended to identify IPs that are usually available when power is present.Source: §3.4.2 · p. 4
3.4.3 Reliable Watchlist Scans
- Faster regional scans probe a smaller reliable subset, and their results update IP scores only when the region is hypothesized to have power.Excluding scans during perceived outages limits outage-related nonresponses from biasing IP scores.Source: §3.4.3 · p. 4
3.4.4 Reliable Watchlist Sizing
- The expected response rate E_R determines regional watchlist sizing, and regions with E_R < 10 are not tracked.The threshold is an arbitrary trade-off between detection accuracy and the number of covered regions; the watchlist grows linearly with E_R at small values and then sub-exponentially.Source: §3.4.4 · p. 4
3.4.5 Reliable Watchlist Formation
The reliable watchlist is reconstructed for each sample, with selection favoring IPs that have higher availability scores while retaining randomness. Regional failures are assessed against expected responses, checked across ISPs for power-outage attribution, and followed by adaptive scan rates.
3.4.5 Reliable Watchlist Formation
- Figure 2 relates reliable-watchlist size N_R to the expected response probe rate E_R for a region.Source: Figure 2 · p. 5
- For each sample, the reliable watchlist is reconstructed, with IP selection probability proportional to its current score.Random selection also includes other IPs, reducing the burden on high-scoring addresses and sampling lower-scoring ones more often than six-hour scans alone.Source: §3.4.5 · pp. 4–5
3.4.6 Regional Failure Determination
- A regional failure is indicated when the gap between expected and observed reliable-watchlist responses exceeds threshold τ.The threshold can be varied to detect smaller or larger-scale outages; the study uses τ = 0.07 to decide whether a sample updates IP scores.Source: §3.4.6 · p. 5
3.4.7 Distinguishing Power Outages from General Internet Outages
- After regional failure detection, the method reports a power outage only when the same condition holds across the region's ISPs.The heuristic assumes power loss affects clients served by different ISPs similarly, while other causes typically affect only a subset.Source: §3.4.7 · p. 5
- Multi-ISP failures can still have non-power causes, including a shared submarine cable cut or nationwide Internet blackout.The authors state that limited data leave distinguishing these cases for future work.Source: §3.4.7 · p. 5
3.4.8 Reliable Watchlist Scan Frequency
- Regional scan frequency adapts to detected status, increasing toward every two minutes during suspected outages and decreasing to every ten minutes when power is assumed available.The rate changes gradually as the region's counter changes after each scan.Source: §3.4.8 · p. 5
3.5 Preliminary Measurements
IP scores distinguish residential addresses by ping responsiveness, supporting a focus on high-scoring addresses. Reliable watchlists are much smaller than residential watchlists, reducing monitoring traffic and bandwidth use.
3.5 Preliminary Measurements
- Many residential addresses have low IP scores and respond infrequently to pings, so the method focuses monitoring on high-scoring addresses.Figure 3 shows the distribution of IP scores among U.S. residential addresses.Source: §3.5 · p. 5, Figure 3 · p. 6
- Reliable watchlists are generally four orders of magnitude smaller than residential watchlists, reducing ICMP ping flooding and bandwidth utilization.Count distributions are shown per county; counties with NR < 10 are excluded, leaving a gap at zero in the reliable-watchlist distribution.Source: §3.5 · p. 6
4 Evaluation
The evaluation compares DDII’s regional IP-probe detections with utility outage reports, while noting that the reports are an imperfect reference. It examines baseline report patterns, monitoring behavior, county-level detection, and performance during Hurricane Florence.
4.1 Existing Baseline: Utility Company Reports
- Utility outage reports showed a diurnal pattern peaking near midday during a two-week period without major outages, with nighttime declines offered as hypotheses rather than confirmed causes.The report percentages were normalized to local time zones.Source: §4.1 · p. 6
4.3.1 Detection thresholds
- DDII increased county scanning frequency after detecting failure, reaching a maximum rate of 0.0083 s^-1 and sustaining it longer for Robeson’s longer outage.Scanning frequency varied during recovery until it stabilized.Source: §4.3.1 · p. 7
- Concurrent drops across multiple ISPs supported interpreting the Robeson and Lancaster events as power outages rather than failures confined to one ISP.The evaluation compares ISP-specific active-IP fractions.Source: §4.3.1 · p. 7, Figure 7 · p. 8
4.4 DDII Validity
- DDII recorded more detections than utility reports at the beginning and end of outages, which the authors suggest could reflect evacuation, reduced Internet activity, or slower connection recovery.The authors describe the widened detection window as an advantage for identifying outages earlier.Source: §4.4 · p. 8
- At Hurricane Florence’s peak, false omission and false positive rates were slightly higher and accuracy slightly lower; the measures stabilized as outages declined.The evaluation uses a 6-hour buffer when matching detections to utility reports.Source: §4.4 · p. 9, Figure 10 · p. 10