Source-linked AI summary
Runaway Feedback Loops in Predictive Policing
Danielle Ensign, Sorelle A. Friedler, Scott Neville, Carlos Scheidegger, Suresh Venkatasubramanian
TL;DR
Predictive-policing systems can create runaway feedback because police deployment determines where discovered crime is observed and fed back into the model. The paper develops urn-based formal and empirical analyses, then proposes black-box input filtering; it finds that feedback worsens with crime-rate disparities and reliance on discovered incidents, while reported incidents only attenuate it.
Problem
Predictive-policing models learn from discovered incidents observed where police are sent, creating feedback that can repeatedly redirect deployment away from true crime rates.
Method
The paper uses urn models to analyze predictive-policing feedback and filters inputs to existing systems without modifying their internal workings.
Results
Runaway feedback is exacerbated by greater crime-rate disparities and heavier reliance on discovered incidents; reported incidents reduce but do not eliminate it.
Takeaways & Limitations
Black-box input filtering provides a way to counteract runaway feedback and support learning the true crime rate.
Takeaways & Limitations
The analysis assumes reliable reporting and discovery rates, while noisy, area- or crime-type-skewed reporting may leave the proposed solutions ineffective.
Abstract
from arXiv · showhide
Predictive policing systems are increasingly used to determine how to allocate police across a city in order to best prevent crime. Discovered crime data (e.g., arrest counts) are used to help update the model, and the process is repeated. Such systems have been empirically shown to be susceptible to runaway feedback loops, where police are repeatedly sent back to the same neighborhoods regardless of the true crime rate. In response, we develop a mathematical model of predictive policing that proves why this feedback loop occurs, show empirically that this model exhibits such problems, and demonstrate how to change the inputs to a predictive policing system (in a black-box manner) so the runaway feedback loop does not occur, allowing the true crime rate to be learned. Our results are quantitative: we can establish a link (in our model) between the degree to which runaway feedback causes problems and the disparity in crime rates between areas. Moreover, we can also demonstrate the way in which \emph{reported} incidents of crime (those reported by residents) and \emph{discovered} incidents of crime (i.e. those directly observed by police officers dispatched as a result of the predictive policing algorithm) interact: in brief, while reported incidents can attenuate the degree of runaway feedback, they cannot entirely remove it without the interventions we suggest.
1. Introduction
Predictive policing illustrates a broader problem in batch-mode machine learning: deployment decisions influence the feedback data used for later updates. Because discovered incidents are observed where police are sent, this process can compound sampling bias into runaway feedback loops.
- 1. Introduction: Decisions made by deployed models influence the data available for future training, making traditional batch learning frameworks inappropriate for these feedback settings.The paper gives hiring and predictive-policing examples in which feedback is available only for selected people or neighborhoods.
- 1. Introduction: Predictive policing allocates patrol officers using historical crime incident data and updates the model with observations from patrolled neighborhoods.The paper distinguishes discovered incidents observed by police from reported incidents submitted by residents.
- 1. Introduction: Discovered incidents can compound sampling bias because they occur only in neighborhoods selected by the predictive-policing algorithm.This mechanism has been shown to produce runaway feedback loops in predictive policing.
- 1. Introduction: Lum and Isaac’s Oakland simulation found that increasing policing based on discovered incidents caused PredPol’s predictions to substantially diverge from true crime.Their study modeled drug crime using historical incident data and a synthetic population of likely drug users.
- 1. Introduction: The paper uses urn theory to analyze predictive-policing feedback formally and empirically, then proposes black-box input remedies with theoretical justification.The interventions modify inputs rather than the predictive system’s internal workings.
2. Related Work
Related work establishes predictive policing as a concrete setting where models learn from outcomes shaped by their own deployment. The paper builds on prior PredPol analyses while positioning the problem as relevant to other selectively observed prediction systems.
- 2. Related Work: The paper connects predictive-policing feedback to related concerns in recidivism prediction, where outcomes are observed only for released prisoners.It also notes that proprietary implementation details limit direct comparison with systems such as COMPAS.
- 2. Related Work: PredPol assumes an earthquake-aftershock model in which previous crimes increase the predicted likelihood of subsequent crimes in a region, with decay over time.Its parameters are estimated using an expectation-maximization procedure.
- 2. Related Work: PredPol uses incident data by region to estimate crime rates and sends officers to areas with the highest predicted intensity.The resulting discovered incidents are fed back into the system, so it predicts where incidents will be reported or discovered rather than directly observing where crime occurs.
3. Predictive Policing with Urns
The paper models predictive-policing dynamics with generalized Pólya urns, using sampled colors and a replacement matrix to represent feedback-driven updates. This framework supports formal analysis of how observed incidents reshape future allocation.
- 3. Predictive Policing with Urns: Urn models provide the paper’s mathematical framework for modeling predictive-policing processes and their convergence behavior.The authors draw on urn models’ established use in machine learning and reinforcement learning.
- 3. Predictive Policing with Urns: A generalized Pólya urn contains red and black balls; each step samples and replaces a ball before updating the urn with a replacement matrix.The sampled color determines which row of the matrix governs the additions.
- 3. Predictive Policing with Urns: The replacement matrix specifies that sampling red adds a more red balls and b more black balls, while sampling black uses the corresponding c and d additions.The matrix therefore encodes how observations reinforce or alter the future sampling distribution.
- 3. Predictive Policing with Urns: The standard Pólya urn is the generalized model with a = d = 1 and b = c = 0.Under these settings, the sampled color is reinforced without cross-color additions.
- 3. Predictive Policing with Urns: PredPol critically conflates the amount of crime with the amount of incident data.This limitation motivates analyzing how feedback can distort incident-based estimates.
3.1. Goals and assumptions
The model’s goal is to allocate police in proportion to underlying crime rates while representing decisions, observations, and incident reporting with simplified assumptions. Discovered incidents are the feedback-linked data source, whereas reported incidents are not directly selected by deployment.
- 3.1. Goals and assumptions: The basic setting has one officer choosing between regions A and B each day, with the objective of deploying police in proportion to each region’s crime.Both officer observations and resident reports enter the incident data.
- 3.1. Goals and assumptions: Achieving effective policing requires learning the relative crime rates of the regions.A region with Λ percent of precinct crime should receive Λ percent of the police.
- 3.1. Goals and assumptions: The predictive model is simplified to use current statistics, with the officer tossing a coin based on those statistics to choose the next region.The model therefore abstracts allocation as a probability determined by current data.
- 3.1. Goals and assumptions: The model retains only crime counts as context and represents underlying crime through discovered and reported incident rates.For region A, total incident-data rate is wd · dA + wr · rA, with wr + wd = 1.
- 3.1. Goals and assumptions: The truth-in-crime-data assumption sets both discovered and reported rates equal to the underlying crime rate, although the paper notes that this optimistic assumption may not hold.The analysis therefore begins in a generous setting before examining feedback effects.
- 3.1. Goals and assumptions: Under the discovery-only assumption, neighborhoods without officers contribute no incidents to the data because wd = 1 and wr = 0.This assumption isolates the feedback channel created by police presence.
3.2. Uniform crime rates
The uniform-crime model represents predictive-policing updates with a slowed Pólya urn, but feedback prevents the system from learning that areas have identical crime rates.
- Urn model: The model represents crime statistics in areas A and B as red and black balls in an urn.Visiting an area samples its corresponding color, and observing crime adds another ball of that color.
- Urn model: Crime observations occur with probability λ, slowing standard Pólya-urn updates while preserving their long-term convergence behavior.The process either adds a same-color ball or merely replaces the sampled ball.
- Result: The limiting probability of sampling red is a draw from Beta(nr, nb), governed only by the urn’s initial red and black counts.These initial counts represent the system’s prior belief.
- Comparison: Without feedback, uniformly random patrols would converge to the uniform crime rate in each area.This resembles the standard update for a coin’s bias under a Beta prior.
3.3. Non-uniform crime rates
With non-uniform crime rates, the urn update can be reduced to a deterministic model whose limiting allocation commits entirely to the higher-crime region, even when the rates differ only slightly.
- Model: The non-uniform model assigns area-specific crime probabilities λA and λB, making urn updates random rather than deterministic.The update adds a ball according to Bernoulli variables with these probabilities.
- Reduction: Conditioning on time steps when a ball is added eliminates update randomness while retaining sampling randomness.The resulting process is equivalent to a deterministic Pólya urn that adds λi balls after sampling color i.
- Result: The limiting fraction of red balls is 1 when λA > λB and 0 when λB > λA.The deterministic urn’s limit follows from the asymptotic characterization of its replacement matrix.
- Interpretation: A 10% versus 11% crime-rate difference makes the process settle on the 11% region with probability 1.The authors characterize this as a feedback-driven “go with the winner” runaway effect.
3.4. Accounting for reported incidents
Adding reported incidents to discovered incidents changes the urn’s replacement process and yields a limiting estimate governed by both reported-crime weight and the weighted discovered-crime difference.
- Model: The combined-input model weights discovered and reported incidents separately, producing a replacement matrix based on both sources.For a visited area, the total weight is wddA + wrrA; otherwise it is wrrA.
- Analysis: The replacement matrix is reinterpreted as a deterministic update process so its limiting behavior can be analyzed with the urn lemma.The associated quadratic equation determines the candidate limiting fraction.
- Parameterization: Defining R = wr(rA + rB) and Δd = wd(dB − dA) rewrites the dynamics in terms of reported-incident weight and weighted discovered-crime disparity.These quantities summarize the two incident sources in the limiting equation.
- No feedback: The no-feedback case wd = 0 converges to λ* = rA/(rA + rB), the expected fraction from reported crime rates.The urn replacement matrix is then fixed regardless of which area is sampled.
- Interpretation: As λ* approaches 0 or 1, the limiting fraction x* approaches λ*, while intermediate disparities require either weak feedback or similar discovered-crime rates for accurate convergence.The paper notes that weak feedback means down-weighting discovered incidents, whereas similar discovered rates make feedback less consequential.
3.5. Modifying the urn model to account for feedback
The standard urn update overweights discovered incidents because their frequency depends on deployment, producing estimates unrelated to true crime rates. The proposed correction uses rejection sampling on discovered data while preserving reported incidents appropriately, so expected contributions track the underlying rates.
- The standard urn can converge to a distribution unrelated to true crime rates when update rates are stochastic.
- Discovered Incidents Only: The correction samples an additional ball and adds the new ball only when the colors differ, counteracting deployment-dependent sampling.
- Discovered Incidents Only: The rejection-sampling correction makes expected contributions proportional to the respective replacement rates, up to a state-dependent constant.
- Discovered Incidents Only: If police visit region A 90% of the time, discovered incidents from A are down-sampled 90% of the time, with analogous down-sampling for region B.
- Reported and Discovered Incidents: Applying the correction only to discovered incidents but down-weighting reported incidents unchanged gives an incorrect limit because the unvisited region is undercounted.
- Reported and Discovered Incidents: Removing the down-weighting of reported incidents in the unvisited neighborhood makes expected contributions equal to λA A-balls and λB B-balls.
4. Evaluating the urn model
Experiments instantiate the urn model with historical Oakland incident data and decay, showing runaway feedback under standard updates. The improvement policy instead converges toward the true crime-rate distribution with discovered incidents alone or with equally weighted reported incidents.
- The evaluation models limited memory by allowing each urn ball to disappear independently after every round.
- The full decay model lacks a theoretical characterization of its asymptotic distributions, so the study presents empirical evidence instead.
- 609, 379, and 7 initial balls represent historical incidents in Top1, Top2, and Random regions, respectively.
- Illustrating runaway feedback in urns: With discovered incidents only, the standard urn converges to sending police only to the neighborhood with the most crime, replicating the reported PredPol feedback problem.
- Illustrating runaway feedback in urns: With equally weighted reported and discovered incidents, deployment error decreases but the urn still fails to converge to the correct crime rate.
- Evaluating the modified urn: Under the improvement policy, urns converge to a distribution representing the true crime rate with discovered incidents alone or with both incident types.
5. Fixing PredPol
The paper repairs PredPol by filtering discovered-incident inputs rather than modifying the system internally. Simulations show that this policy brings deployment predictions toward the correct crime-rate ratio, although reported incidents alone do not eliminate runaway feedback.
- Simulation setup: PredPol trains on a rolling 180-day incident window and probabilistically deploys police according to predicted regional crime rates.The deployment probability is p = rA(rA+rB)^-1.
- Black-box repair: The improvement policy filters discovered incidents before training, making incidents from more heavily targeted districts less likely to enter the dataset.The filtering probability is p = rO(rA + rB)^-1, where rO is the predicted rate for the district not policed that day.
- Simulation results: Across 300 one-year simulations, regular PredPol fluctuates widely and fails to converge to the appropriate crime rates.The comparison examines Top1 deployment relative to Top2 and Random.
- Simulation results: Under the improvement policy, PredPol’s prediction rates fluctuate around the correct crime ratio, though the process remains noisy.The figure compares discovered-only inputs with inputs containing both discovered and reported incidents.
- Reported and discovered incidents: Even with many reported incidents, PredPol remains susceptible to runaway feedback, whereas correcting discovered incidents produces convergence toward appropriate crime-rate predictions.The correction is applied only to discovered incidents in the mixed reported-and-discovered setting.
6. Discussion and Limitations
The paper formally and empirically models predictive-policing feedback with urn models and proposes black-box input filtering as a remedy. It finds that feedback worsens with crime-rate disparities and reliance on discovered incidents, while several modeling and reporting assumptions limit the analysis.
- Contributions: Urn models formally model predictive-policing feedback and indicate remedies, which the paper demonstrates both formally and empirically.The proposed solution filters inputs fed to the predictive-policing system.
- Discussion: Runaway feedback is exacerbated when crime rates vary more between regions and when the model relies more heavily on discovered incident reports.When regional crime rates are similar, feedback is much less, although predictions can still be inaccurate.
- Limitations: The analysis does not cover richer predictive systems using information such as demographics or arrest profiles.The urn abstraction captures feedback issues but omits these additional inputs.
- Limitations: The approach assumes reported and discovered incident rates track true crime rates, although reporting can be noisy and skewed by area and crime type.Without this assumption, the analysis becomes more complicated and the proposed solutions might not continue to work.