PaperlayerView exact revision on arXiv

Source-linked AI summary

Early Warning Signals and the Prosecutor's Fallacy

Carl Boettiger, Alan Hastings

arXiv:1210.1204v1q-bio.PEphysics.soc-phq-bio.QM

TL;DR

Historical observations are used to assess early-warning signals, but selecting systems because they already transitioned raises a risk of the Prosecutor’s Fallacy. The paper analyzes simulated chance transitions and finds that selected cases are more likely to show warning-associated patterns.

  • Problem

    Historical observations have been used to evaluate early-warning signals, but it matters whether systems are selected because a transition has already occurred.

    Source: 201. Introduction · p. 2

  • Method

    The authors analyze simulated systems, comparing cases selected for chance transitions with the broader simulations and examining common warning indicators.

    Source: 902. Methods and Results · pp. 4–5

  • Results

    Chance-transition systems are statistically different from counterparts driven by the same stochastic process and are more likely to show signs associated with early warning.

    Source: 1513. Discussion · p. 7

  • Takeaways & Limitations

    Historical-transition examples can show warning-associated patterns more often after selection, so they require care when used to evaluate warning signals.

    Source: Testing patterns on historical data · p. 3, 1513. Discussion · p. 7

    Choosing where to truncate a trajectory is necessarily arbitrary without an explicit criterion, limiting practical application.

    Source: §3.2 · p. 8

Abstract

from arXiv · show

Early warning signals have been proposed to forecast the possibility of a critical transition, such as the eutrophication of a lake, the collapse of a coral reef, or the end of a glacial period. Because such transitions often unfold on temporal and spatial scales that can be difficult to approach by experimental manipulation, research has often relied on historical observations as a source of natural experiments. Here we examine a critical difference between selecting systems for study based on the fact that we have observed a critical transition and those systems for which we wish to forecast the approach of a transition. This difference arises by conditionally selecting systems known to experience a transition of some sort and failing to account for the bias this introduces -- a statistical error often known as the Prosecutor's Fallacy. By analysing simulated systems that have experienced transitions purely by chance, we reveal an elevated rate of false positives in common warning signal statistics. We further demonstrate a model-based approach that is less subject to this bias than these more commonly used summary statistics. We note that experimental studies with replicates avoid this pitfall entirely.

201. Introduction

The introduction frames historical observations as a way to assess early-warning signals, while warning that conditioning on observed transitions can create a Prosecutor’s Fallacy. It distinguishes the probability of evidence given innocence from the probability of innocence given evidence.

201. Introduction

  • Historical-data evaluations risk the Prosecutor’s Fallacy when they confuse the probability of observing evidence given innocence with innocence given the evidence.The two conditional probabilities are related by Bayes’ theorem but need not be equal; this error was recognized in the Collins case (Thompson and Schumann, 1987).Source: 201. Introduction · p. 2
  • The paper examines whether a similar statistical mistake arises when historical transitions are used to evaluate early-warning methods.The question is relevant because historical observations have been used alongside experimental tests of the indicators.Source: 201. Introduction · p. 2
  • Early-warning patterns such as increasing variance and autocorrelation have been proposed to identify approaching tipping points and give managers time to respond.The introduction also notes that these indicators have been tested with experimental manipulations and historical observations.Source: 201. Introduction · p. 2

Testing patterns on historical data

Testing warning indicators on historical transitions can be biased by selecting systems because a transition has already occurred. Rare transitions caused purely by chance may nonetheless look like transitions that warning signals are designed to detect.

Testing patterns on historical data

  • Because historical transition records are selected for having undergone a transition, evaluating warning methods on them is susceptible to statistical mistakes.The concern is conditional selection: the transition itself determines which systems enter the analysis.Source: Testing patterns on historical data · p. 3
  • Early-warning theory concerns a system slowly approaching a tipping point before a sudden shift, but some sudden transitions instead arise by chance.The chance scenario is an unlikely excursion away from a stable attractor, rather than a system driven across a tipping point.Source: Testing patterns on historical data · p. 3
  • Conditioning analysis on systems that have already transitioned can make chance transitions appear to provide evidence of an approaching tipping point.Such chance transitions may be rare in advance but become consequential when selected from many possible historical transitions.Source: Testing patterns on historical data · p. 3
  • A chance transition in the stable-system simulation occurred in only 1% of simulations, despite resembling a transition expected to contain an early-warning signal.The figure contrasts a bifurcation trajectory with a similar-looking trajectory from a stable system.Source: Testing patterns on historical data · p. 4

902. Methods and Results

The authors simulate stable stochastic systems that sometimes collapse by chance, then examine warning indicators and a model-based estimate on collapse-selected records. In these selected simulations, the model-based estimates did not differ from zero.

902. Methods and Results

  • The stochastic birth-death model has an Allee threshold: below it the population declines to zero, while above it the population returns to a positive equilibrium.The model’s key feature is the alternate stable states introduced by the Allee effect; the authors also provide an example using a discrete-time model with two non-zero stable states.Source: 902. Methods and Results · pp. 4–6
  • Across 1,000 replicates, 266 experienced population collapse; simulations spanned 50,000 time units, sampled every 50 time units, and analysis focused on a 500-time-point data section.The selected window around each system ended just before collapse.Source: 902. Methods and Results · p. 5
  • The authors measured variance and autocorrelation in moving windows and used Kendall’s τ to test for increasing trends across the collapse-selected replicates.The simulations were compared with unconditioned simulations using time series of equal length.Source: 902. Methods and Results · p. 5
  • None of the model-based estimates across 266 collapse-selected simulations differed from zero, so the analysis found no evidence of bias in these estimates.The estimates concern parameter m, for which negative values are expected in systems approaching a bifurcation and values near zero in stable systems.Source: 902. Methods and Results · p. 6

1513. Discussion

Historical transitions may reflect either a changing environment or chance, but chance-selected systems can look statistically different despite identical stochastic forcing. The discussion explains how chance excursions create apparent warning signals, why truncating observations can bias analysis, and why model-based estimates and replicated experiments offer alternatives.

1513. Discussion

  • Chance-selected systems can look statistically different from counterparts driven by the same stochastic process, making historical transitions difficult to classify as environmental change or chance.The discussion notes that historical observations often do not reveal which category an observed large change belongs to.Source: 1513. Discussion · pp. 6–7

1693.1. Chance transitions are false positives for early warning signals

  • Chance excursions can raise variance and autocorrelation, creating apparent warning signals without an approaching critical transition.An excursion that occupies much of a moving window can create these patterns; the autocorrelation can arise from rapid movement away from a stable state rather than critical slowing down.Source: 1693.1. Chance transitions are false… · pp. 7–8

3.2. The truncation of observations

  • Without an underlying process model, the observation cutoff is arbitrary, and choosing a termination rule can produce the bias under discussion.The deterministic equilibrium is unknown and may move as a transition approaches, so truncating at the potential minimum is not practical.Source: §3.2 · p. 8

2223.3. Comparing to the model-based method

  • In the numerical experiment, model-based estimates matched across conditionally selected replicates and random samples, appearing more robust than summary statistics but not immune to bias.The approach requires a more rigid pattern matching its saddle-node approximation, rather than any increase in variance or autocorrelation.Source: 2223.3. Comparing to the model-based… · p. 9

2323.4. Importance of experimental approaches

  • Experimental replication avoids the Prosecutor’s fallacy by generating a complete sample of replicates instead of selecting cases from a larger historical sample.The discussion frames replication as a way to address the problem of having only a single realization.Source: 2323.4. Importance of experimental… · p. 9

Science (New York, N.Y.) 311 (5767), 1557–8. 250

Chance-driven exits from stable states can produce patterns also seen during genuine approaches to bifurcation. The figures show how selecting only systems that collapsed by chance creates positive warning-statistic patterns.

Science (New York, N.Y.) 311 (5767), 1557–8. 250

  • Figure 1 illustrates how a chance excursion can carry a trajectory beyond the Allee threshold, while genuine transitions involve movement toward a bifurcation.In the potential-energy picture, wells correspond to stable points and peaks to unstable points.Source: Figure 1 · p. 13
  • Conditionally selected chance collapses show positive-τ patterns in variance and autocorrelation, and a similar bias appears in the second model.Positive τ corresponds to an indicator increasing with time, a pattern typically taken as evidence of an approaching transition.Source: Figure 2 · p. 14, Figure 3 · p. 15
  • Figure 2 contrasts τ distributions for collapsed-by-chance replicates (grey bars) with estimates from a random, unselected sample (solid lines).The positive-τ pattern in the selected simulations arises from conditional selection, not an approaching transition.Source: Figure 2 · p. 14
  • In Figure 3, the same analysis in a second model shows similar statistical bias, particularly toward positive τ.The analysis uses the model in Eq. (4).Source: Figure 3 · p. 15
Loading 1210.1204v1…