Monitoring Count Time Series in R: Aberration Detection in Public Health Surveillance

Introduction

• Timely detection of aberrant counts can support early intervention to prevent additional disease cases in human and animal health.Aggregated case reports can be monitored as count time series.Source: Introduction · pp. 1–2
• The surveillance package implements a range of aberration-detection methods for count and proportion time series, supporting objective and reproducible analysis.The authors identify ready-to-use, understandable implementations and routine automation as important to epidemiologists.Source: Introduction · p. 2
• The open-source package combines prospective outbreak-detection methods with visualization and is reported in use at several European public-health institutions.The paper focuses on the surveillance system at Germany’s RKI.Source: Introduction · p. 2
• This paper extends two earlier articles on the package by reviewing added features and illustrating support for routine disease surveillance at the RKI.Source: Introduction · p. 2

1.1. How to store time series and related information

• The sts S4 class represents univariate or multivariate count series, with each monitored entity and time point represented in its data structure.The example uses weekly Salmonella Newport counts from Germany’s 16 federal states over 525 weeks.Source: §1 · p. 2, §1.1 · p. 3

Slots of the class sts

• The observed slot stores counts in an n × m matrix, while epoch, epochAsDate, freq, and start encode the time index and aggregation period.Additional slots record known aberrations, population, geography, and monitoring results.Source: Slots of the class sts · p. 3

Creation of an object of class sts

• A sts object can be created with new or converted from ts; conversion through ts also connects to other time-series classes such as zoo and xts.Source: Creation of an object of class sts · pp. 3–4

Basic manipulation of objects of the class sts

• Subsetting and aggregation can select units or periods, while plots show either the national series over time or separate panels for each unit.The weekly Salmonella Newport series is shown for Germany and, in a separate figure, for Bavaria and Berlin.Source: Figure 1 · p. 4, Basic manipulation of objects of the… · p. 5, Figure 2 · p. 5

Statistical framework for aberration detection

• Monitoring distinguishes in-control from out-of-control counts using past observations and a statistic compared with a threshold to determine alarm time.Source: Statistical framework for aberration… · pp. 5–6

1.2. How to use aberration detection algorithms

• A monitoring call specifies the algorithm, time range, and algorithm-specific parameters; thresholds, alarms, and control arguments are stored in the sts object.Source: Aberration detection in the package · p. 6, Statistical framework for aberration… · p. 6

Example with the EARS C1 method

• EARS C1 estimates the expected count from the preceding seven time points and raises an alarm when the observed count exceeds its upper confidence bound.The result is returned in an sts object with upperbound and alarm slots populated for the monitored range.Source: Example with the EARS C1 method · p. 6
• After the known 2011 sprout outbreak, EARS C1’s upperbound is elevated enough that a smaller outbreak would not be detected.The four final alarms correspond to the known outbreak; EARS methods use only the very recent past and are less suited to series with important trend or seasonality.Source: Example with the EARS C1 method · p. 7

2. Using surveillance in selected contexts

• The package’s selected methods include Farrington and Bayesian one-timepoint detectors, a negative-binomial CUSUM for sustained shifts, and a cumulative-sum method for categorical data.One-timepoint methods alarm when the monitored count exceeds a threshold; CUSUM methods accumulate evidence across timepoints.Source: §2 · pp. 7–8

2.1. One size fits them all for count data

• farringtonFlexible supports the original Farrington method and Noufaily et al.’s improvement, which aims to improve specificity without reducing sensitivity.The package also retains the original method as the farrington function.Source: §2.1 · p. 8
• The improved baseline can use more historical data while accounting for seasonality through a zero-order periodic spline represented as a noPeriods-level factor.The implementation lets users choose the number of periods per year; noPeriods = 1 recovers the original reference-window-only model.Source: §2.1 · pp. 8–9
• The default pastWeeksNotIncluded value is 26, excluding recent observations from the baseline to avoid reducing sensitivity when an outbreak has recently started.The paper reports this exclusion as an improvement described by Noufaily et al. (2012).Source: §2.1 · p. 9
• The delta threshold assumes normality; the negative-binomial plug-in threshold ignores estimation uncertainty, while muan is described as more statistically correct but can yield very large bounds.The negative-binomial threshold uses plug-in estimates of the mean and overdispersion; muan disregards uncertainty in the overdispersion estimate.Source: §2.1 · pp. 9–10
• Farrington optionally transforms counts, includes a time trend only when significant and conditions are met, then reweights past outbreaks using Anscombe residuals before raising an alarm when the count exceeds its upperbound.The trend requires more than three years of reference data and no overextrapolation; the reweighting threshold is configurable.Source: §2.1 · p. 10
• The RKI system already uses the improved method, which the authors hope will eventually replace the original in routine use.The original method is described as widely used in public health surveillance.Source: §2.1 · p. 10

Similar methods in the package

• The improved Farrington method produces fewer alarms while retaining the summer outbreak; on the Campylobacter series, the Bayes procedure adjusts for seasonality poorly and may produce many false alarms.The improved Farrington procedure acts similarly to the original while its reweighting and trend inclusion produce fewer alarms.Source: Figure 5 · p. 11, §2.2 · p. 14
• The package also offers bayes, rki, and cdc detectors based on historical data, but these lack linear trends, down-weighting of past outbreaks, and power transformation; the authors recommend farringtonFlexible.bayes uses a conjugate prior-posterior approach with a negative binomial distribution; rki uses a normal or Poisson distribution, and cdc aggregates weekly data into four-week counts.Source: Similar methods in the package · pp. 10–11

2.2. A Bayesian refinement

• boda extends the GLM approach to a Bayesian GAM with penalized splines, adjusting for previous outbreaks and concurrent influences while carrying parameter uncertainty into predictive distributions.The paper uses INLA for fast approximate Bayesian inference.Source: §2.2 · p. 11
• boda monitoring is substantially slower than Farrington, and detailed regression modelling is meaningful only when external influences are known and information about them is available.The paper describes its typical institutional use as detailed analysis of a few selected series, such as critical series or those with known trends.Source: §2.2 · p. 12
• In the Campylobacter series, case counts are strongly associated with humidity, while the 2011 increase is attributed to increased awareness and testing rather than presumed increased disease activity.The reporting system records positive tests but not the number of tests performed, limiting adjustment for testing volume.Source: §2.2 · p. 12
• For each monitored timepoint, boda estimates parameters from earlier data and flags an alarm when the observed count exceeds the chosen quantile of its posterior predictive distribution.The model includes a time-varying intercept, periodic spline, possible linear trend, previous-outbreak effect, and time-varying covariates.Source: §2.2 · p. 13
• The covariate example includes absolute-humidity lags and year-turn indicators to account for reporting behavior, plus an indicator for the 2011 O104:H4 awareness period.No correction for past outbreaks is made in this covariate model.Source: §2.2 · p. 13

2.3. Beyond one-timepoint detection

• glrnb combines traditional surveillance modelling with cumulative-sum monitoring to detect sustained shifts that one-timepoint methods may miss.A sequence of slightly elevated counts can characterize an outbreak even when no single weekly count is flagged.Source: §2.3 · p. 14

Definition of the control chart

• The in-control mean uses a Poisson or negative-binomial GLM with a log link, selected according to overdispersion, and can include harmonic seasonality and flexible predictors.Splines, concurrent covariates, or an offset can be added; the model can also resemble the Farrington approach with reweighting of past outbreaks and trend selection.Source: Definition of the control chart · p. 15
• The out-of-control mean represents a change either as a multiplicative shift or as an autoregressive component based on the preceding count.The multiplicative shift can be supplied or reestimated; the autoregressive alternative has an unknown positive parameter.Source: Definition of the control chart · p. 15
• The in-control model is fitted on phase 1 data, while phase 2 is monitored using a likelihood-ratio rule when the autoregressive parameter is fixed.When that parameter is unknown, a generalized likelihood ratio estimates it at each time point; this avoids specifying the change size but is more computationally intensive.Source: Definition of the control chart · pp. 15–16

Practical use

• Model selection for the in-control GLM is compulsory, and its fit should reflect the data under surveillance.Users can supply external predictions for the in-control means and overdispersion or use the built-in estimator with a selected harmonic count and optional trend.Source: Practical use · p. 16
• The example monitors Salmonella Newport counts in Germany during 2011 using a known multiplicative shift of 2 and a threshold chosen for a false-alarm probability below 0.1 within one year.The reference period is the years before 2011, and the model uses one seasonal harmonic and a trend.Source: Practical use · p. 16
• For large surveillance workloads, the authors suggest fitting each time series’ in-control model once yearly and using a reasonable predefined shift with LR, which is less computationally intensive than GLR.Historical-data length, buffer-period length, and shift size should be fixed for each application using simulations and/or expert discussion.Source: Practical use · p. 17

Similar methods in the package

• The package also provides glrPois, cusum, rogerson, and outbreakP for count monitoring; outbreakP targets changes from a constant level to a monotonically increasing incidence.The text notes that cusum and rogerson are discussed and compared with glrnb in prior work.Source: Similar methods in the package · p. 17

2.4. A method for monitoring categorical data

• categoricalCUSUM uses a multivariate GLM for the historical in-control distribution and a likelihood-ratio detector to compare current responses under in-control and changed distributions.It improves on earlier CUSUM approaches limited to binary data that do not handle overdispersion.Source: §2.4 · p. 18

Categorical CUSUM for Binomial Models

• GAMLSS supports beta-binomial modelling for overdispersed binary data and permits monitoring distribution parameters beyond the mean, including a time-trend parameter.The paper presents this as a more flexible alternative to binary GLMs, which do not handle inherent overdispersion well.Source: Categorical CUSUM for Binomial Models · pp. 18, 22
• The example assumes hospitalized counts follow a beta-binomial distribution and models the expected hospitalized count with a logit-link GAMLSS.The model includes an intercept, trend, two harmonic terms, and indicators for the first two weeks of each year.Source: Categorical CUSUM for Binomial Models · p. 19
• The in-control model accounts for seasonal and year-end variation and an increasing hospitalized proportion, reflecting the described reporting patterns.The passage notes lower hospitalized proportions in summer, higher proportions over December holidays, and a rising trend as fewer stool diagnoses are prescribed by GPs.Source: Categorical CUSUM for Binomial Models · p. 19
• The monitoring flagged the early-2014 peak as larger than expected; the authors suggest holiday weekday effects may explain the alarm.The example uses the preceding 2004–2012 data to estimate in-control parameters and monitors 2013 and early 2014.Source: Categorical CUSUM for Binomial Models · pp. 18, 21
• A threshold of h = 2 gives an estimated false-alarm probability below 0.1 over the 56 monitored time points, and the Markov Chain approximation is recommended because Monte Carlo took about 300 times longer.The two methods produced close results.Source: Categorical CUSUM for Binomial Models · p. 21

Categorical CUSUM for Multinomial Models

• The example tracks monthly rotavirus cases in Brandenburg across five age groups (2002–2013) to detect possible age shifts; because vaccine was recommended for routine vaccination in 2009, monitoring starts then.The multinomial logit model includes a trend and one harmonic wave, is fit to 2002–2008 data, and uses ages 0–4 as the reference category.Source: Categorical CUSUM for Multinomial Models · pp. 22–23
• A multiplicative shift of 7 in each age category’s odds against the reference category specifies the example’s out-of-control proportions.The in-control and out-of-control proportions are obtained by prediction from the respective models.Source: Categorical CUSUM for Multinomial Models · pp. 23–24
• The first aberration was detected in July 2009; using 10,000 Monte Carlo simulations, the estimated false-alarm probability with threshold h = 2 within the 60 phase-2 time points was 0.02.Source: Categorical CUSUM for Multinomial Models · p. 24
• The likelihood-ratio categorical CUSUM can use multivariate GLM models, but requires the direction of change to be specified in advance.Detection is optimal for the specified direction.Source: Categorical CUSUM for Multinomial Models · p. 24
• Multinomial proportions may be overdispersed, so the multinomial model can understate variation and yield a false-alarm probability that is too low.The authors recommend repeating the analysis with a Dirichlet-multinomial model, though differing parametrizations complicate specifying its out-of-control proportions.Source: Categorical CUSUM for Multinomial Models · p. 25
• The Dirichlet-multinomial CUSUM behaved similarly to the multinomial version despite specifying a slightly different shift type.The comparison is shown in Figure 13(b).Source: Categorical CUSUM for Multinomial Models · p. 25

Categorical data in routine surveillance

• Public-health categorical time series can track case distributions by sex, age group, or reporting region.Source: Categorical data in routine surveillance · p. 25
• Categorical CUSUM monitors a category distribution, whereas an alternative is to monitor each category’s time series separately.Source: Categorical data in routine surveillance · p. 26
• Distribution-level monitoring requires specifying the change parameter R in advance; an alternative for positive diagnostics is to use test counts as a population offset with farringtonFlexible.Source: Categorical data in routine surveillance · p. 26

Similar methods in the package

• For binary data, pairedbinCUSUM does not account for overdispersion as well as glrnb, while rogerson also supports binomial analysis.Source: Similar methods in the package · p. 26

2.5. Other algorithms implemented in the package

• Method selection depends partly on how much historical data is available and how much of that history an algorithm uses.EARS methods need only recent time points, whereas Farrington methods use data up to b years in the past; historical-reference methods use a subset of past data, while others use all included reference data.Source: §2.5 · pp. 26–27
• Choosing an algorithm also requires deciding whether to detect single-timepoint aberrations or more prolonged shifts.Source: §2.5 · p. 27
• The package’s algorithm comparisons support practical choices about introducing methods into routine use.The table provides an overview of algorithms implemented in surveillance.Source: §2.5 · p. 27, Table 1 · p. 27

3.1. A simple surveillance system

• A lightweight R-and-Sweave workflow runs farringtonFlexible on sts time series and reports observed counts, upper bounds, and alarms, but is not interactive.The example table presents reported S. Newport counts in two German states, with bold upper bounds marking alarms.Source: §3.1 · p. 28, Table 2 · p. 29

3.2. Automatic detection of outbreaks at the Robert Koch Institute

• At the RKI, surveillance underpins a daily system that analyzes about half a million time series and communicates results to epidemiologists.The system separates an R-based analysis process from a reporting component; pathogen-specific reports are sent weekly by email.Source: §3.2 · p. 29
• The RKI analysis queries database backends with RODBC, aggregates case reports with data.table, applies farringtonFlexible, and stores results in a separate SQL database.The authors report positive experience using R with surveillance for a larger production software component.Source: §3.2 · p. 29
• Having state-of-the-art detection methods already implemented let developers focus on workflow integration and efficient analysis at scale, with R-component developments shareable with other institutes.Source: §3.2 · p. 29

• surveillance has grown from a framework for developing and evaluating outbreak-detection algorithms into a tool for routine practice.Other package functions for modelling and incidence-case back-projection are documented elsewhere.Source: Discussion · p. 30
• Future areas of interest include better handling of multivariate and hierarchical data streams and adjustment for reporting delays.Source: Discussion · p. 30
• Overdispersion, low counts, and past outbreaks also occur in financial, occupational-safety, and environmental surveillance contexts.Source: Discussion · p. 30
• Related tools cover process control, structural-change detection, and epidemic modelling; SaTScan detects spatial, temporal, and space-time clusters, while package contributions are welcome.Source: Discussion · p. 30