A framework for automated anomaly detection in high frequency water-quality data from in situ sensors

1. Introduction

• Technical sensor errors can create anomalies that confound the assessment or identification of real water-quality changes.The study defines anomalies as observations affected by sensor-equipment technical errors rather than ecological patterns and processes.Source: §1 · pp. 2–3
• Near-real-time, high-volume data make manual checking, labelling, and correction of every observation unfeasible, motivating automated anomaly detection.The authors identify automated detection as necessary before sensor data can be used with confidence for visualization, analysis, and reporting.Source: §1 · p. 2
• The authors develop a ten-step framework to implement and compare anomaly-detection methods for high-frequency in situ sensor data.They demonstrate it with turbidity, conductivity, and river-level data from rivers flowing into the Great Barrier Reef lagoon in northeast Australia.Source: §1 · p. 2
• Turbidity and conductivity are the detection targets because they are typically more temporally stable than dissolved oxygen and temperature and were the case-study end-user’s target variables.The framework treats anomalies as technical errors, not real ecological patterns or processes.Source: §1 · p. 3

2.1 Identify end-user needs and goals (Step 1)

• End-user needs and goals guide which anomaly types matter most and, in turn, which detection methods are suitable.The framework describes these components among its steps for developing and comparing anomaly-detection methods.Source: §2.1 · p. 3, §2 · p. 3
• The agency’s foremost short-term need was near-real-time graphical visualization of streaming sensor data, with potential anomalies identified and labelled.The longer-term goals included complete quality control and assurance, possible automated correction, and using corrected data to estimate sediment and nutrient concentrations in near-real time.Source: §2.1 · p. 3
• Other end-users may prioritize online, real-time warnings of water-quality breaches associated with real events rather than technical anomalies.The passage notes possible economic and public-health consequences for commercial and recreational sites.Source: §2.1 · p. 3

2.2 Identify data characteristics (Step 2)

• The study focuses on Pioneer River and Sandy Creek, in a tropical region with wet-season event flows and a drier season with low or zero river flows.Seasonal rainfall and temperature patterns affect water-quality patterns through time.Source: §2.2 · p. 3
• Sensors recorded turbidity and conductivity approximately hourly or every hour and a half, while river level was measured every 10 minutes.River-level observations were linearly interpolated to match the turbidity and conductivity measurements; level was examined for context, not anomaly detection.Source: §2.2 · p. 4
• Turbidity tends to rise rapidly during runoff and may rise gradually during low flow, while conductivity can increase in low-flow periods and decrease with fresh-water inputs.Measurements may be more frequent during event flows, and relationships among river level, turbidity, and conductivity are complex.Source: §2.2 · p. 4
• The sensor time series covered 12 March 2017 to 12 March 2018, with 6,280 observations at Pioneer River and 5,402 at Sandy Creek.The data included turbidity, conductivity, and time-matched river level.Source: §2.2 · p. 4

2.3 Define anomalies and their types (Step 3)

• An anomaly is a water-quality datum or set of data caused by technical error in the in situ sensor equipment.A rare high turbidity value during a real high-flow event can be valid, whereas a similarly high value beyond the sensor’s detection range can be anomalous.Source: §2.3 · p. 4
• The end-user consultation identified sudden spikes, low or high variability, constant offsets, sudden shifts, impossible or out-of-range values, drift, spike clusters, missing values, and other untrustworthy data.The study also labels the first observation after more than 180 minutes without observations as a missing-value anomaly.Source: §2.3 · pp. 4–5
• Class 1 covers sudden-change anomalies; Class 2 covers types suited to simple hard-coded rules; Class 3 may require post hoc user confirmation alongside automated detection.Class 1 includes types A, D, I, and J; Class 2 includes F, G, and K; Class 3 includes B, C, E, H, and L.Source: §2.3 · p. 5
• Experts and the end-user visually identified and labelled potential anomalies and their types for implementing and assessing detection methods.Labels drew on knowledge of riverine water-quality dynamics and the specific sites and watersheds.Source: §2.3 · p. 5

2.4 Rank anomaly types by importance (Step 4)

• The ranking reflects each anomaly type’s potential impact if undetected in light of end-user needs, shaping which methods count as most suitable.A method detecting the same number of anomalies may be less suitable if it catches lower-impact types instead of the end-user’s priority anomalies.Source: §2.4 · p. 5
• Large sudden spikes ranked first; shifts, calibration offsets, and changes in variance followed, with shifts and calibration offsets tied for third.Source: §2.4 · p. 5

2.5 Select suitable methods of anomaly detection (Step 5)

• Methods were selected for near-real-time, nonstationary data streams, and physical-process models and dynamic Bayesian networks or hidden Markov models were not pursued.The authors cite complexity and transferability challenges for physical-process models and limited water-quality software for the latter methods.Source: §2.5 · p. 6

2.5.1 Automated classification rules

• Hard-coded rules flag missing-data gaps over 180 minutes, negative or zero turbidity and conductivity, and values outside sensor-specific ranges.The missing-data threshold depends on end-user needs and sensor sampling frequency.Source: §2.5.1 · p. 6

2.5.2 Regression-based methods

• Regression methods forecast one step ahead and flag observations outside a prediction interval, using past observations and, in RegARIMA, other water-quality variables as covariates.ARIMA components can include differencing for nonstationary series; the methods also include naïve prediction, linear autoregression, and an anomaly-mitigation variant that substitutes forecasts for detected anomalies.Source: §2.5.2 · pp. 7–8
• The general regression model combines covariates with an ARIMA error structure, and its prediction interval uses a t quantile and the training residual standard deviation.The interval was set to 99% to limit false-positive probability to 1% given relatively few labelled anomalies.Source: §2.5.2 · p. 7

2.5.3 Feature-based methods

• Feature-based methods transform turbidity and conductivity series to expose anomalies, then apply unsupervised detection in a multivariate feature space.HDoutliers uses normalized series, representative cluster members, nearest-neighbour distances, and an extreme-value threshold; kNN-agg aggregates weighted distances across neighbour counts, while kNN-sum sums distances, using k = 10.Source: §2.5.3 · p. 9

2.6 Select metrics to evaluate and compare methods (Step 6)

• Methods were compared using accuracy, error rate, negative and positive predictive values, and regression-based root mean square error.The predictive values were included to better capture performance when anomalous and normal observations are unbalanced.Source: §2.6 · p. 9
• Feature methods are expected to take less time than regression methods in batch runs, but near-real-time feature classification may make their computation times more comparable.HDoutliers considers one nearest neighbour, whereas the kNN methods consider all k neighbours; regression methods may also require training and parameter optimization.Source: §2.6 · pp. 9–10

2.7 Prepare data for anomaly detection (Step 7)

• Regression training used clean data with labelled Class 1 and 3 anomalies removed, while testing used the original series after natural-log transformation.Linear autoregression instead used differences of the natural logarithms; zero or negative values were replaced with the previous observation to permit forecasting.Source: §2.7 · p. 10
• Feature-based detection used one-sided and two-sided derivatives of log-transformed turbidity and conductivity to highlight priority sudden spikes.The one-sided transformation used the negative derivative for turbidity and the positive derivative for conductivity.Source: §2.7 · p. 10

• Regression-based methods were implemented with the forecast package in R.Source: §2.8 · p. 10
• Feature-based methods were implemented with DDoutliers through the oddwater package in R.Source: §2.8 · p. 10
• The same rule-based code was used for automated classification within regression- and feature-based methods.Source: §2.8 · p. 10

3.1 Anomalies and their types

• The data contained 1651 anomalous turbidity observations, 521 conductivity observations, and 8 level observations, mainly types E, H, and L.Anomaly counts were imbalanced across classes and types; some types comprised multiple observations while others had only one.Source: §3.1 · p. 11
• No examples of types C, G, or clusters of spikes (I) were present, while missing data (K) occurred in every time series.Impossible values (F) were numerous and can be detected by rules in the same way as out-of-sensor-range values.Source: §3.1 · p. 11

3.2.1 Automated classification rules

• Rule-based classification correctly detected every Class 2 anomaly (types F, G, and K) with no false positives in each time series.Source: §3.2.1 · p. 11

3.2.2 Regression-based methods

• Regression-based methods generally had accuracy above 0.80 and error rates below 0.20, except when ADAM was used.ADAM could detect Class 3 anomalies missed by detection alone, but its false positives were high; naïve prediction with ADAM produced over 5000 false positives for PR turbidity, versus 133 without mitigation.Source: §3.2.2 · p. 11
• Adding water-quality covariates did not improve RegARIMA over ARIMA and reduced conductivity classification at PR, with PPV 0.49 versus 0.93.The authors link this result to complex relationships among conductivity, turbidity, and level at the site.Source: §3.2.2 · p. 12
• Regression residual diagnostics indicated heteroscedasticity, which may have reduced prediction-interval accuracy and affected anomaly classification.Source: §3.2.2 · p. 12

3.2.3 Feature-based methods

• Feature-based methods had high false negative rates, especially for Class 3 anomalies; no method detected any of the 718 type L anomalies in PR turbidity.For PR conductivity, accuracy was 0.92 and error rate was 0.08, but false negatives remained high; at SC, all methods had over 900 false negatives for turbidity.Source: §3.2.3 · pp. 12–13

• ARIMA without mitigation may be most useful when end-users prioritize detecting sudden spikes and shifts, which can distort summary statistics used in water-quality assessments and compliance checking.Source: §4 · p. 13
• ARIMA with mitigation could be used to detect Class 3 anomalies such as drift and periods of high variability, which can indicate a need for sensor recalibration.Source: §4 · p. 13
• Regression models assume evenly spaced observations, and training data should cover the full range of natural water-quality dynamics.Less frequent or irregular measurements may be problematic, particularly if natural events are missed.Source: §4 · p. 13
• Regression-based methods are strongly influenced by training data, and the one-year case study used the same data, minus anomalies, for training and testing, raising possible over-fitting.Training on another nearby site or a different period at the same site could lessen this issue, but such nearby sensor data were unavailable.Source: §4 · p. 13
• Because water-quality patterns can change with river flow regimes, segmenting time series into regimes may improve regression-based anomaly detection.The authors suggest hidden Markov models as one possible way to segment regimes before applying regression models.Source: §4 · pp. 13–14
• Feature-based methods were not proficient at detecting Class 3 anomalies because their transformations and algorithms were designed to prioritize Class 1 anomalies.Other transformations may be needed to target Class 3 anomalies in different applications.Source: §4 · p. 14

5. Conclusions

• Combining methods is likely to improve correct anomaly classification while minimizing false detections in streaming water-quality data.The framework also emphasizes communication between end-users and anomaly-detection developers for outcomes suited to both detection performance and end-user application.Source: §5 · p. 14
• The framework is presented as transferable to other high-frequency time series and anomaly-detection applications.Within water-quality monitoring, the authors suggest applying it to dissolved oxygen, water temperature, and nitrate.Source: §5 · p. 14
• Applying anomaly detection to more dynamic water-quality properties may make it harder to distinguish anomalies from real events.The authors contrast dissolved oxygen, water temperature, and nitrate with the less dynamic turbidity and conductivity studied here.Source: §5 · p. 14

Supplementary materials

• Supplementary figures show bivariate relationships and time series for Pioneer River and Sandy Creek, including end-user-identified anomalies and maintenance events.The time-series figures distinguish non-anomalous and potentially anomalous observations and mark probe swaps and other maintenance activities.Source: Supplementary materials · p. 34, Figure S1 · p. 35, Figure S2 · p. 36
• Supplementary classification plots show results for regression methods and feature methods across the two sites and water-quality variables.The supplement also provides diagnostic plots and partial autocorrelation plots for regression-based methods.Source: Supplementary materials · p. 34, Figure S4 · p. 42, Figure S5 · p. 43
• Separate files provide the time-series data, anomaly-type coding, and R code used to implement regression methods and calculate performance metrics.The listed scripts include site-specific files and files for naïve prediction, prediction, and performance assessment.Source: Supplementary materials · p. 34
• The supplement includes a table of anomaly-detection method characteristics and tables describing classification and performance metrics.It also lists methods spanning rule-based, regression, uncertainty-fusion, and feature-based approaches.Source: Table S1 · p. 38, Supplementary materials · p. 38, Table S2 · p. 39
• Additional performance tables report regression-based results for Sandy Creek conductivity and feature-based results for multivariate Sandy Creek conductivity data.The regression table incorporates automated classification rules detecting all Class 2 anomalies.Source: Table S4 · p. 41, Table S5 · p. 41