PaperlayerView exact revision on arXiv

Source-linked AI summary

Sequential Anomaly Detection using Inverse Reinforcement Learning

Min-hwan Oh, Garud Iyengar

arXiv:2004.10398v1cs.LGstat.ML

TL;DR

Sequential anomaly detection lacks reliable labels and is especially challenging when abnormal behavior is intentional or goal-directed. The paper uses Bayesian IRL with neural reward functions to model agents’ normal behavior and uncertainty, and reports effective anomaly identification on public real-world data. Its scope includes sequential trajectories and settings where training data may contain some anomalies.

  • Problem

    Sequential anomaly detection is difficult because anomalies are poorly characterized, often unlabeled, and may be intentional or goal-directed across time.

    Source: §1 · p. 1

  • Method

    The method infers neural reward functions from agent trajectories using Bayesian IRL, then scores new observations by learned normality and predictive uncertainty.

    Source: §4 · p. 4

  • Results

    The proposed IRL-AD and IRL-ADU methods outperform baseline methods on the GeoLife GPS and TST datasets using F1, recall, and precision.

    Source: §5 · p. 6, §5.4 · p. 8

  • Takeaways & Limitations

    The results support IRL as a viable framework for sequential anomaly detection and provide a scalable way to model uncertainty over reward functions.

    Source: §6 · p. 8

    Maximum-entropy IRL can become intractable in large or continuous domains because policy and state-visitation computations scale exponentially with state-space dimensionality.

    Source: §3.2 · p. 3

Abstract

from arXiv · show

One of the most interesting application scenarios in anomaly detection is when sequential data are targeted. For example, in a safety-critical environment, it is crucial to have an automatic detection system to screen the streaming data gathered by monitoring sensors and to report abnormal observations if detected in real-time. Oftentimes, stakes are much higher when these potential anomalies are intentional or goal-oriented. We propose an end-to-end framework for sequential anomaly detection using inverse reinforcement learning (IRL), whose objective is to determine the decision-making agent's underlying function which triggers his/her behavior. The proposed method takes the sequence of actions of a target agent (and possibly other meta information) as input. The agent's normal behavior is then understood by the reward function which is inferred via IRL. We use a neural network to represent a reward function. Using a learned reward function, we evaluate whether a new observation from the target agent follows a normal pattern. In order to construct a reliable anomaly detection method and take into consideration the confidence of the predicted anomaly score, we adopt a Bayesian approach for IRL. The empirical study on publicly available real-world data shows that our proposed method is effective in identifying anomalies.

1 INTRODUCTION

Sequential anomaly detection is difficult because anomalies are rarely labeled and may reflect intentional, goal-oriented behavior across multiple time periods. The paper proposes Bayesian IRL to infer normal behavior from agent trajectories and detect deviations with uncertainty-aware scores.

Motivation

  • Anomaly detection is challenging because anomalies are insufficiently characterized and usually have limited labels for discriminative classification.Consequently, common approaches learn a generative model of normal patterns and flag poorly explained observations.Source: §1 · p. 1
  • Sequential anomalies require modeling continuity across multiple time periods and potentially multiple dimensions, making them more complex than snapshot anomalies.Source: §1 · p. 1

Contribution

  • Bayesian IRL infers an agent’s reward function from behavior and uses the learned function to assess whether new observations follow normal patterns.The method represents rewards with a neural network and incorporates posterior uncertainty into anomaly scoring.Source: §1 · pp. 1–2

Method

  • The framework accepts trajectories with coordinates and optional metadata such as date, time, and landmark features.Source: §1 · pp. 1–2

Contribution

  • Model uncertainty can accommodate some anomalous trajectories during training and route highly uncertain cases for human validation.Source: §1 · p. 2

Empirical study

  • The empirical study on publicly available real-world data reports that the proposed method is effective at identifying anomalies.Source: §1 · p. 2

2 RELATED WORK

Prior sequential anomaly detectors use trajectory modeling, clustering, reconstruction, density, or deep learning, but often require engineered features, batch processing, fixed-length inputs, or predefined rewards. The paper targets these limitations with an IRL-based approach for intentional or goal-directed behavior.

Trajectory methods

  • Many sequential anomaly methods require feature engineering or preprocessing, limiting end-to-end analysis of trajectory data.Source: §2 · pp. 2–3

Clustering methods

  • Several clustering-based methods are batch-oriented and cannot be used for real-time online detection.Source: §2 · p. 2

Deep learning methods

  • Deep reconstruction approaches often evaluate only univariate or highly regular time series and typically require fixed-length sequences or windows.Source: §2 · pp. 2–3

Goal-directed anomalies

  • Most reviewed methods do not directly address agents whose anomalous behavior may be intentional or goal-directed.One related method uses inexplicability scores but still requires feature engineering and is limited online.Source: §2 · p. 3

Reinforcement learning

  • Forward reinforcement-learning approaches require predefined reward signals, whereas this work infers reward functions when they are unknown.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries formulate forward and inverse reinforcement learning in an MDP, then introduce maximum-entropy and Bayesian IRL. These foundations motivate a sample-based Bayesian extension that estimates uncertainty over neural reward functions without requiring known dynamics.

3.1 Forward and Inverse Reinforcement Learning (RL) Basics

  • In an MDP, forward RL learns an optimal policy maximizing discounted rewards, while IRL learns a reward function explaining demonstrated behavior.Demonstrations are trajectories of state-action pairs generated by an optimal or near-optimal policy, with variable horizon lengths.Source: §3.1 · p. 3

3.2 Maximum Entropy IRL

  • Maximum-entropy IRL assigns exponentially higher likelihood to higher-reward trajectories and learns reward parameters by maximizing the likelihood of demonstrations.Source: §3.2 · p. 3
  • Maximum-entropy IRL becomes computationally difficult in large or continuous domains because it repeatedly computes soft policies and state-visitation frequencies.Sample-based variants avoid known dynamics by generating background trajectories, while adapting the sampling distribution through policy optimization.Source: §3.2 · p. 3

3.3 Bayesian framework for IRL

  • Bayesian IRL combines a reward prior with a behavior-compatibility likelihood to produce a posterior distribution over reward functions.The reward can be inferred through posterior-mean estimation using MCMC, although posterior means may yield policies inconsistent with demonstrations.Source: §3.3 · pp. 3–4, MAP Inference · p. 4

MAP Inference.

  • Both earlier Bayesian and MAP IRL formulations are limited to tabular settings without function approximation.The proposed method therefore extends sample-based IRL to combine reward-function uncertainty with function approximation.Source: MAP Inference · p. 4

4 METHOD

The method approximates a Bayesian distribution over neural-network reward functions using bootstrap sampling, then alternates reward and policy updates to score sequential observations and incorporate uncertainty into anomaly decisions.

4 METHOD

  • The method extends sample-based maximum entropy IRL to a Bayesian framework that approximates a distribution over neural-network reward functions.It uses a prior over reward parameters and samples reward functions during training.Source: §4 · p. 4, §4.1 · p. 4

4.1.1 Bootstrap ensemble.

  • Bootstrap ensembles train K independently bootstrapped models to approximate function uncertainty, but their computational cost grows linearly with K.Each model uses a different resampled dataset; this approach scales to high-dimensional spaces but requires multiple base models.Source: §4.1.1 · p. 4, Figure 1 · p. 4
  • A shared network with K independently trained output heads reduces computation by sharing representations, while lowering diversity between heads.The heads can be trained in a single forward/backward pass, and one head is sampled at each iteration.Source: §4.1.1 · p. 5, Figure 2 · p. 5

4.2 Parameter Update

  • Reward updates combine demonstrated target trajectories with policy-generated background trajectories, reweighting biased samples through importance sampling.The algorithm alternates reward optimization with policy optimization using TRPO or another policy-gradient method.Source: §4.2 · p. 5

4.3 Algorithm Overview

  • Training alternates between sampling a reward head, generating trajectories with the current policy, updating the reward, and improving the policy until convergence.The procedure returns optimized reward parameters and a trajectory-generating policy, focusing here on a single target agent while permitting group-level extensions.Source: §4.3 · pp. 5–6

4.4 Normality Score

  • The normality score standardizes the mean reward across all output heads, and low scores indicate observations or trajectories that deviate from normal behavior.The normalization accounts for differences in reward variability across target agents and supports application-specific thresholds.Source: §4.4 · p. 6

4.5 Incorporating Model Uncertainty when Deciding an Anomaly

  • Model uncertainty can make anomaly decisions conservative by requiring both a low normality score and sufficiently low predictive variance when false alarms are costly.A second threshold on reward variance controls how certain the model must be before assigning an anomaly.Source: §4.5 · p. 6

5 EXPERIMENTS

The experiments evaluate IRL-based sequential anomaly detection on GeoLife GPS and taxi trajectories against classical and autoencoder baselines, using anomaly definitions tailored to each dataset. Both proposed methods outperform the baselines overall, while uncertainty improves precision and the non-uncertainty variant slightly improves recall.

5.2 Comparison with Other Methods

  • The study compares IRL-AD and uncertainty-aware IRL-ADU with LOF, One-Class SVM, feed-forward autoencoders, and LSTM autoencoders using F1-score, recall, and precision.Higher values on all three metrics indicate more accurate anomaly detection.Source: §5 · p. 6, §5.2 · p. 7, §5.3 · p. 7

5.1 Datasets

  • Publicly available GPS trajectory data lack labeled anomalies, so the study defines anomalies through injected, hand-labeled, or behavior-based constructions.The authors note that the closest labeled driving dataset has only six drivers and two trajectories per category for each driver.Source: §5.1 · p. 6
  • GeoLife uses separate reward functions for 10 selected users, with injected or hand-labeled anomalous trajectories evaluated alongside normal trajectories.The dataset contains more than 17,000 trajectories from 182 users, and trajectories vary substantially in length and sampling rate.Source: §5.1 · p. 7
  • TST evaluation learns one reward function across 442 taxi drivers and defines cropped long trips as anomalies requiring discrimination from their earlier segments.Long trips are defined by cruise time greater than 50 minutes and cropped to prevent trivial detection from trip length.Source: §5.1 · p. 7
  • The experiments use states containing current and initial locations plus elapsed time, and actions represented by two-dimensional velocity.The Bayesian model can tolerate anomalous trajectories in training data and update its reward function as online data arrive.Source: Data Preparation · p. 7

5.4 Results

  • Sample GeoLife scores identify a normal trajectory, localize a hand-labeled detour anomaly, and assign persistently low scores to an injected trajectory from another user.Scores return to a normal range on the normal return path after the detour, supporting detection of anomaly onset and offset in streams.Source: §5.4 · p. 7, Figure 3 · p. 8
  • Both IRL-AD and IRL-ADU outperform the baseline methods across the GeoLife GPS and TST datasets.IRL-ADU achieves higher overall precision, whereas IRL-AD achieves slightly higher recall.Source: §5.4 · p. 8

6 DISCUSSION

The discussion presents IRL as a natural framework for sequential anomaly detection and highlights a scalable Bayesian extension for approximating reward-function uncertainty.

6 DISCUSSION

  • The paper presents IRL-based sequential anomaly detection as a viable bridge between reinforcement learning and practical anomaly detection.It describes this as the first use of IRL for sequential anomaly detection.Source: §6 · p. 8
  • A simple scalable extension approximates the posterior distribution of reward functions at low additional computational cost.The authors suggest the extension may also be useful for general IRL problems.Source: §6 · p. 8

A NEURAL NETWORK DETAILS

The appendix specifies the neural architectures used for autoencoder baselines, reward modeling, and policy optimization.

A NEURAL NETWORK DETAILS

  • FNN-AE and LSTM-AE provide feed-forward and recurrent autoencoder baselines, while the reward network uses two hidden layers and K = 10 output heads.The reward network has hidden dimensions 64 and 16, and policy optimization uses TRPO.Source: A NEURAL NETWORK DETAILS · p. 10

B PER USER EXPERIMENTS ON GEOLIFE-GPS

The per-user GeoLife experiments select users with many trajectories, filter short trajectories, and evaluate controlled anomaly proportions.

B PER USER EXPERIMENTS ON GEOLIFE-GPS

  • The GeoLife per-user experiments select 10 users with the largest trajectory counts, remove trajectories shorter than 100 time-steps, and set test anomaly proportions to 5% and 10%.Anomalies are injected or identified independently for each experiment, with results reported in Tables 2 and 3.Source: B PER USER EXPERIMENTS ON GEOLIFE-GPS · p. 10

C BAYESIAN NEURAL NETWORK

Bayesian neural networks represent network weights as random variables and infer a posterior rather than optimizing fixed parameters. This enables predictive-uncertainty estimation, although exact posterior computation is difficult and common approximations can be computationally expensive.

C BAYESIAN NEURAL NETWORK

  • Bayesian neural networks place a prior distribution over random network weights and infer their posterior from data.This contrasts with treating weights as fixed parameters optimized during training.Source: C BAYESIAN NEURAL NETWORK · p. 10
  • Exact posterior learning requires integration over the full parameter space, and a closed-form solution often does not exist.Laplace approximation and Hamiltonian Monte Carlo are cited as alternatives, with the latter described as computationally prohibitive.Source: C BAYESIAN NEURAL NETWORK · p. 10
  • Variational inference approximates the posterior with a tractable distribution qη(θ), optimizing closeness to p(θ|D) through KL divergence.The variational distribution is selected from a predetermined family of distributions.Source: C BAYESIAN NEURAL NETWORK · p. 10
  • Monte Carlo dropout estimates predictive uncertainty through test-time dropout and has an approximate Bayesian interpretation as variational inference.It is relatively simple to implement and can also be interpreted as averaging predictions from an ensemble of neural networks.Source: C BAYESIAN NEURAL NETWORK · p. 10
  • The authors report that dropout did not produce satisfying confidence intervals for their crowd-counting problem, motivating a non-parametric bootstrap of functions.This limitation is specific to the reported crowd-counting setting.Source: C BAYESIAN NEURAL NETWORK · p. 10

Related papers

Loading 2004.10398v1…