PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-timescale Trajectory Prediction for Abnormal Human Activity Detection

Royston Rodrigues, Neha Bhargava, Rajbabu Velmurugan, Subhasis Chaudhuri

arXiv:1908.04321v1cs.CV

TL;DR

Fixed-timescale abnormal-activity methods may miss events occurring at other durations, while scene-dependent normality and incomplete training coverage complicate detection. This paper predicts past and future pose trajectories at multiple timescales and combines prediction errors to detect abnormal activity. The authors report that the framework captures anomalies of different durations and outperforms existing methods, alongside introducing a large annotated dataset.

  • Problem

    Fixed-timescale methods may not capture anomalies occurring at other durations, while scene-dependent normality and incomplete training coverage make abnormal activity difficult to characterize.

    Source: §I · p. 1

  • Method

    The framework predicts past and future pose trajectories at multiple timescales and combines their prediction errors to detect abnormal activity.

    Source: §I · p. 2

  • Results

    The authors report that the multi-timescale framework captures anomalies of different durations and outperforms existing methods.

    Source: §VI · p. 8

  • Takeaways & Limitations

    The paper introduces a challenging abnormal-activity dataset for research use, with diverse activities and single-person to group anomalies.

    Source: §I · p. 2

Abstract

from arXiv · show

A classical approach to abnormal activity detection is to learn a representation for normal activities from the training data and then use this learned representation to detect abnormal activities while testing. Typically, the methods based on this approach operate at a fixed timescale - either a single time-instant (eg. frame-based) or a constant time duration (eg. video-clip based). But human abnormal activities can take place at different timescales. For example, jumping is a short term anomaly and loitering is a long term anomaly in a surveillance scenario. A single and pre-defined timescale is not enough to capture the wide range of anomalies occurring with different time duration. In this paper, we propose a multi-timescale model to capture the temporal dynamics at different timescales. In particular, the proposed model makes future and past predictions at different timescales for a given input pose trajectory. The model is multi-layered where intermediate layers are responsible to generate predictions corresponding to different timescales. These predictions are combined to detect abnormal activities. In addition, we also introduce an abnormal activity data-set for research use that contains 4,83,566 annotated frames. Data-set will be made available at https://rodrigues-royston.github.io/Multi-timescale_Trajectory_Prediction/ Our experiments show that the proposed model can capture the anomalies of different time duration and outperforms existing methods.

I. INTRODUCTION

Abnormal activity detection is difficult because normality depends on scene context and training data may cover only subsets of normal and abnormal activities. The paper addresses the limits of fixed-timescale methods with bidirectional, multi-timescale pose prediction and introduces a large abnormal-activity dataset.

I. INTRODUCTION

  • Abnormality is difficult to define generically because it depends on scene context, and training may not include all normal or abnormal activities.Treating rare or unseen events as abnormal may also label unseen normal activities as abnormal.Source: §I · p. 1
  • Fixed-timescale methods may miss anomalies whose duration differs from the modeled timescale, such as loitering or jumping.Loitering can resemble normal walking at short timescales, while a larger timescale may be needed to detect it.Source: §I · p. 1
  • The framework predicts past and future pose trajectories at multiple timescales, with intermediate layers supervised to produce predictions at different durations.Prediction errors across directions and timescales are combined into an anomaly score at each input time instant.Source: §I · p. 2
  • The paper introduces a diverse abnormal-activity dataset spanning single-person and group anomalies, with 4,83,566 frames.The authors describe it as the largest dataset in terms of volume, to their knowledge.Source: §I · p. 2

II. RELATED WORK

Prior abnormal-activity methods use varied representations and learning approaches, but the related-work discussion identifies single-timescale operation as a limitation. The proposed framework uses pose trajectories to predict both past and future at multiple timescales and combines those predictions into errors for abnormality detection.

II. RELATED WORK

  • The related-work discussion identifies single-timescale operation as a major limitation of existing methods and motivates a multi-timescale model.The reviewed approaches include reconstruction, sparse coding, GAN-based representations, future-frame prediction, and trajectory prediction with reconstruction.Source: §II · p. 2

III. PROPOSED MODEL

  • The framework uses human pose trajectories as input and adds a past-prediction model alongside future prediction to represent dynamics at different timescales.The paper describes pose trajectories as compact and effective for capturing human movement.Source: §III · p. 2
  • Past and future predictions are combined at each timescale, then predictions across timescales yield an error sequence used to flag instants above a predefined threshold.The past model receives the input sequence in reverse order; both prediction models share an architecture but are trained differently.Source: §III · p. 3, Figure 2 · p. 3

A. Problem setup

The model represents each person's pose over time and produces averaged predictions and anomaly scores from multiple timescales. Its hierarchical 1D-convolutional architecture uses intermediate supervision to generate past and future trajectory predictions.

A. Problem setup

  • The input is a human pose trajectory represented by 25 skeleton points over time, with each person's pose defined from image coordinates of those points.The corresponding predicted pose is denoted separately, and errors across timescales are combined into an abnormality score.Source: A. Problem setup · p. 3
  • Sliding-window predictions at each timescale are averaged for each time instant, and errors from all timescales are combined into an abnormality score.The same averaging approach is used for past predictions.Source: A. Problem setup · p. 3

B. Model

  • A seven-layer 1D convolutional model receives frame-level pose features encoded to length 1024, with intermediate layers supervised to predict past and future steps.The first two convolutional layers use filters of length 3 and the next five use filters of length 5; the layer receptive field determines its timescale.Source: B. Model · p. 3
  • The predicted-pose illustration contrasts a walking scene with three abnormal-action scenes, where the model generates larger prediction errors for abnormal poses.Green and blue poses indicate actual and predicted poses, respectively.Source: Figure 3 · p. 4
  • The model uses timescales 3, 5, 13, and 25, and its decoder produces an output whose length depends on the timescale.Each decoder has two fully connected layers of lengths 1024 and timescale × 50.Source: B. Model · p. 4

C. Loss function

Training jointly supervises intermediate layers with node-level and layer-level losses. Prediction errors are aggregated across nodes and time, while pose-point confidence weights the weighted MSE.

C. Loss function

  • Supervised intermediate layers jointly minimize node-level prediction loss and layer-level sequence loss.The node loss measures prediction error against ground truth; the layer loss captures error for a complete input sequence.Source: C. Loss function · p. 4
  • Layer loss averages the errors from different nodes at each time instant, then sums those average errors across time.The sliding-window approach produces multiple predictions at an instant, whose errors are averaged before temporal summation.Source: C. Loss function · p. 4
  • Each node predicts over a duration T(i), and its node loss is based on prediction errors within that interval.The node-level loss compares a node’s predictions with the corresponding ground truth.Source: C. Loss function · p. 4
  • Prediction error uses weighted mean square error, with pose-estimator confidence supplying the weight for each pose point.The confidence score is provided by the pose detector.Source: C. Loss function · pp. 4–5
  • Jointly minimizing layer loss drives nodes to interact to reduce that loss; sliding-window training also simulates testing on long sequences.Testing commonly applies a sliding window to long sequences to form model inputs.Source: C. Loss function · p. 5

D. Anomaly detection

At test time, past and future prediction errors across timescales are combined by voting. The resulting error is thresholded to label time instants as abnormal.

D. Anomaly detection

  • At testing, a voting mechanism combines past and future prediction errors produced at different timescales into a final prediction error.The model produces these errors for a human at multiple timescales.Source: D. Anomaly detection · p. 5
  • At time t, the timescales included in the combination are those with predictions for that instant, using their layer losses.The set S contains timescales with predictions at t.Source: D. Anomaly detection · p. 5
  • When multiple humans are present, the method takes the maximum prediction error across individuals at each time instant.The per-individual quantity is the layer loss at that time.Source: D. Anomaly detection · p. 5
  • A time instant is tagged abnormal when its combined error exceeds the threshold.Source: D. Anomaly detection · p. 5

IV. CORRIDOR DATASET

The Corridor dataset contains college-campus corridor videos with normal walking and standing, plus enacted abnormal activities ranging from single-person to group events. It provides frame-level annotations and abnormal-activity classes.

IV. CORRIDOR DATASET

  • The dataset contains varied enacted anomalies, from single-person to group-level, in a college-campus corridor where normal activities are walking and standing.The authors report 4,83,566 frames across the dataset.Source: §IV · pp. 5–6
  • Annotations mark normal and abnormal activity at frame level, and abnormal frames also have activity classes for classification.Among 1,81,567 test frames, 1,08,278 are abnormal.Source: §IV · pp. 5–6

V. RESULTS AND DISCUSSIONS

The experiments evaluate the method on the proposed dataset and two public datasets, with human-anomaly variants used for the primary comparisons. The section proceeds from pose-trajectory preprocessing and training/testing schemes to comparisons with state-of-the-art methods.

V. RESULTS AND DISCUSSIONS

  • The method is tested on the proposed dataset, ShanghaiTech Campus, and Avenue.ShanghaiTech includes videos from 13 campus cameras; Avenue was captured at CUHK campus.Source: §V · p. 6
  • The experiments focus primarily on HR-ShanghaiTech and HR-Avenue because the task targets human anomalies.The authors describe examples including running, fighting, loitering, jumping, throwing a bag, and dancing.Source: §V · p. 6
  • The section compares the model's performance with state-of-the-art methods after presenting preprocessing, training, and testing schemes.Source: §V · p. 6

A. Data preparation

Pose trajectories are obtained by detecting people, tracking their bounding boxes, and running a pose detector on those trajectories.

A. Data preparation

B. Training scheme

The training scheme supervises selected layers for four timescales, using progressively longer input and prediction sequences. At test time, predictions from past and future models are combined by voting, and thresholded errors mark abnormal instants.

B. Training scheme

  • To produce predictions at different timescales, the model receives supervision at pre-chosen layers: layer 1 for timescale 3, layer 2 for 5, layer 4 for 13, and layer 7 for 25.Source: B. Training scheme · p. 6
  • Each epoch has four sub-epochs, progressively training from layer 1 alone to the complete model through layer 7.The first and last sub-epochs use trajectory lengths of 6 (3 in, 3 out) and 50 (25 in, 25 out), respectively; Adam is the optimizer.Source: B. Training scheme · p. 7

C. Testing

  • At test time, sequence lengths 6, 10, 26, and 50 feed layers 1, 2, 4, and 7 for their respective timescales.Source: C. Testing · p. 7
  • Past- and future-prediction errors are combined by voting; an error above threshold marks that instant abnormal.Source: C. Testing · p. 7

D. Performance Evaluation

The proposed model is compared with existing methods using Frame-AUC, and the section also introduces ablation studies of timescales and past prediction. The model outperforms other methods on HR-ShanghaiTech and is comparable on HR-Avenue, while not capturing non-human anomalies.

D. Performance Evaluation

  • The comparison with,,,,, and uses Frame-AUC; Table I presents the comparison.The passage states that the proposed model outperforms the other methods.Source: D. Performance Evaluation · p. 7
  • The model outperforms other methods on HR-ShanghaiTech and provides comparable performance on HR-Avenue, though it does not capture non-human anomalies.The proposed dataset is described as more challenging than the comparison datasets.Source: D. Performance Evaluation · p. 8

E. Ablation studies

1) Effect of multi-timescale framework:

The framework predicts pose trajectories in both past and future directions at multiple timescales, and the experiments report improved performance in most cases when adding timescales or past prediction. The paper concludes that it outperforms state-of-the-art models and releases a challenging abnormal activity dataset.

1) Effect of multi-timescale framework:

2) Effect of past prediction model:

VI. CONCLUSIONS

  • The multi-timescale framework outperforms state-of-the-art models and is used to detect abnormal activities through predictions of past and future pose trajectories.The paper also releases a challenging abnormal activity dataset for research use.Source: §VI · p. 8

Related papers

Loading 1908.04321v1…