PaperlayerView exact revision on arXiv

Source-linked AI summary

SSMTL++: Revisiting Self-Supervised Multi-Task Learning for Video Anomaly Detection

Antonio Barbalau, Radu Tudor Ionescu, Mariana-Iuliana Georgescu, Jacob Dueholm, Bharathkumar Ramachandra, Kamal Nasrollahi, Fahad Shahbaz Khan, Thomas B. Moeslund, Mubarak Shah

arXiv:2207.08003v4cs.CVcs.LG

TL;DR

Video anomaly detection is difficult because abnormality depends on context and abnormal events are rare, limiting fully supervised training data. This paper revises SSMTL's detection, backbone, and proxy tasks, and reports gains over SSMTL and recent methods on three benchmarks.

  • Problem

    Context determines whether an event is abnormal, and the rarity of abnormal events makes training data difficult to collect for fully supervised models.

    Source: §1 · pp. 1–2, §4.6 · p. 14

  • Method

    The paper revises SSMTL through changes to object detection, the backbone, and self-supervised proxy tasks.

    Source: §1 · p. 3

  • Results

    The revised frameworks report considerable gains over SSMTL and superior results to recent methods on Avenue, ShanghaiTech, and UBnormal.

    Source: §1 · p. 3, §5 · p. 14

  • Takeaways & Limitations

    The authors report optical flow with YOLOv3, 3D multi-head attention blocks, adversarial training on pseudo-anomalies, and patch inpainting as beneficial updates.

    Source: §5 · p. 14

    The work prioritizes anomaly-detection precision over running time; faster detectors or background subtraction can trade some precision for inference speed.

    Source: §4.6 · p. 14

Abstract

from arXiv · show

A self-supervised multi-task learning (SSMTL) framework for video anomaly detection was recently introduced in literature. Due to its highly accurate results, the method attracted the attention of many researchers. In this work, we revisit the self-supervised multi-task learning framework, proposing several updates to the original method. First, we study various detection methods, e.g. based on detecting high-motion regions using optical flow or background subtraction, since we believe the currently used pre-trained YOLOv3 is suboptimal, e.g. objects in motion or objects from unknown classes are never detected. Second, we modernize the 3D convolutional backbone by introducing multi-head self-attention modules, inspired by the recent success of vision transformers. As such, we alternatively introduce both 2D and 3D convolutional vision transformer (CvT) blocks. Third, in our attempt to further improve the model, we study additional self-supervised learning tasks, such as predicting segmentation maps through knowledge distillation, solving jigsaw puzzles, estimating body pose through knowledge distillation, predicting masked regions (inpainting), and adversarial learning with pseudo-anomalies. We conduct experiments to assess the performance impact of the introduced changes. Upon finding more promising configurations of the framework, dubbed SSMTL++v1 and SSMTL++v2, we extend our preliminary experiments to more data sets, demonstrating that our performance gains are consistent across all data sets. In most cases, our results on Avenue, ShanghaiTech and UBnormal raise the state-of-the-art performance bar to a new level.

1. Introduction

The paper revisits SSMTL by changing object detection, the backbone, and proxy tasks. Its SSMTL++ variants report gains over SSMTL and superior results to recent methods on Avenue, ShanghaiTech, and UBnormal.

1. Introduction

  • Context determines whether an event is abnormal, while the rarity of abnormal events makes training data difficult to collect for fully supervised models.The introduction contrasts a pedestrian walking on a sidewalk with crossing far from a crosswalk.Source: §1 · pp. 1–2
  • The work builds on SSMTL, which uses multiple self-supervised proxy tasks and a shared 3D convolutional backbone to detect anomalies without anomalies at training time.Georgescu et al. (2021) reported state-of-the-art performance for the original framework.Source: §1 · p. 3
  • The revisions target three components: object detection, the multi-task backbone, and self-supervised proxy tasks.Detection variants include optical flow and background subtraction; backbone variants add 2D or 3D CvT blocks, and proxy-task variants include segmentation, jigsaw, pose estimation, inpainting, and adversarial learning.Source: §1 · p. 3
  • The SSMTL++ frameworks report considerable gains over SSMTL and superior results to recent state-of-the-art methods on Avenue, ShanghaiTech, and UBnormal.The authors evaluate the two configurations after preliminary experiments identified promising combinations.Source: §1 · p. 3
  • The authors report performance improvements from CvT blocks in the backbone and empirical benefits from additional object-detection methods.The paper also studies new proxy-task combinations that produce superior performance levels.Source: §1 · p. 3

2. Related work

The related-work section situates SSMTL across video-anomaly taxonomies and presents it as a multi-task approach for anomaly detection without anomalous training examples. It also distinguishes SSMTL from prior reconstruction and self-supervised approaches.

Video anomaly detection

  • SSMTL addresses the multi-scene formulation, where anomalies are not expected to depend on location, and spans multiple method categories rather than fitting only one.Its middle bounding box prediction can be viewed as reconstruction-based, and the framework operates at object-patch level.Source: §2 · p. 3

Closest method

  • The paper identifies SSMTL as the closest prior method and presents SSMTL++v1 and SSMTL++v2 as its successors.The stated contributions also mark the differences from Georgescu et al. (2021).Source: §2 · pp. 3–4

Multi-task learning

  • SSMTL uses multi-task learning to generalize to anomalous patterns despite the absence of anomalous training data.The section characterizes SSMTL as the first framework to use multi-task learning explicitly for that purpose.Source: §2 · p. 4

Self-supervised learning

  • SSMTL is described as the first framework to use middle-patch prediction at the object level, unlike prior frame-level reconstruction and prediction approaches.The cited approaches include frame-level reconstruction, future-frame prediction, and middle-frame prediction.Source: §2 · p. 4

3. Method

SSMTL++ extends the original object-centric, multi-task framework by revising object detection and the shared backbone, adding proxy tasks, and defining inference scores for the tasks.

3. Method

  • The original SSMTL uses a shared 3D CNN with four proxy-task heads, and averages their task anomaly scores for each detected object.The original tasks predict temporal direction, motion irregularity, a missing middle crop, and outputs from pretrained models.Source: §3 · p. 4

3.1. Introducing new detection methods

  • Optical flow and background subtraction add connected components not already detected by YOLO, addressing missed unknown-class, motion-blurred, or fast-moving objects.Optical flow selects pixels above a motion-magnitude threshold; background subtraction thresholds foreground pixels after updating a scene background.Source: §3.1 · pp. 4–5

3.2. Introducing new backbones

  • The two CvT variants apply transformer blocks either after temporal pooling to 8 × 8 tokens or before it to 7 × 8 × 8 tokens.The 3D variant uses 3D depthwise convolutions and applies temporal pooling after the CvT module.Source: §3.2 · p. 5

3.3. Introducing new proxy tasks

  • Five added proxy tasks cover adversarial pseudo-anomaly reconstruction, object-level patch inpainting, segmentation distillation, jigsaw classification, and pose-estimation distillation.The tasks use out-of-distribution images, masked object patches, a Mask R-CNN teacher, shuffled image patches, and a UniPose teacher, respectively.Source: §3.3 · pp. 5–6

3.4. Inference

  • At inference, task scores are averaged per object, mapped to pixel-level anomaly maps, and temporally smoothed to produce frame-level scores.The adversarial reconstruction task limits reconstruction power but does not contribute to the final anomaly score.Source: §3.4 · p. 6

4.1. Data sets

The experiments use Avenue, ShanghaiTech, and UBnormal, each with normal-only training for this self-supervised framework.

Avenue

  • Avenue has 16 normal-only training videos and 21 test videos with normal and abnormal actions, with frame- and pixel-level annotations.Frames are 360 × 640 pixels.Source: §4.1 · p. 6

ShanghaiTech

  • ShanghaiTech has 330 normal-action training videos and 107 test videos with normal and abnormal events, with frame- and pixel-level annotations.The full dataset contains 437 videos; frames are 480 × 856 pixels.Source: §4.1 · p. 6

UBnormal

  • UBnormal has 268 training, 64 validation, and 211 test videos, with abnormal training actions disjoint from test abnormal actions.Only normal videos are used to train the framework; annotations are at frame and pixel levels.Source: §4.1 · p. 6

4.2. Evaluation and implementation details

Evaluation reports frame-level AUC and region- and track-based localization criteria; implementation retains the original framework settings while specifying optimizer and new-task parameters.

Evaluation metrics

  • Frame-level AUC is evaluated against ground-truth annotations, with micro and macro scores reported for state-of-the-art comparisons.Preliminary experiments use micro AUC; the authors prioritize it because many prior works use that measure.Source: §4.2 · pp. 6–7

Hyperparameter settings

  • Training uses 20 Adam epochs at η = 10^-3, with batch sizes of 64, 128, or 256 and validation-based proxy-task model selection.The implementation retains original SSMTL hyperparameters and uses 15% of training data for validation.Source: §4.2 · p. 7

Additional proxy-task hyperparameters

  • New-task settings use an adversarial learning rate of −0.2·η for T5 and random 4–32-pixel mask patches for T6.T6 patch centers are sampled from a 2D Gaussian centered in the input image, with standard deviation 20 in each direction.Source: §4.2 · p. 7

4.3. Preliminary experiments

The preliminary experiments test detection methods, transformer-based backbones, and additional proxy tasks before selecting two SSMTL++ configurations. They find gains from optical-flow detections and a 3D CvT backbone, while showing that task combinations and training choices matter.

4.3.1. Experimenting with new detection methods

  • YOLOv3 with optical flow reaches 89.6% micro AUC on Avenue, outperforming the tested detection alternatives.Background subtraction yields higher recall but adds false positives; optical flow is selected for subsequent experiments.Source: §4.3.1 · pp. 7–8, Table 1 · p. 7

4.3.2. Experimenting with new backbones

  • A 3D transformer module adds 1% micro AUC over the 3D CNN for one task, and the best tested multi-task configuration uses 3 blocks with 12 attention heads.With three tasks, the new backbone improves on the old backbone by up to 1.8%.Source: §4.3.2 · p. 8, Table 2 · p. 8, Table 3 · p. 8

4.3.3. Experimenting with new tasks

  • Tasks T5 (adversarial reconstruction) and T6 (patch inpainting) are the most promising additions, but combining them does not retain their separate gains.T8 (jigsaw) is the only standalone task reported to fail sufficiently at detecting anomalies; five or more jointly optimized tasks cause significant drops.Source: §4.3.3 · pp. 8–9, Table 5 · p. 8, Table 6 · p. 9

4.3.4. Chosen SSMTL ++ configurations

  • The selected configurations pair YOLOv3 and optical flow with a 3D CNN + 3D CvT backbone: SSMTL++v1 uses T1, T2, T3, and T5, while SSMTL++v2 uses T1, T2, T3, and T6.Task T5 is introduced after the other tasks have had time to converge; epoch 5 is the reported optimal starting point.Source: §4.3.4 · p. 9, §4.3.3 · p. 9

4.4. Comparison with the state of the art

SSMTL++v1 and SSMTL++v2 generally outperform SSMTL across Avenue, ShanghaiTech, and UBnormal, with most comparisons reaching new state-of-the-art levels. Their relative performance varies by dataset and metric, including an Avenue trade-off between track-based and region-based detection scores.

4.4. Comparison with the state of the art

  • 93.7% micro AUC makes SSMTL++v1 the top model on Avenue, where both variants improve micro AUC, macro AUC, and TBDC over SSMTL but decline on RBDC.The TBDC gains are +23.8% and +26.9%; mistaken anomaly labels among new detections increase false positives and lower RBDC.Source: §4.4 · p. 9, Table 7 · p. 10
  • On ShanghaiTech, SSMTL++v2 achieves the top performance on each metric, while both SSMTL++ variants improve over SSMTL across all metrics.SSMTL++v1 shares second or third place, depending on the metric, with two state-of-the-art frameworks.Source: §4.4 · p. 9, Table 8 · p. 11
  • On UBnormal, both variants surpass SSMTL; SSMTL++v1 leads on macro AUC, RBDC, and TBDC, while SSMTL++v2 ranks second on macro AUC and TBDC.The reported TBDC and RBDC differences favor the proposed models over TimeSformer despite lower micro AUC scores.Source: §4.4 · p. 10
  • Across the three benchmarks, SSMTL++v1 and SSMTL++v2 generally surpass competing methods, reaching new state-of-the-art levels in most cases.On ShanghaiTech, SSMTL++v2 leads on every reported metric; on UBnormal, SSMTL++v1 sets new highs on macro AUC, RBDC, and TBDC.Source: §4.4 · pp. 9–11, Table 7 · p. 10, Table 8 · p. 11

4.5. Ablation study

Ablations show that optical flow detections and the CvT backbone improve performance, with additional proxy tasks contributing to the strongest configurations. Removing optical flow or CvT reduces performance in both framework versions.

4.5. Ablation study

  • 89.3% to 90.7% to 92.5% frame-level micro AUC: adding optical flow to the baseline, then CvT, improves Avenue performance.The baseline uses YOLOv3, a 3D CNN, and tasks T1, T2, and T3.Source: §4.5 · p. 11
  • 93.7% to 90.2% frame-level micro AUC: removing CvT reduces SSMTL++v1 performance, and removing optical flow also lowers it.The highest performance is obtained with optical flow, CvT, and task T5 all in place.Source: §4.5 · p. 12
  • More than 1.2% lower frame-level micro AUC results when optical flow or CvT is removed from SSMTL++v2.The ablations assess task T6 across different component settings.Source: §4.5 · p. 12
  • Each component of SSMTL++v1 and SSMTL++v2 plays a key role in obtaining higher performance and surpassing current state-of-the-art results.The ablation summary attributes gains to the frameworks’ individual components.Source: §4.5 · p. 12

4.6. Running time

SSMTL++ adds preprocessing and inference costs, though its authors report real-time processing on a single GPU using two parallel threads. They identify optical flow as the main bottleneck and note that faster alternatives trade away some precision.

4.6. Running time

  • Training takes roughly 16 minutes per Avenue epoch, 3 hours per ShanghaiTech epoch, and 1 hour 15 minutes per UBnormal epoch.With 20 epochs, ShanghaiTech training takes up to 60 hours; training-time differences from SSMTL are negligible.Source: §4.6 · p. 12
  • Optical flow, appended transformer blocks, and SSMTL++v2’s inpainting task add processing; pseudo-anomaly task T5 affects training only.T6 requires three model passes during prediction.Source: §4.6 · p. 13
  • 3 ms per object-centric sequence for SSMTL++v1 with a batch of seven samples, versus 4 ms for SSMTL++v2 at the same batch size.Without batching, SSMTL++v1 takes 12 ms per sequence.Source: §4.6 · p. 13
  • About 15 FPS sequentially and about 20 FPS with two threads: both SSMTL++ versions are reported fast enough for real-time processing.SSMTL processes Avenue at about 50 FPS; the reported timings use a GeForce GTX 3090 with 24 GB VRAM.Source: §4.6 · p. 13
  • The authors identify optical flow as the main bottleneck and note that faster object detectors or 55 FPS background subtraction trade away a small amount of precision.The work prioritizes anomaly-detection precision over running-time optimization.Source: §4.6 · pp. 13–14

5. Conclusion

The conclusion credits optical-flow detections, attention-based backbone updates, and new proxy tasks with improving the revisited framework. It also identifies a challenge in scaling to more than five tasks.

5. Conclusion

  • Optical flow with YOLOv3 finds more objects, while 3D convolutional multi-head attention blocks and new proxy tasks add performance gains.The cited proxy tasks are adversarial training on pseudo-anomalies and patch inpainting.Source: §5 · p. 14
  • Pseudo-anomaly and patch-inpainting tasks improve the pipeline when replacing the original knowledge-distillation task T4, rather than being added alongside it.Source: §5 · p. 14
  • Models trained on more than five tasks seem to underperform, motivating future study of learning from more tasks.Source: §5 · p. 14

Related papers

Loading 2207.08003v4…