PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Masked Convolutional Transformer Block for Anomaly Detection

Neelu Madan, Nicolae-Catalin Ristea, Radu Tudor Ionescu, Kamal Nasrollahi, Fahad Shahbaz Khan, Thomas B. Moeslund, Mubarak Shah

arXiv:2209.12148v2cs.CVcs.AIcs.LG

TL;DR

Anomaly detection is commonly trained on normal examples alone, while collecting abnormal training examples is difficult. The paper introduces SSMCTB, a flexible self-supervised masked-reconstruction block, and reports consistent improvements across models and tasks. Its evaluation avoids per-model and per-dataset hyperparameter tuning because representative validation splits are unavailable.

  • Problem

    Anomaly detection is often trained only on normal data, while context-dependent anomalies are difficult to define and abnormal examples are hard to gather.

    Source: §1 · p. 1

  • Method

    SSMCTB combines masked convolution and channel-wise transformer attention in a self-supervised block that reconstructs masked information from context using Huber loss.

    Source: §1 · p. 2, §3.3 · p. 7

  • Results

    SSMCTB yields consistent improvements across models and tasks, with gains over SSPCAB in most cases.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The block can add masked-information reconstruction at different network layers and integrate with a broad variety of neural architectures.

    Source: §1 · p. 2

    The evaluation avoids hyperparameter tuning for each model and dataset because anomaly-detection datasets generally lack validation splits with representative normal and abnormal examples.

    Source: §4.4 · p. 9

Abstract

from arXiv · show

Anomaly detection has recently gained increasing attention in the field of computer vision, likely due to its broad set of applications ranging from product fault detection on industrial production lines and impending event detection in video surveillance to finding lesions in medical scans. Regardless of the domain, anomaly detection is typically framed as a one-class classification task, where the learning is conducted on normal examples only. An entire family of successful anomaly detection methods is based on learning to reconstruct masked normal inputs (e.g. patches, future frames, etc.) and exerting the magnitude of the reconstruction error as an indicator for the abnormality level. Unlike other reconstruction-based methods, we present a novel self-supervised masked convolutional transformer block (SSMCTB) that comprises the reconstruction-based functionality at a core architectural level. The proposed self-supervised block is extremely flexible, enabling information masking at any layer of a neural network and being compatible with a wide range of neural architectures. In this work, we extend our previous self-supervised predictive convolutional attentive block (SSPCAB) with a 3D masked convolutional layer, a transformer for channel-wise attention, as well as a novel self-supervised objective based on Huber loss. Furthermore, we show that our block is applicable to a wider variety of tasks, adding anomaly detection in medical images and thermal videos to the previously considered tasks based on RGB images and surveillance videos. We exhibit the generality and flexibility of SSMCTB by integrating it into multiple state-of-the-art neural models for anomaly detection, bringing forth empirical results that confirm considerable performance improvements on five benchmarks. We release our code and data as open source at: https://github.com/ristea/ssmctb.

1 INTRODUCTION

The paper introduces SSMCTB, a self-supervised block that reconstructs masked information from context and can be inserted at different network layers. Experiments report improvements across models and tasks, while extending evaluation to medical images and thermal videos.

1 INTRODUCTION

  • SSMCTB encapsulates masked-information reconstruction in a neural block that can introduce self-supervision at any network layer and integrate with varied architectures.The block predicts masked information from visible context, with dilation controlling the context from local to global.Source: §1 · p. 2
  • The block combines masked convolution with channel-wise transformer attention and replaces SSPCAB's MSE objective with Huber loss.The Huber loss is described as less sensitive to outliers than MSE.Source: §1 · p. 2
  • Experiments cover five benchmarks across industrial and medical images, RGB video, and thermal video, including the new Thermal Rare Event benchmark.The benchmark was created by manually labeling abnormal events in one week of Seasons in Drift video.Source: §1 · pp. 2–3
  • SSMCTB brings consistent improvements across models and tasks, with gains over SSPCAB in most cases.The authors attribute the gains over SSPCAB to multi-head self-attention and Huber loss replacing standard channel attention and MSE, respectively.Source: §1 · p. 3
  • The paper extends masked convolution to 3D and integrates 3D SSMCTB into two 3D anomaly-detection networks.The 3D filters incorporate spatiotemporal context.Source: §1 · p. 3
  • The contribution set also includes an extensive ablation study and integrations into several state-of-the-art anomaly-detection methods.The integrations span multiple models and domains.Source: §1 · p. 3

2 RELATED WORK

The related work situates SSMCTB among transformer attention, masked self-supervision, and anomaly-detection methods. It distinguishes the block by embedding masked reconstruction within a neural component and reports benefits from channel-wise attention.

2.1 Transformers

  • Channel-wise self-attention in SSMCTB outperforms spatial attention and processes faster in the reported ablation.The block combines transformer attention with masked convolution and can be integrated into models that are not themselves transformer-based.Source: §2.1 · p. 3

2.2 Self-Supervision via Information Masking

  • Unlike masked pretraining methods, SSMCTB embeds information masking inside neural networks and trains the block with reconstruction loss for anomaly detection.The authors report that SSMCTB can also boost masked auto-encoders for anomaly detection.Source: §2.2 · p. 3

2.3 Anomaly Detection

  • Where reconstruction models may generalize too well to anomalies, SSMCTB increases reconstruction difficulty by masking information wherever the block is inserted.The approach is presented as a single neural block that can be integrated into CNNs and transformers across domains.Source: §2.3 · p. 4

3 METHOD

SSMCTB places masked-information reconstruction inside a neural block, using contextual information to predict masked values. Its dilated masked convolutions and channel-wise transformer are trained with a self-supervised reconstruction loss.

3.1 Motivation and Overview

  • SSMCTB is designed to reconstruct masked information from context, addressing CNNs’ limited receptive fields and difficulty modeling the global arrangement of local features.The block can be inserted at any level of almost any CNN or transformer model; the authors propose anomaly detection as an application because models trained on normal data learn to recover masked normal information.Source: §3.1 · p. 5

3.2 Architecture

  • The 2D masked convolution hides the receptive-field center and predicts its value from four learnable visible sub-kernels, with dilation controlling their distance from the masked region.Stride 1 and zero-padding preserve the input’s spatial dimensions while masking each input element once; k′ and d are the layer’s tunable hyperparameters.Source: §3.2 · p. 5, Figure 2 · p. 5
  • The 3D masked convolution extends the kernel to eight visible sub-kernels around a central masked region, enabling SSMCTB to integrate into networks using 3D convolutional layers.Its receptive field is k × k × k, where k = 2k′ + 2d + 1; the output spatial dimensions match the input’s.Source: §3.2 · p. 6, Figure 3 · p. 6
  • A transformer replaces SSPCAB’s Squeeze-and-Excitation module, using self-attention to model channel-wise interrelations among masked-convolution activation maps.Spatially pooled features are reshaped into channel tokens, processed by transformer blocks, and used to produce channel attention weights for the reconstructed output.Source: §3.2 · pp. 6–7

3.3 Self-Supervised Reconstruction Loss

  • SSMCTB is trained with a Huber reconstruction loss between its output and the input, replacing SSPCAB’s mean squared error and reducing sensitivity to outliers.The loss switches from squared error below threshold δ to absolute error at or above δ, and is added to the host model’s loss with weight λ.Source: §3.3 · p. 7

4.1 Data Sets

The evaluation spans industrial and medical images, surveillance videos, and a newly constructed thermal-video benchmark. The datasets use normal-only training and abnormal examples for testing.

4.1 Data Sets

  • MVTec AD contains over 5,000 images across 15 categories, with 3,629 defect-free training samples and 1,725 test images.Source: §4.1 · p. 7
  • BRATS is split so training images are lesion-free: 11,280 slices from 125 scans for training and 27,745 slices from 180 scans for testing.Source: §4.1 · p. 7
  • Avenue has 16 normal-only training videos and 21 test videos; ShanghaiTech has 330 training and 107 test videos, with anomalies appearing only at test time.Source: §4.1 · p. 7
  • Thermal Rare Event was manually annotated from 330 Seasons in Drift clips, yielding 36,120 test frames and 3,480 training frames.The sampled week contains 29 clips with rare events; the source thermal surveillance data spans eight months.Source: §4.1 · pp. 7–8

4.2 Evaluation Measures

Image anomaly detection is evaluated at both image and pixel levels, while video anomaly detection uses frame-level and region/track localization measures.

4.2 Evaluation Measures

  • Image detection and localization are measured with AUROC and average precision (AP), with localization evaluated at the pixel level.Localization AP averages results across threshold levels.Source: §4.2 · p. 8
  • Video detection uses frame-level AUC, reported as micro AUC over concatenated frames and macro AUC averaged across videos.Source: §4.2 · p. 8
  • Video localization uses region-based detection criterion (RBDC) and track-based detection criterion (TBDC), with overlap thresholds α = 0.1 and β = 0.1.Source: §4.2 · p. 8

4.3 Implementation Details

SSPCAB and SSMCTB are added to existing models while retaining their prescribed hyperparameters, with a few specified implementation exceptions and loss-weight adjustments.

4.3 Implementation Details

  • The authors integrate SSPCAB and SSMCTB into ten state-of-the-art approaches while preserving baseline hyperparameters, and compare against reproduced baselines.The reproduced Park et al. and Liu et al. results do not exactly match their originals, but remain close.Source: §4.3 · p. 8
  • SSMCTB usually replaces the penultimate convolutional layer, with exceptions for Georgescu et al. and the masked auto-encoder.For, it is placed in the decoder; for, it is placed before the first transformer block.Source: §4.3 · p. 8
  • The self-supervised loss weight λ is 0.1 generally, reduced to 0.001 for two baselines,; the channel-wise transformer uses 64-dimensional tokens, four heads, and two blocks.The transformer activation map is pooled to 1 × 1; RGB-to-thermal transfer for Park et al. [44] also changes epochs and mini-batch size.Source: §4.3 · p. 8

4.4 Preliminary Results

Preliminary Avenue experiments select SSMCTB settings for later evaluations, while the authors avoid per-model and per-dataset tuning because anomaly-detection training sets lack representative validation splits.

4.4 Preliminary Results

  • Masked convolution without attention improved performance by more than 1% for dilation rates d = 2 and d = 3, so attention was not the only factor associated with gains.Source: §4.4 · p. 8
  • 86.7% micro AUC is the best Avenue score, obtained with d = 1, k′ = 1, and channel attention; larger sub-kernels and combined channel-plus-spatial attention did not improve it.The final configuration uses these settings across underlying models and datasets.Source: Table 2 · p. 9, §4.4 · p. 9
  • Huber loss generally outperformed MAE, MSE, and SSIM, while channel attention generally outperformed spatial attention in the preliminary comparisons.Source: §4.4 · pp. 8–9
  • SSMCTB retains SSPCAB’s tuned channel-attention, d = 1, and k′ = 1 settings, but uses Huber loss instead of MSE.SSPCAB hyperparameters were tuned in the previous work.Source: §4.4 · p. 9
  • The authors avoid per-model and per-dataset hyperparameter tuning because normal-only training data cannot provide a representative validation subset containing both normal and abnormal examples.They describe this choice as avoiding overfitting in hyperparameter space and making evaluation fairer.Source: §4.4 · p. 9

4.5 Anomaly Detection in Images

On MVTec AD and BRATS, adding SSMCTB improves anomaly detection results across several image models, with the 3D version yielding further gains on BRATS. Qualitative examples also show anomaly localizations more closely aligned with ground truth.

Results on MVTec AD

  • On MVTec AD, adding SSPCAB or SSMCTB improves detection for all three models; DRAEM localization AP gains around 2%, while its localization AUROC does not improve.Localization AUROC also rises for NSA and FastViT+NSA.Source: §4.5 · p. 9, Table 3 · p. 10

Qualitative results

  • On MVTec AD and BRATS, qualitative examples show SSMCTB localizations better aligned with ground-truth regions.The BRATS examples are from DRAEM; the MVTec AD examples are also from DRAEM.Source: §4.5 · pp. 9–10, Figure 5 · p. 10

Results on BRATS

  • On BRATS, SSMCTB significantly improves DRAEM, NSA, and FastViT+NSA, and 3D SSMCTB yields still higher performance by using the MRI scans’ 3D nature.The reported gains from SSMCTB are always higher than those from SSPCAB.Source: §4.5 · p. 10, Table 4 · p. 10

4.6 Anomaly Detection in Videos

The video experiments evaluate SSMCTB across RGB and thermal benchmarks and report improvements across several underlying models. The reported results include frame-level gains on Thermal Rare Event and improvements from the 3D block on RGB-video metrics.

Results on RGB videos

  • On Avenue and ShanghaiTech, SSMCTB and SSPCAB generally improve RGB-video models; their relative gains vary by model and metric, with SSMCTB leading for Wang et al.’s object-centric transformer.For Park et al. on Avenue, SSMCTB raises micro AUC from 82.8% to 87.0%, while SSPCAB raises macro AUC from 86.8% to 88.6%.Source: §4.6 · pp. 11–12, Table 5 · p. 11
  • On Avenue, 3D SSMCTB gives the highest reported micro AUC (93.2%), macro AUC (93.9%), and RBDC (66.04%); SSPCAB has the highest TBDC (89.28%).On ShanghaiTech, 3D SSMCTB improves three metrics for the underlying model.Source: Table 5 · p. 11, §4.6 · p. 12

Qualitative results

  • Video examples show SSMCTB detecting running and throwing events on Avenue, a bike rider in a pedestrian area on ShanghaiTech, and a vehicle moving backwards in thermal video.For the Avenue video of a person throwing an object, SSMCTB also raises the score for a penultimate abnormal event missed by the baseline.Source: §4.6 · pp. 12–13, Figure 6 · p. 12, Figure 7 · p. 12

Results on thermal videos

  • On Thermal Rare Event, SSMCTB improves the micro AUC by more than 5%, with larger gains than SSPCAB; the selected Park et al. baseline scores 53.2% micro AUC and 66.5% macro AUC.The object-centric and very deep baselines attain micro AUC values under 50% in the thermal domain.Source: §4.6 · pp. 12–13, Table 6 · p. 12

4.7 Inference Time

The authors assess the inference-time cost of integrating SSMCTB into existing frameworks. They report small added delays for both 2D and 3D versions and judge the accuracy gains to outweigh them.

Inference time

  • The experiments generally add one SSMCTB, usually at the penultimate convolutional layer.The authors consider the measured delay increases marginal and the accuracy gains worthwhile.Source: §4.7 · p. 13
  • Adding 3D SSMCTB to DRAEM increases running time by around 0.2 ms, within the same range as the 2D version’s added delay.The 3D runtime was measured on DRAEM.Source: §4.7 · p. 13

4.8 Ablation Study

The ablations examine block placement, masked-region size, transformer settings, Huber-loss threshold, and the convolution operation. Results favor one block near the output, a 1 × 1 masked region, and masked over dilated convolution.

Block placement

  • On Avenue, the best micro AUC is 87.0% with one SSMCTB in the decoder’s last convolution block at d = 1; every tested placement and dilation rate beats the 82.8% baseline.Earlier placement favors d = 4, middle placement d = 3, and multiple blocks bring no major improvement.Source: §4.8 · pp. 13–14

Size of masked region

  • A 1 × 1 masked region performs best on Avenue; increasing the masked-region size lowers anomaly-detection scores.The compared sizes are 1 × 1, 2 × 2, and 3 × 3.Source: §4.8 · p. 14, Table 9 · p. 14

Transformer architecture

  • The best Avenue transformer settings include 1 × 1 activation maps (global average pooling), token size dt = 64, four or five heads, and two successive blocks.All tested transformer configurations surpass the baseline model.Source: §4.8 · p. 14, Table 10 · p. 14

Huber loss hyperparameter

  • For the Huber loss, δ = 1 gives the maximum improvement, although δ = 0.5 and δ = 2 also outperform the baseline.δ is the threshold that switches between L1 (MAE) and L2 (MSE) loss.Source: §4.8 · p. 14, Table 11 · p. 14

Comparison with dilated convolution

  • Masked convolution outperforms dilated convolution at every tested dilation rate, supporting the importance of reconstructing the masked center region.The comparison tests dilation rates from 1 to 3.Source: Table 12 · p. 14, §4.8 · p. 15

5 CONCLUSION

SSMCTB improves anomaly detection across most tested settings and reaches state-of-the-art levels on Avenue and ShanghaiTech. The authors also extend the block to 3D networks and identify self-supervised pre-training as a future application.

5 CONCLUSION

  • Across multiple benchmarks and underlying models, SSMCTB improves performance in a vast majority of cases and reaches new state-of-the-art levels on Avenue and ShanghaiTech.Source: §5 · p. 15
  • The work extends SSMCTB to 3D masked convolution and integrates it into 3D DRAEM and SSMTL++v2 for image and video anomaly detection.It also adds medical-image and thermal-video benchmarks, expanding beyond the domains considered in the previous work.Source: §5 · p. 15
  • SSMCTB combines masked convolution with a channel-wise transformer to predict the masked center region, using its own self-supervised reconstruction loss.Source: §5 · p. 15
  • Future work proposes using the masked-region loss for self-supervised pre-training at multiple architectural levels before downstream tasks.Source: §5 · p. 15
Loading 2209.12148v2…