PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent Space Autoregression for Novelty Detection

Davide Abati, Angelo Porrello, Simone Calderara, Rita Cucchiara

arXiv:1807.01653v2cs.CV

TL;DR

Novelty detection must identify observations unlike normal training data even though novel examples are unavailable during training. The paper combines a deep autoencoder with autoregressive latent-density estimation, showing that joint reconstruction and maximum-likelihood training regularizes representations through differential-entropy minimization. Experiments report on-par or superior performance against state-of-the-art methods in one-class and video anomaly detection.

  • Problem

    Novelty detection is difficult because novelties are unpredictable and unavailable during training, making the task unsupervised.

  • Method

    A deep autoencoder is paired with an autoregressive density estimator that learns the distribution of latent representations by maximum likelihood.

  • Results

    The model delivers on-par or superior performance against state-of-the-art methods in one-class and video anomaly detection settings.

  • Takeaways & Limitations

    Joint reconstruction and latent maximum-likelihood training acts as a novelty-detection regularizer by minimizing the differential entropy of encoder representations.

  • Takeaways & Limitations

    Replacing multinomial conditional probability models with Gaussians did not yield considerable improvements and introduced numerical instabilities.

Abstract

from arXiv · show

Novelty detection is commonly referred to as the discrimination of observations that do not conform to a learned model of regularity. Despite its importance in different application settings, designing a novelty detector is utterly complex due to the unpredictable nature of novelties and its inaccessibility during the training procedure, factors which expose the unsupervised nature of the problem. In our proposal, we design a general framework where we equip a deep autoencoder with a parametric density estimator that learns the probability distribution underlying its latent representations through an autoregressive procedure. We show that a maximum likelihood objective, optimized in conjunction with the reconstruction of normal samples, effectively acts as a regularizer for the task at hand, by minimizing the differential entropy of the distribution spanned by latent vectors. In addition to providing a very general formulation, extensive experiments of our model on publicly available datasets deliver on-par or superior performances if compared to state-of-the-art methods in one-class and video anomaly detection settings. Differently from prior works, our proposal does not make any assumption about the nature of the novelties, making our work readily applicable to diverse contexts.

1. Introduction

The paper proposes novelty detection by combining autoencoder-based reconstruction with autoregressive density estimation in latent space. Joint training encourages low-entropy, predictable representations while retaining surprisal as a novelty signal.

  • Motivation: Novelty detection identifies samples with traits significantly different from a regularity model learned from normal samples.
  • Novelty measure: Novelty is assessed through both poor reconstruction and surprising latent representations under the learned prior.
  • Proposed framework: The framework combines a deep autoencoder with an autoregressive estimator that models the distribution of latent representations by maximum likelihood.
  • Autoregressive estimation: The autoregressive formulation factorizes the latent distribution into conditional densities and avoids imposing a predefined distributional family.
  • Entropy regularization: Maximum likelihood minimizes the differential entropy of encoder-produced codes, encouraging features that are predictable, common, and recurrent in normal data.

4. Experiments1

Experiments evaluate the model across one-class image novelty detection, video anomaly detection, semantic inputs, autoregressive density-estimator architectures, and cognitive temporal processes. The model combines reconstruction and latent log-likelihood terms, and achieves strong reported performance across these settings.

  • Experimental setup: The novelty score sums reconstruction and latent log-likelihood terms, with individual scores normalized using an experiment-specific reference set.The reconstruction term captures memory, while the log-likelihood term captures surprisal from latent representations.
  • One-class novelty detection: Image experiments train separately on each MNIST or CIFAR-10 class and test against all-class test sets using AUROC comparisons with OC-SVM, KDE, DAE, VAE, Pix-CNN, and GAN-based baselines.The model is trained on normal samples from one class, while test sets contain examples from all classes.
  • One-class novelty detection: The proposal outperforms all baselines in both MNIST and CIFAR10 settings.The comparison is reported in Table 1 using AUROC, the standard metric for the task.
  • One-class novelty detection: On CIFAR-10, the proposal is the only model outperforming KDE, while its likelihood term adds surprisal-based assessment beyond the DAE’s reconstruction-only score.The authors also relate the gain over DAE to minimizing latent differential entropy, which increases reconstruction discriminability.
  • Model analysis: The ablation study reports that combining reconstruction and log-likelihood is advantageous on each dataset, while deeper MFC and MSC estimators outperform their shallow or recurrent alternatives.MFC and MSC use disentangled parametrizations for each output conditional distribution; the architecture comparison is reported with test AUROC.
  • Video anomaly detection: In video surveillance, the model is evaluated on UCSD Ped2 and ShanghaiTech, using patch or clip aggregation for scoring and localization strategies for anomalous regions.For ShanghaiTech, occluding the anomaly source with a rectangular patch is expected to reduce the novelty score.
  • Cognitive temporal processes: With fixation-map clips from DR(eye)VE, the model achieves AUROC 0.926 for detecting uncommon attentional patterns such as distractions or shifts in attention.The novelty score is used as a proxy for the uncommonness of a driver’s attentional pattern.

5. Conclusions

The paper proposes a general novelty-detection framework combining a deep generative autoencoder with an autoregressive latent-density estimator. It reports state-of-the-art performance across one-class and anomaly-detection settings without assumptions about novelty types.

  • The framework models novelty through both failure to remember unseen data and surprisal in latent representations.
  • A deep generative autoencoder is paired with an autoregressive density estimator trained by maximum likelihood on latent vectors.
  • The auxiliary latent-space module minimizes encoder differential entropy, providing a regularizer for novelty detection.
  • Experiments report state-of-the-art performance in one-class and anomaly-detection settings.
  • The framework is designed for different tasks without making data-related assumptions about the nature of novelties.

6. On the implementation details

The experiments use specified architectural and optimization settings, with hyperparameters selected on held-out validation data using the raw objective.

  • Architectures and hyperparameters are reported by block type, autoregressive layers, batch size, learning rate, and log-likelihood weight.
  • All intermediate layers use Leaky ReLU activations, and the objective is optimized with Adam.
  • Hyperparameters are tuned on a held-out validation set by minimizing the raw objective with λ = 1.

7. On the log-likelihood objective

The log-likelihood objective models discretized latent variables with multinomial conditional distributions. Multinomials are one possible choice, while Gaussian alternatives introduce numerical instabilities without considerable improvement.

  • The log-likelihood term is computed by modeling each conditional probability distribution through a multinomial.
  • The implementation includes image- and video-oriented autoregressive layers alongside convolutional, residual, upsampling, dense, and temporally shared fully connected blocks.
  • The encoder uses a sigmoid so latent representations z = f(x; θf) lie in [0, 1]^d.
  • Each latent coordinate is linearly quantized into B bins, producing a B-dimensional categorical distribution that identifies its assigned bin.
  • Replacing multinomial conditional distributions with Gaussians leaves the framework intact but causes numerical instabilities without considerable improvements in observed trials.

8. On the relations to Variational Autoencoders

The proposed model and VAEs both combine reconstruction with latent-prior regularization, but the proposed model generates sharper CIFAR-10 samples. VAE samples are described as blurred or over-smooth and receive worse FID scores.

  • Both models minimize reconstruction error under a regularization constraint involving a prior distribution on latent vectors.
  • The proposed model’s CIFAR-10 samples show fine-grained details and sharpness, unlike the heavily blurred VAE samples.
  • The VAE produces over-smooth shapes lacking detail, while the proposed model’s samples are visually more realistic under equal settings.
  • FID scores further confirm substantial differences between the two models’ CIFAR-10 samples, with lower scores being better.

9. On the dual nature of novelty

The proposed novelty score combines reconstruction error with latent log-likelihood to capture complementary aspects of novelty. Experiments support the contribution of both terms and suggest that surprisal minimization regularizes reconstruction-based discrimination.

  • The novelty score combines reconstruction error (REC) and latent log-likelihood (LLK) to capture memory capability and latent surprisal.Table 3 compares LLK, REC, and the combined novelty score (NS) using AUROC.
  • Except for ShanghaiTech, accounting for both REC and LLK systematically improves novelty-detection performance.
  • 0.942 and 0.590 are the reported performance gaps between the proposed reconstruction error and denoising-autoencoder variants on MNIST and CIFAR-10, respectively.
  • Surprisal minimization acts as a novelty-oriented regularizer by improving the discriminative capability of reconstruction.

10. On the causal structure of representations

The encoder is trained with an autoregressive latent-density objective, and its representations are evaluated using Bayesian Networks with alternative causal orders. The training-consistent autoregressive order yields the best fit despite equal graph size and parameter counts.

  • The experiment fits Bayesian Networks with Linear Gaussian conditionals to latent representations from ten MNIST models.
  • The tested structures use the training autoregressive order, a random order, or its inverse.The corresponding parent sets are defined as Pa(zi) = {zj | j = 1, 2, . . . , i −1} for the autoregressive order and Pa(zi) = {zj | j = i + 1, i + 2, . . . , d} for the inverse order.
  • The three Bayesian Network structures have the same number of edges and independent parameters, isolating the effect of causal order on fitting capability.
  • The training-consistent autoregressive order delivers a better fit, supporting autoregressive properties in the learned latent features.

11. On the entropy minimization

Increasing the weight of the latent log-likelihood objective imposes stricter entropy constraints on the representation. On MNIST digit 7, stronger regularization produces near mode-collapsed reconstructions that retain fewer prototypes.

  • Higher log-likelihood regularization weights produce near mode-collapsed reconstructions for MNIST digit 7.The reconstructions lose sharp variations while capturing fewer prototypes from the input distribution.
  • Stricter entropy constraints trade sharp reconstruction variation for a smaller set of captured input-distribution prototypes.

12. On the complexity of autoregressive layers

The appendix discusses the parameter and computational complexity of masked autoregressive layers and provides qualitative reconstruction, density-fitting, and video-anomaly examples. The supplied passages do not state a quantitative complexity comparison or figure outcome.

  • Masked Fully Connected layers have d^2+d^2·ci·co+d·co trainable parameters and O(d^2·ci·co) computational complexity.
  • Figure 10 presents MNIST reconstructions under different λ values controlling the impact of differential-entropy minimization.
  • Table 4 reports train, validation, and test log-likelihoods for Bayesian Networks using autoregressive, random, and inverse orders on normal MNIST samples.
  • Masked Stacked Convolution layers have 3d^2+d^2·ci·co+d·co free parameters and O(d^2·ci·co·t) time complexity.
  • The complexity passage notes that type B masked layers provide an upper bound to type A layers.
  • Figure 11 presents novelty scores and localization maps for test clips from UCSD Ped2 and ShanghaiTech.
Loading 1807.01653v2…