PaperlayerView exact revision on arXiv

Source-linked AI summary

Fully Learnable Deep Wavelet Transform for Unsupervised Monitoring of High-Frequency Time Series

Gabriel Michau, Gaetan Frusque, Olga Fink

arXiv:2105.00899v2cs.LGcs.SDeess.AS

TL;DR

Industrial monitoring needs useful representations of high-frequency signals, yet deep-learning tools often use limited-size inputs and manually extracted features. This paper makes an FDWT-inspired network learn its filters and coefficient denoising without supervision. It reports results above baselines across three tasks, including a 16% performance improvement on MIMII.

  • Problem

    Industrial deep-learning pipelines often depend on manually extracted features, while extracting meaningful sparse representations from high-frequency signals without labels remains a challenge.

    Source: §1 · pp. 1–2

  • Method

    The paper builds a fully learnable, unsupervised FDWT-inspired framework that learns wavelet filters and coefficient denoising thresholds.

    Source: §1 · p. 2, §6 · p. 14

  • Results

    DeSpaWN outperforms baseline models across the evaluated tasks and reports a 16% performance improvement on MIMII.

    Source: §4.4.2 · p. 8, §6 · p. 14

  • Takeaways & Limitations

    The authors conclude that DeSpaWN provides a sparse representation of high-frequency signals without preprocessing or postprocessing and is competitive with other state-of-the-art approaches on three tasks.

    Source: §6 · p. 14

    Choosing suitable hard-thresholding parameters is difficult and usually depends on the use case and dataset.

    Source: §3.4 · p. 5

Abstract

from arXiv · show

High-Frequency (HF) signals are ubiquitous in the industrial world and are of great use for monitoring of industrial assets. Most deep learning tools are designed for inputs of fixed and/or very limited size and many successful applications of deep learning to the industrial context use as inputs extracted features, which is a manually and often arduously obtained compact representation of the original signal. In this paper, we propose a fully unsupervised deep learning framework that is able to extract a meaningful and sparse representation of raw HF signals. We embed in our architecture important properties of the fast discrete wavelet transformation (FDWT) such as (1) the cascade algorithm, (2) the conjugate quadrature filter property that links together the wavelet, the scaling and transposed filter functions, and (3) the coefficient denoising. Using deep learning, we make this architecture fully learnable: both the wavelet bases and the wavelet coefficient denoising are learnable. To achieve this objective, we propose a new activation function that performs a learnable hard-thresholding of the wavelet coefficients. With our framework, the denoising FDWT becomes a fully learnable unsupervised tool that does neither require any type of pre- nor post-processing, nor any prior knowledge on wavelet transform. We demonstrate the benefits of embedding all these properties on three machine-learning tasks performed on open source sound datasets. We perform an ablation study of the impact of each property on the performance of the architecture, achieve results well above baseline and outperform other state-of-the-art methods.

1 Introduction

Industrial monitoring uses high-frequency signals, but turning raw signals into compact, useful representations is difficult when labels are scarce and feature pipelines require careful manual tuning. The paper proposes a fully unsupervised, learnable FDWT-based framework and evaluates it on three machine-learning tasks.

1 Introduction

  • Industrial monitoring needs meaningful, sparse representations of high-frequency signals, often when labels are unavailable and decisions are low-dimensional.The motivating signals include electric current, vibrations, and sound.Source: §1 · p. 1
  • Deep-learning applications often rely on manually extracted features that require careful hyperparameter selection and may be sensitive to noise or changing conditions.Designing domain-invariant unsupervised features remains an open research question.Source: §1 · p. 2
  • Earlier spectral-inspired networks were often supervised, linked to spectral transforms mainly through architecture or initialization, and Fourier-based models can become heavy for long inputs.The paper positions unsupervised adaptation and input size as remaining challenges.Source: §1 · p. 2
  • The proposed FDWT-based framework learns decomposition filters and denoising thresholds, and the paper evaluates it on three tasks against baselines and other architectures.The framework uses a CQF structure and a learnable, continuous activation that approximates hard thresholding.Source: §1 · p. 2

2 Background on the Cascade and FDWT

The FDWT repeatedly decomposes a signal into coarse approximations and detail coefficients, then reconstructs it using linked filters. Its cascade, CQF filter relations, and coefficient thresholding provide the background for the learnable network.

2.1 Cascade algorithm

  • The FDWT cascade repeatedly splits each approximation into a lower-resolution approximation and detail coefficients, reusing the wavelet operation across levels with down-sampling and zero-interpolated reconstruction.With an orthonormal basis, the decomposition is invertible.Source: §2.1 · p. 2, Figure 1 · p. 3

2.2 CQF properties

  • CQF links wavelet, scaling, and reconstruction filters so their combined properties support perfect reconstruction, anti-aliasing, and anti-cancellation.The wavelet filter is an alternating flip of the scaling filter, while reconstruction filters are related through conjugation.Source: §2.2 · p. 3

2.3 Signal denoising with FDWT

  • Wavelet denoising typically thresholds coefficients before reconstruction, relying on structured signals being sparse while noise produces small-amplitude activations.Choosing suitable threshold parameters is difficult and has been extensively studied.Source: §2.3 · p. 3

3 Learnable Denoising Sparse Wavelet Network

DeSpaWN is a learnable cascade network that adapts wavelet filters and coefficient thresholds while retaining FDWT-inspired structure. Its objective balances reconstruction and sparsity, and CQF constraints link decomposition and reconstruction filters.

3.1 Architecture overview

  • DeSpaWN uses L encoding and L decoding blocks with learnable stride-two filters, thresholding layers, and skip connections from detail coefficients to reconstruction.Its parameter count is (kn + 2) · L; with Daubechies-4 filters, that is 10 · L parameters.Source: §3.1 · p. 4

3.2 Objective function

  • The objective balances reconstruction fidelity with sparse coefficients by minimizing ℓ1 norms for both the reconstruction error and wavelet coefficients.The loss averages the residual and coefficient terms to make them comparable.Source: §3.2 · p. 4

3.3 CQF-constrained architecture

  • CQF constraints derive reconstruction filters from decomposition filters, simplifying learning and mitigating solutions that obtain sparse latent coefficients through large reconstruction filters.The architecture does not impose additional orthonormality constraints.Source: §3.3 · pp. 4–5

3.4 Learnable Denoising

  • A learnable hard-thresholding activation makes denoising part of the network, with positive and negative thresholds learned for coefficients.The thresholds address a choice that is difficult and use-case dependent; α is fixed to 10 to sharpen the approximation.Source: §3.4 · p. 5, Figure 2 · p. 5

4.1 Machine Learning for Sound Data

Sound is an attractive industrial monitoring signal because it is relatively easy to deploy and may contain information about process state. However, factory noise and the difficulty of linking sounds to specific processes make robust automated analysis important.

4.1 Machine Learning for Sound Data

  • Sound monitoring is relatively simple and inexpensive to deploy, and machine operators may already recognize non-nominal operation by listening.Microphones are easy to install or retrofit and are non-intrusive.Source: §4.1 · p. 6
  • Factory environments introduce high noise levels, while relating recorded sounds to specific industrial processes is difficult.The paper identifies automatic, noise-robust sound analysis as valuable for industrial applications.Source: §4.1 · p. 6

4.2 Classification and Anomaly Detection

DeSpaWN is trained as a robust autoencoder using denoising and sparsity, then its reconstruction and coefficient statistics support classification and anomaly detection.

4.2 Classification and Anomaly Detection

  • Denoising learned coefficients and encouraging sparsity are intended to produce similar sparse latent representations and low reconstruction residuals for similar signals.These properties underpin the paper’s classification and anomaly-detection strategies.Source: §4.2 · p. 6
  • For classification, the method trains one autoencoder per class and assigns a sample to the class whose autoencoder minimizes the objective loss.The paper relates this approach to dictionary learning.Source: §4.2 · p. 6
  • For anomaly detection, the method combines average and maximum residuals with average and maximum per-level coefficient sparsity statistics before applying a one-class classifier.The paired statistics are intended to capture both global trend changes and local intermittent anomalies.Source: §4.2 · p. 6
  • The reported anomaly-detection results use a one-class ELM with 50 neurons; Isolation Forest, one-class SVM, and Elliptic Envelope gave similar results.Source: §4.2 · p. 6

4.3 Datasets

The experiments use industrial-machine and birdsong audio to evaluate anomaly detection and classification, with the machine recordings spanning operating conditions and noise levels.

4.3 Datasets

  • The study uses MIMII for industrial-machine anomaly detection and birdsong for classification and anomaly detection, and examines the resulting latent spaces for interpretability.Source: §4.3 · p. 6

4.3.1 Malfunctioning Industrial Machine Detection

  • MIMII covers valves, pumps, fans, and slide rails in normal and malfunctioning states; its unlabelled operating states and faults support unsupervised anomaly detection.Source: §4.3.1 · pp. 6–7
  • MIMII recordings cover four individual machines across three SNR settings—0, 6, and -6 dB—yielding 48 anomaly-detection experiments.The audio was recorded at 16 kHz and 16-bit resolution in 10-second samples; the first microphone was used.Source: §4.3.1 · p. 7

4.3.2 Bird Song Dataset

  • The birdsong data allow evaluation in both classification and anomaly detection, with recordings of five species selected to share a recording source and similar recording conditions.Source: §4.3.2 · p. 7
  • Birdsong recordings were decimated by four, split into approximately 24-second signals, and evaluated using five-fold cross-validation.The reported signal counts are 30 for Corn Bunting, 24 for Eurasian Skylark, 21 for Barn Swallow, and 20 each for Sedge Warbler and Common Nightingale.Source: §4.3.2 · pp. 7–8

4.4 Ablation Study

Across the three evaluated tasks, DeSpaWN outperforms literature baselines; ablations examine sparsity, threshold learning, wavelet learning, and filter constraints.

4.4.1 From FDWT to DeSpaWN

  • The ablation compares fixed db4, learnable hard-thresholding, shared and layer-wise CQF filters, layer-wise wavelet-plus-scaling learning, and unconstrained kernels.The study presents these variations as steps from the traditional FDWT toward DeSpaWN.Source: §4.4.1 · p. 8

4.4.2 Results and discussion

  • 16%: DeSpaWN's global MIMII performance improvement; its performance drops 4% from 6 to 0 dB versus the baseline's 9%, and remains +16% over baseline at -6 dB.Across all three tasks, DeSpaWN significantly outperforms the literature baselines.Source: §4.4.2 · p. 8, Table 2 · p. 9
  • Increasing the sparsity weight can strongly affect performance, while setting γ to one without fine-tuning is reported as near-optimal.The authors note that excessive sparsity trades off against reconstruction loss and the ability to distinguish signal variations, including anomalies and class-specific coefficient behaviours.Source: §4.4.2 · p. 9
  • Architectures without learned hard-thresholding perform significantly worse than their thresholded counterparts, whether the wavelets are fixed or learned.The authors identify learned coefficient thresholds as the denoising component that provides robustness to small signal variations.Source: §4.4.2 · p. 9
  • Wavelet learning is pivotal for classification accuracy; for anomaly detection, learning the wavelets adds a reported 1–2% improvement averaged over several tens of experiments.The authors attribute the smaller anomaly-detection effect to CQF wavelets already providing good reconstruction and relatively sparse representations.Source: §4.4.2 · p. 9
  • The figures show Slide Rail latent-space clusters across machines and SNRs, normal and abnormal signal examples, and learned layer filters with hard-threshold ranges.Source: Figure 3 · p. 10, Figure 4 · p. 10, Figure 5 · p. 11
  • 52%: db4+HT classification accuracy; DeCWN reaches 84.9%, while DeSpaWN and DeSpaWN-2 reach 97.9% and 97.4%, respectively.The most constrained option performs worst; leaving all kernels free also reduces performance, likely because of training instabilities.Source: §4.4.2 · p. 10

4.5 Additional Diagnostics Potential

DeSpaWN’s latent representations separate operating and anomaly conditions in MIMII and generally separate bird species in the bird-song task. Learned coefficient patterns and filters offer additional cues for distinguishing signal conditions.

4.5.1 Insights on MIMII

  • MIMII latent-space clusters separate normal operating conditions from anomalies, with some anomalous points becoming less distinct at -6 dB SNR.The clusters are visualized with t-SNE; the reduced separability is consistent with the lower AUC reported for that noise level.Source: §4.5.1 · p. 12
  • Healthy MIMII signals show different coefficient profiles, including a dominant-level shift from level 10 to 12, corresponding to a fourfold spectral change.Unhealthy examples also differ: one activates level 12 more than healthy signals, while another has much larger high-level coefficients.Source: §4.5.1 · p. 12
  • Learned filters and thresholds vary by scale: high-frequency layers have thresholds up to 0.3 and near-zero coefficients, while lower-frequency filters depart further from Daubechies wavelets.The authors interpret the higher-frequency layers as likely noise-dominated and lower frequencies as containing information needed for reconstruction.Source: §4.5.1 · p. 12

4.5.2 Insights on the Bird Song Dataset

  • In bird-song one-class detection, the training species generally forms a distinct cluster, but Corn Bunting mixes with Sedge Warbler in one representation.The authors connect this exception to the classification-accuracy drop when Sedge Warbler is added as the fifth species.Source: §4.5.2 · p. 12

5 Comparison to other frameworks

The benchmark compares DeSpaWN with scattering transforms, convolutional autoencoders, and U-nets across anomaly-detection and classification tasks. DeSpaWN and scattering are competitive overall, while performance varies by machine and noise level.

5.1 Scattering Transform, U-Net and Auto-Encoders

  • The comparison uses two scattering settings, four CAE variations, and four U-net variations, with shared training loss and task procedures for the CNN comparisons.The scattering comparison also uses an SVM for Bird Song classification; the CNN variants vary kernel sizes, and the U-net differs from DeSpaWN in filters, initialization, and activation.Source: §5.1 · pp. 12, 14
  • Figure 7 compares birds in the Res −ℓ1 space across training-species choices; the trained species tends to be more compact than when evaluated with another DeSpaWN.Source: Figure 7 · p. 13

5.2 Results of the Benchmark

  • All approaches outperform the reported baselines, while autoencoders are significantly less performing overall than scattering and DeSpaWN; the benchmark covers three tasks.The Table 3 caption identifies average AUC (%) for the MIMII and Bird Song anomaly-detection tasks.Source: Table 3 · p. 13, §5.2 · p. 14
  • Scattering performance drops by 20% and 16% when MIMII noise increases from 6 dB to -6 dB.DeSpaWN and scattering are both described as competitive; the stated drops concern scattering.Source: §5.2 · p. 14
  • The strongest approach depends on machine type: scattering appears most adapted to the pump and slider, while DeSpaWN performs better on the valve and fan.Source: §5.2 · p. 14
  • Bird Song classification results are very similar across the compared approaches.Source: §5.2 · p. 14

6 Conclusion

The paper concludes that DeSpaWN learns sparse representations of raw high-frequency signals without feature-extraction or denoising post-processing. Its results are competitive on sound classification and anomaly detection, and the authors identify further directions for studying and extending the approach.

6 Conclusion

  • DeSpaWN is a fully learnable FDWT-inspired architecture with learnable wavelets and hard-thresholding for automatic denoising.The paper reports competitive results above baselines on three machine-learning tasks involving anomaly detection and classification.Source: §6 · p. 14
  • The framework accepts raw high-frequency data and combines wavelet properties with deep learning using only a few hundred learnable parameters.The authors state it can replace preprocessing such as spectrogram or wavelet-coefficient extraction and provide diagnostic information to domain experts.Source: §6 · p. 14
  • Future directions include degradation monitoring, controlled-noise analysis, stacked supervised architectures, parallel wavelets, and multichannel inputs.The authors suggest that a drop in decomposition sparsity could indicate increased signal complexity or disturbing components associated with system wear.Source: §6 · p. 15
Loading 2105.00899v2…