PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for Audio Signal Processing

Hendrik Purwins, Bo Li, Tuomas Virtanen, Jan Schlüter, Shuo-yiin Chang, Tara Sainath

arXiv:1905.00078v2cs.SDeess.ASstat.ML

TL;DR

Audio processing lacks a single universally established deep-learning approach across speech, music, and environmental sound, while domain-specific signal structure, data availability, and task objectives complicate method selection. This article reviews representations, models, applications, and data practices across these domains, identifying shared methods and open questions. It concludes that model superiority depends on the setting, datasets are unevenly available, and further learning paradigms may be needed for scarce labeled data.

  • Problem

    Audio processing spans distinct signal structures, task types, and domains, while the relative suitability of deep-learning models and the availability of labeled data remain unresolved.

  • Method

    The article reviews audio features, deep-learning architectures, data, evaluation, and applications across speech, music, and environmental sound processing.

  • Results

    The survey finds that different research groups achieve state-of-the-art results with different architectures, making model superiority difficult to determine across settings.

  • Takeaways & Limitations

    Audio research can draw on shared deep-learning methods while developing audio-specific solutions and exploring semisupervised, active, or few-shot learning for scarce labeled data.

Abstract

from arXiv · show

Given the recent surge in developments of deep learning, this article provides a review of the state-of-the-art deep learning techniques for audio signal processing. Speech, music, and environmental sound processing are considered side-by-side, in order to point out similarities and differences between the domains, highlighting general methods, problems, key references, and potential for cross-fertilization between areas. The dominant feature representations (in particular, log-mel spectra and raw waveform) and deep learning models are reviewed, including convolutional neural networks, variants of the long short-term memory architecture, as well as more audio-specific neural network models. Subsequently, prominent deep learning application areas are covered, i.e. audio recognition (automatic speech recognition, music information retrieval, environmental sound detection, localization and tracking) and synthesis and transformation (source separation, audio enhancement, generative models for speech, sound, and music synthesis). Finally, key issues and future questions regarding deep learning applied to audio signal processing are identified.

I. INTRODUCTION

Deep learning has renewed neural-network methods for audio, but audio's one-dimensional, sequential structure and heterogeneous time-frequency axes require domain-specific approaches. The survey organizes audio tasks by target-label structure and reviews analysis, synthesis, and transformation problems across speech, music, and environmental sound.

  • Motivation: Deep learning renewed neural-network research, with DNNs, CNNs, and LSTMs becoming prominent after major successes in speech recognition and image classification.Earlier audio methods, including Gaussian mixture models, hidden Markov models, and non-negative matrix factorization, were often outperformed when sufficient data was available.
  • Task organization: Audio tasks are categorized by the number of labels predicted and by whether labels are classes, class sets, or numeric values.Targets may be global, time-local, or free-length sequences, yielding sequence classification, labeling, event detection, regression, and transduction tasks.
  • Motivation: Audio differs from images because raw signals are one-dimensional time series processed sequentially, while time and frequency axes are not homogeneous.These properties motivated audio-specific solutions even when methods were adapted from image processing.
  • Task organization: Sequence classification predicts one global class, multi-label classification predicts a subset of classes, and sequence regression predicts a continuous value.Examples include language, speaker, musical key, acoustic scene, acoustic events, and musical pitches.
  • Task organization: Sequence labeling and event detection predict time-local outputs, whereas sequence transduction produces a variable-length output sequence unrelated to input length.Examples include chord annotation, vocal activity detection, speaker changes, note onsets, speech-to-text, music transcription, and translation.
  • Task organization: The survey also treats audio synthesis and similarity estimation as regression or transduction problems involving predicted samples or continuous similarity values.Audio synthesis can condition outputs on variables, while similarity compares potentially different-length audio signals.

B. Audio Features

Deep audio systems increasingly learn representations jointly with task objectives rather than separating feature design from classification. Log-mel and constant-Q spectrograms remain important, while raw-waveform and learned-filter approaches reduce reliance on hand-designed features.

  • Representation learning: Deep neural networks jointly perform feature extraction and objective optimization, avoiding a strict separation between representation design and classifier construction.This can produce task-adapted lower-layer features, including speaker-adapted representations in speech recognition.
  • Spectral features: Log-mel spectra omit the DCT used in MFCCs, preserving information and spatial relations that deep models can exploit.MFCCs historically dominated audio analysis, but log-mel spectra became popular across audio domains with deep learning.
  • Spectral features: Constant-Q spectra use logarithmic frequency spacing so pitch transpositions appear as translations, which can suit tasks requiring transposition-aware representations.The representation is built with a filter bank designed around this frequency scale.
  • Spectral features: Spectrograms are temporal sequences of spectra whose neighboring time-frequency bins and harmonic frequencies are correlated.Harmonic-series channels and per-band standardization can help spatially local models account for these audio-specific properties.
  • Spectral features: Spectral window size trades temporal resolution against frequency resolution, motivating multi-window or multi-channel feature combinations.Short windows improve temporal resolution, whereas long windows improve frequency resolution.
  • Learned representations: Raw-waveform and data-driven-filter methods learn task-optimized front ends instead of relying on designed filter banks.Raw waveforms avoid hand-designed features but require more computation and data, and their practical benefits may be difficult to realize.

C. Models

The survey covers convolutional, recurrent, sequence-to-sequence, and generative models for audio, emphasizing how architectures process local structure, long temporal context, and input-output alignments. It also highlights task-specific loss design and unresolved architecture-selection and phase-modeling issues.

  • Models: Audio systems commonly stack feedforward, convolutional, and recurrent layers over raw audio, feature vectors, spectrograms, or higher-dimensional tensors.Increasing depth provides greater modeling capacity across audio tasks.
  • Convolutional models: CNNs apply learnable kernels to temporal, time-frequency, or raw-waveform inputs, with pooling used to downsample feature maps.Their receptive field is fixed by architecture and can be expanded with larger kernels, deeper stacks, or dilation.
  • Design issues: CNN architecture selection lacks operationally validated theories, and magnitude-only phase estimation remains insufficient for high-quality synthesis and transformation.These unresolved issues affect kernel, pooling, feature-map, and phase-modeling choices.
  • Recurrent models: RNNs model temporal dependencies through recurrent hidden states, extending effective context into the past; LSTMs use gates and memory cells to mitigate gradient problems.Bidirectional and convolutional-recurrent designs combine forward and backward context or local convolutional features with longer temporal integration.
  • Recurrent models: Frequency LSTMs and time-frequency LSTMs extend recurrent modeling across frequency, offering alternatives to CNNs for input-feature correlations.These models can capture translational invariance through local filters and recurrent connections without pooling.
  • Sequence-to-sequence models: Sequence-to-sequence models directly transduce input sequences into output sequences, including fully neural ASR systems using CTC, recurrent transducers, or attention-based alignment.CTC inserts a blank symbol and sums over possible alignments, while attention learns input-output correspondences.
  • Generative models: GANs generate realistic samples from random latent vectors through adversarial training between a generator and discriminator.The generator maps latent vectors to samples, while the discriminator distinguishes real from generated data.
  • Design issues: Loss functions can compare spectral or temporal structures and be tailored to application goals such as source-separation intelligibility.Time-domain sample MSE is not robust to phase differences, motivating measures such as differentiable dynamic time warping or earth mover's distance.

D. Data

Audio deep learning benefits from large datasets, but dataset availability and labeling vary substantially across speech, music, and environmental sound. Transfer learning, augmentation, and synthetic data address limited training data, with real-data performance remaining an important boundary.

  • Dataset availability: Deep learning is most profitable with large training datasets, yet no well-labeled dataset spans speech, music, and environmental sounds.Available resources differ by domain, with large speech and environmental-sound datasets but smaller datasets for several music-labeling tasks.
  • Limited labeled data: Transfer learning adapts models trained on large related datasets to target tasks with limited labeled data.Adaptation can retrain final layers or fine-tune weights using a small learning rate.
  • Limited labeled data: Data augmentation expands training coverage by manipulating existing examples, including pitch shifting, time stretching, filtering, mixing, and room simulation.These techniques have been applied across ASR, chord recognition, singing-voice detection, instrument recognition, environmental sounds, and far-field speech.
  • Limited labeled data: Synthetic data can provide known labels and controlled complexity, but training only on generated data may perform poorly on real data.For source separation, models can be trained successfully on datasets synthesized by mixing separated tracks.

E. Evaluation

Evaluation criteria vary across audio tasks, with WER commonly used for speech recognition and accuracy commonly used in music and acoustic scene classification.

  • Speech recognition is usually evaluated with word error rate, which counts insertion, deletion, and substitution errors after aligning reference and hypothesis word strings.WER divides these error counts by the number of reference words.
  • Accuracy is a commonly used evaluation metric for both music tasks and acoustic scene classification.

A. Analysis

Deep learning analysis spans speech, music, and environmental sound, adapting neural architectures to tasks ranging from recognition and event detection to sequence labeling and tempo estimation.

  • Speech: Deep neural networks have driven major advances in speech recognition, with DNNs reducing word error rates and CLDNNs outperforming LSTM-only models.CLDNNs combine convolutional processing, temporal modeling with LSTMs, and feedforward output layers.
  • Speech: Speech modeling shifted toward neural sequence-to-sequence systems such as CTC and LAS, removing the need for conditional independence assumptions in traditional HMM-based phone-state modeling.LAS combines an encoder, attention-based alignment module, and decoder in one neural network.
  • Music: Deep learning is applied across music analysis, including onset, rhythm, harmonic, instrument, transcription, structural, similarity, and classification tasks.
  • Music: Neural onset detection progressed from MLPs on spectrogram excerpts to bidirectional LSTMs and CNNs, with later systems directly tracking beats and downbeats.
  • Music: CNNs can directly estimate global tempo from 12-second spectrogram excerpts, avoiding reliance on beat or downbeat tracking as an intermediate step.
  • Environmental Sounds: Environmental sound analysis commonly includes acoustic scene classification, acoustic event detection, and tagging.These approaches address whole-recording labels, temporally localized events, and sound-class activity, respectively.

3) Environmental Sounds:

Environmental sound processing covers scene classification, event detection, tagging, and multichannel localization, while remaining constrained by smaller and less diverse datasets than speech and music.

  • Acoustic scene classification assigns a single predefined scene label to an entire recording.
  • Acoustic event detection estimates the start and end times of predefined sound events and assigns each event an appropriate label.
  • Tagging predicts activity for multiple sound classes without temporal information, allowing simultaneous classes in multilabel classification.
  • Joint multilabel prediction produces better results than separate single-class classifiers, possibly because it models interactions among simultaneously active classes.
  • Environmental sound datasets are smaller and less diverse than speech and music datasets, making data augmentation common and highly effective.
  • Multichannel audio supports sound-source localization and tracking, with uses in source separation, speech enhancement, and speaker diarization.

4) Localization and Tracking:

Audio localization and source separation use deep models to infer spatial information or isolate sources from mixtures. Approaches operate on spectral, mask-based, and interchannel representations, with domain-specific assumptions across speech, music, and environmental sounds.

  • Localization and Tracking: Microphone arrays estimate source direction using grid-based multilabel classification or direct coordinate regression, and combine array directions to infer locations.Directions may represent azimuth alone or both azimuth and elevation.
  • Localization and Tracking: Localization models use phase, magnitude, or generalized cross-correlation features, while convolutional kernels can learn interchannel information.The kernels may span multiple microphone channels to model relationships between them.
  • Source Separation: Source separation extracts individual source signals from mixtures, supporting music editing, robust classification, and improved speech intelligibility.Multiple sources commonly sum into a mixture signal and can negatively affect downstream processing.
  • Source Separation: State-of-the-art separation typically estimates time-frequency masks, although some systems learn representations directly from time-domain signals.Time-frequency structure makes natural sound sources easier to model.
  • Source Separation: The mixture spectrum is commonly computed with the STFT because it is efficient and readily invertible; separation then multiplies it by a source mask.Constant-Q and mel spectrograms are also possible but have become less common because they reduce output quality.
  • Source Separation: Single-channel systems predict either separation masks or source spectra, while deep clustering and attractor networks support separation of sources absent from training data.Deep attractor networks estimate one attractor vector per source.
  • Audio Enhancement: Speech enhancement reduces noise for improved speech quality and ASR robustness, using networks that reconstruct clean speech or estimate masks.Speech, environmental sounds, and music differ in their source assumptions: independence is usually assumed for environmental sounds, while music has strong simultaneous and temporal dependencies.

2) Audio Enhancement:

Deep learning enhancement methods translate noisy speech toward clean speech or estimate enhanced representations. GAN-based enhancement can improve perceptual quality over noisy inputs and traditional baselines, but ASR gains are not guaranteed.

  • Audio Enhancement: SEGAN improves perceptual speech-quality metrics over noisy speech and a traditional enhancement baseline.The task is posed as translation from noisy signals to clean ones in additive noise.
  • Audio Enhancement: GAN enhancement of log-mel spectra produces no ASR improvement compared with a simpler regression-based enhancement approach.This result distinguishes perceptual speech quality from downstream recognition performance.
  • Generative Models: Generative sound models learn database characteristics such as timbre, pitch content, and rhythm while targeting recognizable or intelligible outputs.The models are also expected to produce original sounds and may be conditioned on speakers, prosody, musical structure, or physical parameters.

3) Generative Models:

Generative audio models synthesize conditioned sounds from spectral or raw-audio representations and are evaluated for recognizability, diversity, originality, and realism. WaveNet is reported to achieve higher MOS than earlier concatenative and parametric methods.

  • Generative Models: Sound synthesis can use log-mel spectrograms or raw audio, with conditioning on speakers, prosody, harmonic structure, or physical parameters.Spectral representations lack phase information, which must be reconstructed for audio generation.
  • Generative Models: Generative audio is evaluated through recognizability, diversity, originality, and human discrimination between real and synthesized examples.Recognizability can use classifiers or forced-choice tests, while diversity and originality can use distances to real-sound neighbors.
  • Generative Models: WaveNet yields a higher MOS than concatenative or parametric methods, which previously represented the state of the art.A Turing test is described as demanding because passing requires no perceivable difference between real and synthesized audio.
  • Cross-domain perspective: The review compares deep learning across audio domains through features, models, data requirements, computational complexity, interpretability, and adaptability.It highlights cross-domain similarities, differences, and shared challenges.

A. Features

Audio deep learning commonly uses log-mel spectrograms, raw waveforms, or complex spectra, while models learn representations jointly with task objectives. Representation choice remains unresolved because raw waveforms can improve analysis but require more computation and data.

  • Features: Log-mel spectrograms are the dominant deep-learning audio feature, followed by raw waveforms and complex spectrograms.Raw waveforms avoid hand-designed features but incur higher computational costs and data requirements.
  • Features: Raw-waveform studies report improvements on some analysis tasks, while other systems learn intermediate representations that mimic engineered features.Open questions concern when raw waveforms are preferable and whether learned representations generalize across tasks or domains.
  • Models: CNNs, RNNs, and CRNNs are all used successfully across audio domains, with no clear preference for one architecture.CNN context is fixed but adjustable, whereas RNNs can theoretically use unlimited temporal context.
  • Models: Which model is superior in a given setting remains open because different research groups obtain state-of-the-art results with different architectures.The review suggests specialized design and tuning knowledge may contribute to these differences.
  • Data Requirements: Audio datasets are relatively small for most domains and tasks, limiting the size and complexity of trainable deep models except possibly in widespread-language speech recognition.There is no single well-labeled dataset spanning speech, music, and environmental sounds.
  • Data Requirements: Audio lacks an ImageNet-like cross-domain resource for transfer learning, motivating questions about shared datasets, flexible adaptation, and learning from scarce labels.Potential alternatives include semisupervised, active, and few-shot learning.

D. Computational Complexity

Deep neural networks achieve state-of-the-art performance at the cost of greater computational demands and training-data requirements. Real-time audio applications with constrained hardware therefore require smaller, more efficient models and audio-specific solutions.

  • State-of-the-art deep neural networks generally require more computation power and training data than conventional approaches.
  • CPUs are not optimally suited to training and evaluating large deep models, so GPGPUs and application-specific integrated circuits are commonly used.
  • Mobile phones and hearing instruments require smaller models because they operate under strict computational-resource limits.
  • Neural-network parameters are typically learned by gradient descent on a task loss, but the relationship between layer parameters and the actual task is difficult to interpret.
Loading 1905.00078v2…