PaperlayerView exact revision on arXiv

Source-linked AI summary

Spectrogram features for audio and speech analysis

Ian McLoughlin, Lam Pham, Yan Song, Xiaoxiao Miao, Huy Phan, Pengfei Cai, Qing Gu, Jiang Nan, Haoyu Song, Donny Soh

arXiv:2603.14917v1eess.AScs.AIcs.LGeess.SP

TL;DR

Spectrograms are widely used for deep-learning audio and speech analysis, but their many resolution and representation settings raise questions about how to match features with classifier architectures and tasks. This paper surveys these choices and application areas; its reviewed evidence indicates that feature pooling can help tested systems, while classifier architecture remains more important than feature optimization.

  • Problem

    The paper addresses how spectrogram resolution and representation choices align with back-end classifier architectures across audio and speech tasks.

    Source: §3.1.1 · p. 10, §5.1 · p. 22

  • Method

    The paper surveys spectrogram-based feature choices and their use across audio and speech analysis tasks.

    Source: §5 · p. 21

  • Results

    In most tested systems, variance-normalized feature pooling outperformed mean or max pooling, but could not compensate for classifier architecture deficiencies.

    Source: §2.10 · p. 9

  • Takeaways & Limitations

    Choosing a good classifier architecture is more important than optimizing features, though variance-normalized pooling can help once the architecture is good.

    Source: §2.10 · p. 9

    In bioacoustics, fixed-size spectrogram windows can truncate short calls or blend closely spaced vocalizations, and PCEN does not fully resolve these challenges.

    Source: §3.4 · p. 16

Abstract

from arXiv · show

Spectrogram-based representations have grown to dominate the feature space for deep learning audio analysis systems, and are often adopted for speech analysis also. Initially, the primary motivator for spectrogram-based representations was their ability to present sound as a two dimensional signal in the time-frequency plane, which not only provides an interpretable physical basis for analysing sound, but also unlocks the use of a wide range of machine learning techniques such as convolutional neural networks, that had been developed for image processing. A spectrogram is a matrix characterised by the resolution and span of its two dimensions, as well as by the representation and scaling of each element. Many possibilities for these three characteristics have been explored by researchers across numerous application areas, with different settings showing affinity for various tasks. This paper reviews the use of spectrogram-based representations and surveys the state-of-the-art to question how front-end feature representation choice allies with back-end classifier architecture for different tasks.

1. Introduction

Spectrograms transform a one-dimensional waveform into a two-dimensional time-frequency image, first valued for visualization and later shown to support speech recognition.

1. Introduction

  • A spectrogram transforms a one-dimensional sound waveform into a two-dimensional image of sound in the time-frequency plane.Its elements represent sound energy across time and frequency.Source: §1 · p. 1
  • The spectrogram was initially generated by a sound spectrograph machine as a stylus-on-paper plot for visualizing sound energy.It is considered to have been invented at Bell Labs in the 1940s.Source: §1 · p. 1
  • By the late 1970s, spectrograms had become established in phonetics and acoustics, and Zue and Cole demonstrated their use for speech recognition.Source: §1 · p. 1

2. Taxonomy of spectrograms

Spectrograms represent sound through stacked frequency vectors, with time-frequency dimensions and element scaling defining a range of possible variants.

2. Taxonomy of spectrograms

  • A spectrogram is a two-dimensional representation whose axes and pixel intensities encode frequency, time, and frequency strength.Linear, nonlinear, and task-tuned representations have all been used.Source: §2 · p. 2

2.1. Basic spectrogram

  • An auditory spectrogram stacks frequency vectors from successive, windowed audio frames, commonly 10–30 ms long, with overlap helping include tapered edge regions.Frame length limits frequency resolution, while the step between frames sets time-axis resolution.Source: §2.1 · p. 2
  • For deep-learning inputs, frequency pooling or downsampling can reduce the frequency dimension after frequency vectors are stacked.The paper refers to this operation as a later topic in its discussion of pooling.Source: §2.1 · pp. 2–3
  • Spectrogram variants include linear, log-, A-law-, µ-law-, Mel-, and log-Mel-scaled forms, alongside other transforms catalogued in Table 1.The taxonomy compares their dimensions, element scaling, and frequency span.Source: §2.1 · p. 3, Table 1 · p. 3

2.2. Spectrograms are not pictures

Spectrograms can use image-processing methods, but their physical meaning differs from ordinary pictures in color, axis behavior, scaling, and local correlations.

2.2. Spectrograms are not pictures

  • Spectrograms support image-processing methods, but their axes have distinct physical meanings that make picture-image assumptions unsafe.Time and frequency relationships differ, and frequency translations can change the represented sound.Source: §2.2 · p. 3, §2.2.2 · p. 4

2.2.1. Colour and greyscale

  • RGB color mapping depends arbitrarily on spectrogram type and software, while three-channel CNN input can raise front-end complexity compared with greyscale.The passage finds no logical justification for color spectrogram processing.Source: §2.2.1 · p. 3

2.2.2. Translation invariance and scaling

  • Scaling a spectrogram structure changes both event duration and frequency span, potentially changing the sound it represents.Training and inference require careful matching of scale.Source: §2.2.2 · p. 4

2.2.3. Local features

  • Local correlations across spectrogram time may resemble frame-to-frame differences in video more than relationships between nearby picture pixels.Source: §2.2.3 · p. 4

2.3. Mel-spectrogram

Mel spectrograms apply a nonlinear frequency scale based on human auditory perception, and the resulting features are used in speech analysis and other filter-bank representations.

2.3. Mel-spectrogram

  • The Mel scale maps linear Hertz frequencies to a nonlinear scale based on human auditory perception.Mel spectrograms stack Mel-frequency vectors from successive analysis frames.Source: §2.3 · p. 4
  • Triangular bandpass filters convert each STFT power spectrum into a Mel-scale vector by accumulating weighted power across frequency.Filter centers are set at Mel-scale frequencies.Source: §2.3 · p. 4
  • Mel-based features have been used in state-of-the-art systems for speaker identification, speech-to-text, and speech emotion detection.Source: §2.3 · p. 4
  • Mel-inspired alternatives include Gammatone filters based on cochlear simulation and Nearest Neighbour filters inspired by image preprocessing.Source: §2.3 · p. 5

2.4. Constant-Q spectrogram

The Constant-Q transform uses geometrically spaced filter-bank centre frequencies with a constant ratio of centre frequency to bandwidth; setting its parameters to musical-note frequencies makes it useful for musical analysis. The correlogram instead represents audio segments as autocorrelation vectors indexed by time lag.

2.4. Constant-Q spectrogram

  • Geometrically spaced CQT filter-bank centres have a constant Q value, the ratio of centre frequency to bandwidth.The number of filters per octave and minimum frequency determine the centre frequencies.Source: §2.4 · p. 5
  • Setting CQT parameters to musical-note frequencies makes its spectrogram effective for capturing musical tones and widely used in musical analysis.For example, b = 12 and fmin can be set to the frequency of a chosen low MIDI note; the representation is often converted to a rectangular matrix before use.Source: §2.4 · p. 5

2.5. Correlogram

  • The correlogram represents each audio segment with autocorrelation coefficients across time lags, forming a matrix whose columns correspond to lags.Its structure is similar to the stabilised auditory image.Source: §2.5 · p. 5

2.6. Stabilised auditory image

The stabilised auditory image (SAI) integrates auditory-image information over time and represents frequency against lag or periodicity within a fixed time window. Spectrogram patches are useful for deep-learning classification, but unlike image-object patches, their use lacks a general physical justification.

2.6. Stabilised auditory image

  • The SAI integrates the auditory image over time, aiming to improve noise robustness and periodicity detection compared with the auditory image model.The auditory image model simulates hearing-related frequency discrimination and amplitude sensitivity; the SAI was formed through strobed temporal integration.Source: §2.6 · p. 5
  • An SAI represents frequency against lag or periodicity within a fixed time window, rather than displaying frequency against time.Figure 2 illustrates an example with a 35 ms window.Source: §2.6 · p. 5, Figure 2 · p. 6
  • Early deep-learning sound-event detectors using SAIs were outperformed by linear spectrogram equivalents, probably because a single SAI represents only a short time window.SAIs had also been used in early audio-recall systems with a pre-deep-learning ordering algorithm based on SAI-region statistics.Source: §2.6 · p. 6

2.7. Patches and regions

  • For spectrogram classification, randomly sized and located image patches lack physical justification; fixed-duration slices spanning the full frequency range are an exception.The passage reports limited benefit from varying time-window duration and notes that fixed-size regions are used in AST.Source: §2.7 · p. 6

2.8. Scaling and number representation

Spectrogram values may be represented as positive magnitudes and perceptually emphasized with logarithmic encoding; µ- or A-law can instead reduce downstream computation at the cost of greater quantization noise.

2.8. Scaling and number representation

  • Spectrograms usually use positive magnitude-spectrum values, which can be scaled to [0 : 1], and often use log encoding for perceptually relevant emphasis.The passage contrasts magnitude spectra with the complex output of a true Fourier transform.Source: §2.8 · p. 7
  • µ- or A-law can convert samples to 8-bit fixed-point values in [0, 255], reducing downstream computational complexity but increasing quantization noise.The input samples are typically converted from 16-bit signed fixed-point values to 32-bit floating point and divided by 2^15 to scale them to [−1 : 1).Source: §2.8 · pp. 6–7

2.9. Pooling and downsampling

Deep-learning systems commonly transform spectrogram inputs into embeddings and classify them after pooling or downsampling, while speech features may also incorporate temporal-delta context. Earlier systems classified statistics extracted from spectrogram regions.

2.9. Pooling and downsampling

  • Deep-learning front ends produce fixed-dimension embeddings from input features, whose time-stepped sequence can reveal signal variation and support classification.The passage describes this as a general pattern across sound-event detection, audio analysis, language identification, speaker verification, and speech emotion recognition.Source: §2.9 · p. 7
  • Before deep learning, systems classified statistics extracted from spectrogram regions or SAI rows and columns.One example divided spectrograms into nine equal-sized regions and used zero- and first-order statistics with an SVM.Source: §2.9 · p. 7
  • Direct neural classification of raw spectrograms has commonly used downsampling or frequency-bin pooling rather than full-resolution input.These approaches were used from the first DNN spectrogram classifier onward.Source: §2.9 · p. 7
  • MFCC, PLP, and filterbank features are commonly extracted from overlapping frames and stacked into time-frequency blocks, with frequency pooling or prior downsampling possible.These non-spectrogram feature representations are used in tasks including language identification.Source: §2.9 · p. 7
  • Frequency-axis pooling or downsampling typically averages 2, 4, or 8 neighboring magnitudes; MFCC processing may also use max-pooling or standard deviation.MFCCs can be concatenated with delta and delta-delta coefficients, while shifted delta cepstral coefficients expand classifier context; comparable delta processing is used for filterbanks and PLPs.Source: §2.9 · pp. 7–8

2.10. Variance normalised features

Variance normalised features (VNF) replace fixed-width spectral pooling with data-driven partitions designed to equalise variance contributions. Across three tasks, VNF usually outperformed standard pooling, but did not compensate for classifier-architecture deficiencies.

2.10. Variance normalised features

  • The method uses a development set to choose pooling rules that maximise Fisher’s criterion, comparing between-class and within-class variance.VNF is applied instead of fixed-size pooling; during operation, each partition is represented by its mean spectral magnitude.Source: §2.10 · pp. 8–9
  • VNF varies spectral pooling-region sizes according to between- and within-class variance, aiming to give each downsampled feature point an approximately equal variance contribution.Partitions are computed iteratively from development-set data to approach equal variance contributions while preserving the total variance budget.Source: §2.10 · pp. 8–9
  • In most tested systems across sound event detection, language identification, and dialect identification, VNF pooling outperformed standard mean or max pooling, but could not compensate for architectural deficiencies.The authors state that using a good classifier architecture matters more than optimising features; VNF may improve results once that architecture is in place.Source: §2.10 · p. 9, Table 2 · p. 9
  • VNF may benefit systems that mean- or max-pool spectral bins before classification, provided a representative development dataset is available for the one-time partition-setting step.Source: §2.10 · p. 9

3. Audio analysis

The section surveys audio-analysis tasks and the feature-to-classification pipeline, then reviews how overlap, early detection, sound-event detection, anomalous-sound detection, and bioacoustics shape task-specific choices.

3. Audio analysis

  • Audio analysis refers to detecting and classifying human-audible sounds, and the section treats it mainly as non-speech while noting overlap with speech analysis.The section surveys a taxonomy and application areas including sound-event detection, anomalous-sound detection, and bioacoustics.Source: §3 · p. 9

3.1. Taxonomy of audio analysis

  • Audio classification commonly follows a sequence from an input of task-dependent duration through feature extraction to class predictions or event timing.Inputs can range from short recordings to entire recordings, depending on the task.Source: §3.1 · p. 10, Table 3 · p. 10

3.1.1. Feature extraction

  • Systems transform audio into features, which may be handcrafted or learned by a front-end network, then classify them and process outputs into clip scores or event timings.Front-end and back-end networks may be trained separately or end-to-end; outputs can include class predictions, posterior probabilities, or timestamped events.Source: §3.1.1 · p. 10

3.1.2. Overlap and occluded sounds

  • Overlapping sounds are harder to classify than isolated events because coincident events mix together; approaches include learning from mixtures, source separation, and mixed classes.Source separation has limited success for single-channel audio, while varied overlap sequences can undermine combined-class approaches; training with random mixtures is common.Source: §3.1.2 · pp. 11–12

3.1.3. Early sound event classification

  • Early sound-event detection asks how soon a system can reliably identify an ongoing event from partial observations, supporting low-latency monitoring.Sliding-window systems dynamically classify input segments and are useful in surveillance and safety-related applications.Source: §3.1.3 · p. 12

3.2. Sound event detection

  • Spectrograms, especially log-Mel, are widely adopted for sound-event detection, but overlap resolution and domain generalisation remain challenges.CNNs and CRNNs model local correlations, while transformer-based models better capture long-range dependencies.Source: §3.2 · pp. 12–13

3.3. Anomalous Sound Detection

  • 67.12% was the official score for diffusion-synthesized log-mel augmentation with a discriminative approach in DCASE 2024 Challenge Task 2.ASD commonly models normal sounds because anomalies are rare and diverse; generative reconstruction can miss anomalies under domain mismatch, while discriminative methods can depend on machine metadata. Spectrogram methods also require tuning across machine types and can struggle under domain shift.Source: §3.3 · pp. 13–15

3.4. Bioacoustics

  • Bioacoustic models face similar-call discrimination, overlapping vocalizations, variable call durations, noise, sparse labels, class imbalance, and taxonomic bias.Log-Mel is widely used for audible-frequency species classification, while linear spectrograms remain relevant for high-frequency segmentation; Mel scaling is inappropriate for ultrasonic signals.Source: §3.4 · pp. 15–16

4. Speech analysis

Speech analysis spans frame-level acoustic and speaker properties through utterance-level linguistic content, with time-stacked features forming spectrograms; the surveyed tasks use different objectives and feature strategies.

4. Speech analysis

  • Frame-level speech features capture vocal production and speaker cues, while utterance-level analysis captures linguistic content and semantic meaning.Frame-level dynamics can also reflect identity, mood, gender, age, and physical or mental conditions; back-end classification is often utterance-level, though other spans occur.Source: §4 · p. 17

4.1. Language and dialect identification

  • LID systems use acoustic cues to identify a language or dialect, with MFCCs often supplemented by delta features or replaced by raw log-Mel spectrograms.MFCCs derived from short frames have limited ability to capture longer-term dependencies; spectrogram time-frequency structure can support CNNs or recurrent networks learning discriminative features.Source: §4.1 · pp. 17–18
  • LID evaluations vary in language count, clip length, and speakers; more languages and shorter clips make the task harder, while recurrent networks provide temporal context.Similar languages can reduce average performance, and reported measures include EER, accuracy, and, in newer work, Cavg.Source: §4.1 · p. 18

4.2. Speaker verification

  • Speaker verification compares test-utterance embeddings with enrolled speaker models in an open-set setting, using generative or discriminative embedding approaches.Reported challenges include channel and domain mismatch, unreliable embeddings from short utterances, and manual, time-consuming spectrogram tuning that constrains robustness and generalisation.Source: §4.2 · pp. 18–19

4.3. Speech emotion recognition

  • SER seeks emotion-related, speaker-invariant features; handcrafted descriptors and utterance pooling can discard spectral detail and temporal evolution, which probably contributes to cross-corpus degradation.The degradation is described for models trained on one corpus and evaluated on another.Source: §4.3 · pp. 19–20
  • CNNs can learn local emotion-associated spectral patterns, recurrent networks capture changes across frames, and attention weights frames; window, hop, and Mel-band choices alter cue granularity.Smaller hops with recurrent networks increase sensitivity to rapid prosodic change, while higher Mel-band resolution supports finer modelling of high-frequency structures.Source: §4.3 · p. 20
  • SER still faces cross-domain mismatch, cues sensitive to short- and long-timescale settings, and difficulty interpreting multi-resolution spectral cues in SSL features.The authors suggest integrating interpretable spectrogram time-frequency structures with waveform-based robustness and abstraction.Source: §4.3 · pp. 20–21

5. Conclusion

The survey traces a shift from statistical and handcrafted features toward spectrograms and pretrained models, while emphasizing trade-offs in resolution, pooling, and adaptation. It identifies unresolved deployment and feature-setting challenges, including the empirical selection of spectrogram configurations for particular tasks and back ends.

5. Conclusion

  • Across the surveyed domains, systems shifted from statistical features through handcrafted MFCCs and filterbanks toward spectrograms, with patch size and temporal span shaped by data and model capacity.Recurrent networks allow temporal context with smaller spectrogram patches or shorter time spans and higher frequency resolution.Source: §5 · p. 21
  • Pooling spectrogram features tends to obscure fine detail, while complex frontend optimization and large-data needs have encouraged use of pretrained models adapted to target tasks.The conclusion notes adaptation performance is crucial and relates the shared auditory front end to specialized task-specific back-end processes.Source: §5 · p. 21
  • The conclusion anticipates improvements in auditory foundation models, adaptation and fine-tuning, and multi-scale analysis for balancing feature size and resolution.The paper identifies sound and event duration as a problematic factor in that balance.Source: §5 · p. 22

5.1. Future directions

  • Future deployment needs include noise robustness, lower model complexity for edge devices, sound separation, early detection, and generalisation to unseen sounds.The listed examples include overlapping sounds, reverberation, polyphonic sources, and few- or zero-shot classification from multimodal prompts.Source: §5.1 · p. 22
  • Choosing spectrogram dimensions, time-frequency resolution, frequency scaling, and element scaling for a task and back end remains largely empirical.The authors call for data-driven settings, while noting tension with generalized pretrained models unless they support multi-resolution or multi-scale processing or data fusion; future methods should also be stable and easy to use.Source: §5.1 · p. 22

Abbreviations

The section lists manuscript abbreviations for transforms, feature representations, models, tasks, and evaluation terms.

Abbreviations

  • Transform and representation abbreviations include DFT, FFT, DCT, DWT, CQT, MFCC, PLP, LMS, LS, and VNF.The list expands these terms as discrete Fourier transform, fast Fourier transform, discrete cosine transform, discrete wavelet transform, Constant-Q Transform, Mel-frequency Cepstral Coefficients, Perceptual Linear Prediction, Log-Mel Spectrogram, Linear Spectrogram, and Variance Normalised Features.Source: Abbreviations · pp. 22–23
  • Model and system abbreviations include CNN, RNN, SVM, VAE, AST, and AE.The list expands them as convolutional neural network, recurrent neural network, support vector machine, variational autoencoder, Audio Spectrogram Transformer, and autoencoder.Source: Abbreviations · pp. 22–23
  • Task abbreviations include SED, ASD, LID, DID, SER, and SV, alongside SNR for signal-to-noise ratio.These expand to sound event detection, anomalous sound detection, language identification, dialect identification, speech emotion recognition, speaker verification, and signal to noise ratio.Source: Abbreviations · pp. 22–23
Loading 2603.14917v1…