Source-linked AI summary
Audio Spectrogram Representations for Processing with Convolutional Neural Networks
L. Wyse
TL;DR
The paper examines how audio should be represented for neural-network generation, focusing on spectrograms for style transfer. It reviews spectral and alternative representations, adapts image-style methods, and finds that audio-trained networks can integrate content and style better, while image-CNN assumptions create important limitations.
Problem
Audio representation must balance information preservation, dimensionality, and reconstructible sound, but representations effective for visual images may not fit sonic structure.
Method
The paper reviews audio representations and tests spectrogram-based style transfer using image-trained and audio-classification networks, including random-versus-trained weights.
Results
Audio-trained weights appear to generate a more integrated synthesis of content and style, while image-style sonogram transfer produces less compelling sounds than visual transfer.
Takeaways & Limitations
Spectral representations are useful for generative audio because they retain substantial information, reduce dimensionality relative to raw audio, and support high-quality reconstruction.
Takeaways & Limitations
Linear-frequency sonograms cannot be treated exactly like images by 2D convolutional networks because sonic objects are not frequency-translation invariant and may overlap across frequencies.
Abstract
from arXiv · showhide
One of the decisions that arise when designing a neural network for any application is how the data should be represented in order to be presented to, and possibly generated by, a neural network. For audio, the choice is less obvious than it seems to be for visual images, and a variety of representations have been used for different applications including the raw digitized sample stream, hand-crafted features, machine discovered features, MFCCs and variants that include deltas, and a variety of spectral representations. This paper reviews some of these representations and issues that arise, focusing particularly on spectrograms for generating audio using neural networks for style transfer.
1 Introduction
Audio can be represented through hand-crafted features, learned features, MFCCs, raw samples, or spectral representations, with the best choice depending on the application and processing machinery. The paper frames representation choice as especially important because audio analysis and synthesis impose different requirements.
- Audio representation choices include hand-crafted statistics, learned features, MFCCs, raw samples, and spectral representations.Examples of hand-crafted features include spectral centroid, zero-crossing statistics, harmonicity, fundamental frequency, and temporal envelopes.
- Modern practice generally lets the network discover the features needed for its task rather than relying entirely on manually designed features.
- MFCCs describe spectral shape and remain effective for speech classification and identification despite being lossy and highly compressed.Their use in environmental sound classification has also been reported, with 65% classification accuracy in one cited result.
- The paper reviews these representation choices in the context of neural-network processing of audio.
2 Sound Representation for Generative Networks
For audio generation and style transfer, spectrograms offer a compact representation that can be reconstructed into sound, but image-style convolutional processing does not transfer cleanly to sonic structure. Experiments show that audio-trained networks and frequency-as-channel representations can produce more integrated results, while noise and architectural choices strongly affect style.
- Representation choices: Generative applications require representations that can synthesize high-quality sound, excluding lossy features such as MFCCs and many hand-crafted feature sets.
- Representation choices: Spectrograms encode time, frequency, and component strength as a 2D image, making image-oriented convolutional architectures applicable in principle.
- Image-based style transfer: Image-based VGG-19 style transfer combines deeper-layer content activations with shallower-layer Gram-matrix style features after adapting single-channel spectrograms to RGB input.The synthesized color output must be converted back to a single channel to represent a meaningful spectrogram.
- Image-based style transfer: Image-style processing of sonograms creates interesting nonlinear combinations, but the resulting sounds are less compelling than visual style-transfer results.The paper attributes this likely to differences between sonic and visual object representations and to how convolutional networks process them.
- Architectural limitations: Linear-frequency sonograms violate image-CNN assumptions because frequency shifts alter sonic-object extent, while sounds can overlap at frequencies and remain nonlocally distributed.
- Audio-trained style transfer: An audio-classification network trained on ESC-50 was compared with random weights for style transfer using 856-frame, 257-bin spectrograms.The network used two convolutional layers, two fully connected layers, and simultaneous spectral-centroid classification as regularization.
- Audio-trained style transfer: Style transfer works without regard to weights under first- and second-order matching, but audio-trained weights appear to produce a more integrated synthesis of content and style.
- Audio-trained style transfer: Noise and random weights impair style more than content, while frequency-as-channel designs require very wide layers for longer-timescale style features to appear.The study used 2048 first-layer channels, compared with Ulyanov’s 4096 channels.
3 Summary
Spectral representations retain more information than traditional hand-crafted features while using lower-dimensional data than raw audio. Their generative usefulness comes from techniques for reconstructing high-quality audio, although linear-frequency sonograms require alternatives to ordinary image-style 2D convolutions.
- Spectral representations retain more information than most traditional hand-crafted audio features while requiring lower-dimensional data than raw audio.
- Their available reconstruction techniques make spectral representations particularly useful for generative neural-network applications.
- Linear-frequency sonograms cannot be treated identically to images by ordinary 2D convolutional networks.
- Treating frequency bins as channels is an alternative approach that has shown promising results.