Source-linked AI summary
Time Domain Neural Audio Style Transfer
Parag K. Mital
TL;DR
Audio style transfer previously relied on magnitude spectrograms and Griffin-Lim phase reconstruction, limiting access to fine temporal information. This paper explores direct time-domain optimization across several neural representations and finds one network that meaningfully retains both content and style, with differences from the original approach.
Problem
Magnitude-based audio style transfer requires phase reconstruction, which can lose fine temporal information, introduce error, and complicate real-time applications.
Source: §1 · p. 1Method
Results
One concatenated real-imaginary-magnitude network meaningfully retained both content and style, while most alternatives produced white noise or unusable stylizations.
Source: §3 · p. 2Takeaways & Limitations
Abstract
from arXiv · showhide
A recently published method for audio style transfer has shown how to extend the process of image style transfer to audio. This method synthesizes audio "content" and "style" independently using the magnitudes of a short time Fourier transform, shallow convolutional networks with randomly initialized filters, and iterative phase reconstruction with Griffin-Lim. In this work, we explore whether it is possible to directly optimize a time domain audio signal, removing the process of phase reconstruction and opening up possibilities for real-time applications and higher quality syntheses. We explore a variety of style transfer processes on neural networks that operate directly on time domain audio signals and demonstrate one such network capable of audio stylization.
1 Introduction
Prior audio style transfer independently manipulates content and style from spectrogram magnitudes but reconstructs phase afterward. This work instead investigates direct optimization of time-domain audio using several network representations.
Background
Motivation
- Magnitude-only stylization can lose fine temporal information such as vibrato and spatial position, while phase reconstruction introduces error and complicates real-time applications.Networks that discard phase still require phase reconstruction for synthesis.Source: §1 · p. 1
- The paper asks whether raw-audio neural networks can enable independently manipulable content and style without phase reconstruction.Prior raw-audio synthesis could generate audio but did not separate content and style.Source: §1 · p. 1
Approach
- The experiments investigate time-domain networks using DFT components, pretrained WaveNet and NSynth activations, and varied content/style parameters.Representations include real and imaginary components, magnitude, phase, and unwrapped phase differentials.Source: §1 · pp. 1–2
2 Experiments
The experiments compare multiple time-domain computational graphs and pretrained feature networks for audio stylization. They retain the original stylization objective while replacing noisy magnitude optimization with direct signal optimization.
DFT-based networks
- DFT-based graphs manipulate real, imaginary, magnitude, phase, and unwrapped phase-differential components before stylizing the optimized time-domain signal.The study also varies whether content and style layers occur before or after fully connected layers.Source: §2 · p. 2
Pretrained networks
- Pretrained WaveNet decoder and NSynth encoder activations are evaluated at multiple layers, alongside Ulyanov’s original network as a baseline.Results are assessed through spectrograms and listening.Source: §2 · p. 2
3 Results
Only one tested network meaningfully retained both content and style: a concatenated real-imaginary-magnitude representation with a height-3 convolution. Other configurations generally failed or were substantially noisier.
Successful network
- Only the concatenated real, imaginary, and magnitude network with a kernel-size-3 height convolution retained both content and style in meaningful audio reconstruction.Its content layer uses concatenated features before linear layers, while its style layer uses magnitudes and later nonlinear-layer features.Source: §3 · p. 2
Comparison
- The successful network produced quicker, busier temporal changes and apparently higher-fidelity low-frequency synthesis than Ulyanov’s network at the same sample rate.It also generated substantially less noise, plausibly because it avoided phase-reconstruction errors.Source: §3 · p. 2
Alternative networks
- The magnitude and unwrapped-phase-differential network with a kernel-size-2 filter produced recognizable but considerably noisy audio, while other tested combinations tended toward white noise.Those alternatives generally did not appear to reduce the loss.Source: §3 · p. 2
Pretrained networks
- WaveNet decoder and NSynth encoder activations did not yield meaningful-sounding stylized synthesis in the explored configurations.The result may depend on the limited combinations of layers and activations tested.Source: §3 · p. 2
4 Discussion and Conclusion
The study demonstrates time-domain neural audio style transfer, with one representation matching the original approach while producing different synthesis characteristics. Pretrained NSynth components were unsuccessful in the configurations explored.
Conclusion
- Two networks produced meaningful results: the magnitude/unwrapped-phase network was noisier, while the real-imaginary-magnitude network retained both content and style at comparable quality to Ulyanov’s approach.The successful network also exhibited interesting differences from the original stylization.Source: §4 · p. 2
Future work
- Failure to stylize with NSynth’s encoder or decoder may reflect the limited layer and activation combinations explored, motivating further investigation.This is presented as a future research direction rather than a definitive limitation of NSynth features.Source: §4 · p. 2
Example synthesis
- Figure 1 presents an example of direct audio optimization in which both source content and style remain audible.The figure is an example synthesis rather than a quantitative comparison.Source: Figure 1 · p. 3