PaperlayerView exact revision on arXiv

Source-linked AI summary

A Universal Music Translation Network

Noam Mor, Lior Wolf, Adam Polyak, Yaniv Taigman

arXiv:1805.07848v2cs.SDcs.AIcs.LGstat.ML

TL;DR

The paper addresses the gap between human and AI musical mimicry by presenting an unsupervised method for translating music across instruments, styles, and genres. Its shared-encoder WaveNet autoencoder is reported to translate from unseen inputs, with performance on instrument conversion approaching professional musicians. The authors also identify evaluation and temporal-context limitations.

  • Problem

    AI remains much inferior to biological systems at musical mimicry, despite music being a digitized domain.

    Source: §1 · p. 1

  • Method

    A multi-domain WaveNet autoencoder uses a shared encoder and target-domain decoder to transform inputs, including undistorted samples from unseen musical domains.

    Source: §1 · p. 1, §3.4 · p. 5

  • Results

    The system is reported on par with or slightly worse than professional musicians for instrument conversion and can process unseen instruments or whistles.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors say the work could open paths to music transcription and automatic composition, and report initial creative outputs when reducing latent-space size.

    Source: §5 · p. 8

    Human judgments may mix audio quality with translation assessment, machine scores capture only one aspect, and the authors are still investigating long-term structure beyond roughly one-second context.

    Source: §4 · p. 6, §5 · p. 8

Abstract

from arXiv · show

We present a method for translating music across musical instruments, genres, and styles. This method is based on a multi-domain wavenet autoencoder, with a shared encoder and a disentangled latent space that is trained end-to-end on waveforms. Employing a diverse training dataset and large net capacity, the domain-independent encoder allows us to translate even from musical domains that were not seen during training. The method is unsupervised and does not rely on supervision in the form of matched samples between domains or musical transcriptions. We evaluate our method on NSynth, as well as on a dataset collected from professional musicians, and achieve convincing translations, even when translating from whistling, potentially enabling the creation of instrumental music by untrained humans.

1 Introduction

The paper targets musical mimicry, where AI remains inferior to biological systems, and presents high-fidelity translation across instruments, styles, and genres. Its shared encoder, domain confusion, and pitch-distortion training support translation from unseen musical inputs; the authors report results approaching professional musicians.

1 Introduction

  • The method aims for high-fidelity translation across instruments, styles, and genres, including transformations such as orchestra-to-piano.The authors describe their claimed result as, to their knowledge, a first.Source: §1 · p. 1
  • The approach combines autoregressive audio synthesis with unsupervised domain translation, avoiding the parallel instrument dataset required by supervised learning.Teacher forcing also makes autoregressive decoders efficient to train.Source: §1 · p. 1
  • A single universal encoder processes all inputs and can translate from musical domains not heard during training to domains encountered in training.The shared encoder also reduces the number of networks to train.Source: §1 · p. 1
  • Domain confusion discourages domain-specific encoding, while random local pitch modulation helps the encoder avoid memorizing input signals.Training reconstructs the undistorted input from distorted audio, projecting out-of-domain inputs toward the desired output domain.Source: §1 · pp. 1–2
  • The authors report translations on par with or slightly below professional musicians, with inputs including unseen instruments and whistles.They also report relatively high-quality output and say new instruments can be added without retraining the entire network.Source: §1 · p. 2

2 Previous Work

Prior work spans unsupervised domain translation, image style transfer, audio conversion, and autoregressive synthesis. The paper distinguishes its approach through a shared encoder, domain-specific decoders, and waveform generation with a WaveNet decoder.

Domain Transfer

  • Unsupervised domain translation commonly uses GAN constraints, but autoregressive teacher-forced outputs differ from test-time generation, making realistic fake samples impractical during training.The paper also argues that circularity constraints are unrealistic when training outputs differ from test-time outputs.Source: §2 · p. 2
  • StarGAN uses one generator conditioned on the target domain, whereas this work uses separate decoders after a shared encoder.Conditioning a single decoder on the output domain did not produce convincing results in the authors’ experiments.Source: §2 · p. 2
  • CoGAN and UNIT use shared latent representations with domain-specific network components; this method instead uses one encoder and domain confusion rather than a VAE loss.CoGAN requires fitting a latent vector at inference, while UNIT adds cycle consistency.Source: §2 · p. 2

Audio Synthesis

  • WaveNet predicts next-sample probabilities from previous samples and conditioning input; this work uses it as the decoder in an encoder-decoder architecture.The paper contrasts its constraints with prior WaveNet-based voice conversion and a prior WaveNet autoencoder, noting consumer-media inputs and added decoders, disentanglement, and augmentation.Source: §2 · pp. 2–3

Style Transfer

  • Image style-transfer methods preserve content while changing style, but the authors argue musical translation requires higher-level mapping rather than local texture changes.They note that classical audio conversion methods are limited to monophonic instruments and use pitch-and-timbre analysis followed by synthesis.Source: §2 · p. 3

3 Method

The method trains multiple domain-specific autoencoder pathways around a shared encoder, using reconstruction, augmentation, and domain confusion to encourage domain-independent representations. At inference, the target-domain decoder transforms inputs, including those from unseen musical domains.

3 Method

  • Multiple autoencoder pathways share an encoder and use separate reconstruction losses, random input augmentation, and domain confusion to discourage memorization and domain-specific encodings.Figure 1 depicts the architecture; its dashed confusion block is used only during training.Source: §3 · p. 3, Figure 1 · p. 4

3.1 WaveNet Autoencoder

  • The WaveNet-like encoder uses three blocks of ten residual layers, then average-pools over 50 milliseconds to produce an R64 encoding.Its fully convolutional design accepts any sequence length; the WaveNet decoder is conditioned on the encoding, with 8-bit mu-law quantization and a 250-millisecond receptive field.Source: §3.1 · pp. 3–4

3.2 Audio Input Augmentation

  • Training locally modulates pitch in a randomly selected segment to encourage encoder generalization and higher-level information.The selected segment is 0.25–0.5 seconds within a one-second training segment, with a random shift of -0.5 to 0.5 half-steps.Source: §3.2 · p. 4

3.3 Training and the Losses Used

  • For each domain, a WaveNet decoder reconstructs its input from the shared encoder representation using elementwise cross-entropy; a classifier predicts the input domain from latent vectors.The decoder receives the target sample from the previous time step during training, while the domain classifier is trained to minimize classification loss.Source: §3.3 · pp. 4–5

3.4 Network during inference

  • At inference, the undistorted input passes through the shared encoder and the selected target-domain decoder, including when the input domain was unseen during training.The autoregressive WaveNet process is the inference bottleneck.Source: §3.4 · p. 5

4 Experiments

Experiments assess music conversions through human ratings, automatic pitch comparisons, source-identification lineups, semantic blending, and NSynth embedding analysis. The reported results include strong same-pitch embedding similarity across instruments, alongside limitations in evaluation and ablation training.

Training and evaluation

  • The experiments used six classical training domains and held-out segments; evaluation also included unseen sources, while both attempted ablations failed to train adequately.The comparison converted 60 segments—including harpsichord, orchestra, and unseen-domain material—to piano; NSynth was not seen during training.Source: §4 · pp. 5–7

Evaluation of translation quality

  • Unseen-domain conversions received higher translation-success ratings than harpsichord or orchestra conversions, though musicians outperformed the system in all three cases.Audio quality was considerably lower than that of humans using a computer-connected keyboard.Source: Table 1 · p. 6, §4 · p. 6
  • After DTW, the pitch-comparison gap between the method and human translators diminished, possibly because the method preserves input timing differently.The method was more conservative than human translators when comparing output pitch with input pitch.Source: Table 2 · p. 6, §4 · p. 6

Lineup experiment

  • In the lineup experiment, participants often failed to identify the source; the general-population listeners tended to choose the same domain regardless of the actual source.Amateur S also showed coherent confusion patterns between pairs of instruments.Source: Figure 2 · p. 7, §4 · p. 7

Semantic blending

  • Blending two Mozart-symphony embeddings over a 1.5-second overlap produced a natural, seamless shift in the generated audio.The experiment linearly weighted the two embeddings during the overlap.Source: §4 · p. 7

NSynth pitch experiments

  • Across instrument pairs, same-pitch embedding cosine similarity averaged 0.90–0.95, and the embeddings clearly encoded pitch despite training on complex polyphonic audio.Figure 3 contrasts self-correlations with cross-instrument correlations.Source: §4 · p. 7, Figure 3 · p. 8

5 Discussion

The discussion frames the encoder as a universal representation that handles varied out-of-domain inputs despite training on six homogeneous domains. It also identifies possible transcription and composition applications while noting that long-term structure remains under investigation.

5 Discussion

  • Despite training on only six homogeneous domains, the encoder represents a wide variety of out-of-domain inputs.The authors connect this to the role of universal representations in other areas of machine learning.Source: §5 · p. 8
  • The encoder may suit music transcription because it captures information in an instrument-dependent way, while reducing latent-space size makes decoder outputs more creative and novel.The authors also report initial results they find interesting for automatic composition.Source: §5 · p. 8
  • Although a WaveNet autoencoder observes only about one second of context, the authors are investigating whether autoregression can capture additional long-term structure.They suggest this may arise from mapping consistency or the model maintaining some context.Source: §5 · p. 8

Related papers

Loading 1805.07848v2…