PaperlayerView exact revision on arXiv

Source-linked AI summary

LiveSynth: A Streaming Neural Synthesizer for Instrument Cloning and Text-to-Instrument

Kyungsu Kim, Yejin Kim, Kyogu Lee

arXiv:2610.13054v1cs.SDeess.AS

TL;DR

Existing neural synthesizers support zero-shot instrument cloning and text-based timbre control, but generate audio offline rather than responding in real time. LiveSynth streams MIDI- and timbre-conditioned audio latents through a causal decoder, achieving quality comparable to offline baselines and real-time inference on an M3 MacBook Air.

  • Problem

    Existing neural synthesizers generate complete audio segments offline, while DDSP adaptation to unseen timbres generally requires instrument-specific training or optimization, leaving streamable zero-shot timbre control unresolved.

    Source: §1 · p. 1

  • Method

    LiveSynth uses a feedback-free causal Transformer conditioned on streaming MIDI and frame-wise CLAP timbre embeddings to generate continuous audio latents for causal VAE decoding.

    Source: §1 · p. 1, §3.3 · p. 3

  • Results

    Across NSynth and Slakh, LiveSynth achieved timbre similarity and spectral reconstruction quality comparable to offline baselines, with RTF 0.88 and 10 ms algorithmic latency on an M3 MacBook Air.

    Source: §1 · p. 1, §4.2 · p. 4

  • Takeaways & Limitations

    LiveSynth combines zero-shot cloning and text-to-instrument synthesis with frame-wise timbre morphing and autonomous continuation, and meets the reported computational requirements for real-time inference.

    Source: §1 · p. 1

    The authors report lower note accuracy and audio quality, weak rhythmic dependence in continuation, and evaluation without listening tests or performer studies.

    Source: §5 · p. 4

Abstract

from arXiv · show

Recently, generative neural synthesizers have enabled zero-shot instrument cloning from reference recordings and timbre generation from text prompts. However, existing systems generate complete audio segments offline, precluding their use as playable instruments. We present LiveSynth, a streaming neural synthesizer with flexible timbre control and interactive generation capabilities, including zero-shot instrument cloning and text-to-instrument synthesis. A feedback-free causal Transformer conditioned on MIDI events and a CLAP timbre embedding outputs continuous audio latents at 100 Hz, which a causal variational autoencoder decodes into audio. Frame-wise timbre conditioning enables continuous timbre morphing, while MIDI-conditioning dropout enables autonomous continuation from a performed prefix. On the NSynth and Slakh datasets, LiveSynth achieved timbre similarity and spectral reconstruction quality comparable to offline baselines. On an M3 MacBook Air, LiveSynth required 8.8 ms of computation per 10 ms audio frame (an RTF of 0.88) with 10 ms algorithmic latency, enabling real-time interaction with a performer.

1. INTRODUCTION

LiveSynth addresses the gap between flexible zero-shot timbre control and playable, real-time neural synthesis. It streams MIDI- and timbre-conditioned audio and achieves quality comparable to offline baselines while meeting a real-time inference budget.

1. INTRODUCTION

  • Existing neural synthesizers support zero-shot timbre control but generate complete audio segments offline, while DDSP adaptation to unseen timbres generally requires instrument-specific training or optimization.Together, these approaches leave streamable zero-shot timbre control unresolved.Source: §1 · p. 1
  • LiveSynth conditions streaming MIDI and frame-wise CLAP timbre embeddings to generate continuous audio latents, which a causal audio decoder converts into waveform samples.Its feedback-free generator avoids feeding previously generated latents back into the model, avoiding feedback-induced error accumulation while enabling strictly causal inference.Source: §1 · p. 1
  • Frame-wise timbre updates enable continuous morphing, while MIDI-conditioning dropout lets LiveSynth continue a preceding performance from a null MIDI condition for call-and-response interaction.These controls are part of the same streaming synthesizer.Source: §1 · p. 1
  • Across NSynth and Slakh, LiveSynth achieved timbre similarity and spectral reconstruction quality comparable to offline baselines, with RTF 0.88 and 10 ms algorithmic latency on an M3 MacBook Air.The reported latency satisfies the computational requirements for real-time inference.Source: §1 · p. 1

2. RELATED WORK

Related work spans controllable neural synthesizers, efficient DDSP systems, and newer models using audio or text for zero-shot timbre control. Separate real-time systems explore streaming, timbre morphing, and musician accompaniment.

2. RELATED WORK

  • Early generative synthesizers such as NSynth and Mel2Mel learned timbre representations for controllable synthesis and interpolation.DDSP-based synthesizers add differentiable signal-processing priors for efficient real-time operation, but unseen timbres generally require target-specific training or optimization.Source: §2 · p. 1
  • TokenSynth, CTD, and AnySynth condition on reference audio or text prompts for zero-shot instrument cloning and open-ended timbre control [4].Unified Timbre Transfer explores within-stream morphing through time-varying timbre embeddings.Source: §2 · p. 1
  • Live Music Models and LiveBand generate continuous audio streams in real time, with LiveBand focused on live-audio-conditioned accompaniment [14] [10]; ReaLJam explores accompanying a musician's input.These works address real-time and interactive music generation across multiple forms of musical interaction.Source: §2 · pp. 1–2

3. METHOD

LiveSynth generates continuous audio latents from streaming MIDI and timbre conditions, then decodes them causally into audio. Its method combines a causal audio autoencoder, frame-wise conditioning, and feedback-free sequence generation.

3.1. Causal Audio Autoencoder

  • A causal autoencoder uses a 100 Hz continuous latent space instead of quantized codec tokens, whose low-latency constraints reduced reconstruction and perceptual quality in preliminary experiments.The separately trained, frozen VAE maps 48 kHz mono audio to 128-dimensional latents and decodes each latent frame into 480 waveform samples.Source: §3.1 · p. 2
  • The VAE combines reconstruction, adversarial, feature-matching, and KL regularization losses, including multi-scale log-mel and STFT magnitude losses.The loss descriptions identify log-mel ℓ1 losses across multiple scales and linear- and log-magnitude ℓ1 losses across multiple STFT scales.Source: §3.1 · p. 2

3.2. MIDI and Timbre Conditioning

  • Frame-wise MIDI states encode each pitch as inactive, offset, or onset/sustain at five velocity levels, with note-age information retained through a RoPE-style rotation.Whole-clip and contiguous-span MIDI dropout support generation without MIDI conditioning and continuation after a performed prefix.Source: §3.2 · p. 2
  • A frozen CLAP encoder supplies audio- or text-derived timbre embeddings; frame-wise inference conditions enable timbre morphing through spherical interpolation.Because audio and text embedding distributions have a modality gap, an orthogonal Procrustes map fitted on NSynth training examples aligns text conditions to audio embeddings for inference.Source: §3.2 · p. 3

3.3. Feedback-Free Sequence Generation

  • The feedback-free generator conditions on frame-wise noise, MIDI, and timbre without using its previous latent predictions; causal hidden states represent temporal dependencies.Sliding-window causal attention uses the most recent 500 frames (5 seconds), and generated-latent errors therefore do not accumulate through later inputs.Source: §3.3 · p. 3
  • A conditional discriminator supports adversarial generator training, while latent reconstruction loss is applied only to fully conditioned sequences.Restricting pointwise reconstruction to fully conditioned sequences avoids penalizing valid alternatives or encouraging averaged predictions; the final objective combines it with the R3GAN generator objective.Source: §3.3 · p. 3

4. EXPERIMENTS

LiveSynth performs competitively on zero-shot instrument cloning and text-to-instrument tasks, while also supporting timbre morphing and MIDI-free continuation. Its streamed generation meets a real-time budget on an M3 MacBook Air.

4.1. Experimental Setup

  • The causal VAE was trained for 400k steps on 1-second, 48-kHz clips; LiveSynth was trained for 200k updates on 10-second NSynth/Lakh renderings and Slakh stems.Evaluation included 530 NSynth and 480 Slakh cloning clips, plus text-to-instrument evaluation on 16 NSynth classes.Source: §4.1 · p. 3

4.2. Results

  • On NSynth and Slakh, LiveSynth has the highest MERT and PANNs similarities and lowest MSS among same-protocol systems, but lower note F1 and PQ.CE falls between the two offline baselines; the causal audio representation may contribute to lower MIDI adherence.Source: §4.2 · p. 3, Table 1 · p. 4
  • With aligned CLAP embeddings, LiveSynth matches unaligned TokenSynth in MusicCoCa similarity and achieves the highest note F1 and CE among generated outputs.Aligned TokenSynth remains stronger in text–audio correspondence and PQ.Source: Table 2 · p. 4, §4.2 · p. 4
  • Across 64 instrument pairs, PANNs similarity to A decreases while similarity to B increases; their curves cross after the midpoint of the 10-second morph.The crossover does not coincide with the midpoint of embedding interpolation; the authors hypothesize that earlier timbre conditions in the Transformer’s KV cache contribute to the delay.Source: Figure 2 · p. 4, §4.2 · p. 4
  • Matched continuations have PCH-JSD comparable to true continuations and substantially below shuffled controls, but their IOI-JSD remains near shuffled controls and above true continuations.Continuation CE and PQ are also below those of true continuations.Source: Table 3 · p. 4, §4.2 · p. 4
  • On an M3 MacBook Air, average processing takes 8.78 ms per 10 ms frame (RTF 0.88), with 10 ms algorithmic latency and 9.77 ms 99th-percentile processing time.The 99th-percentile time remains within the per-frame real-time budget.Source: §4.2 · p. 4

5. CONCLUSION AND LIMITATIONS

LiveSynth combines streaming timbre control with competitive cloning and real-time performance. The conclusion also identifies limitations in note accuracy, audio quality, rhythmic dependence, and human evaluation.

5. CONCLUSION AND LIMITATIONS

  • LiveSynth achieves competitive timbre similarity and lower spectral distance than offline baselines at RTF 0.88 and 10 ms algorithmic latency.It also supports frame-wise timbre morphing and autonomous performance continuation.Source: §5 · p. 4
  • The stated limitations are lower note accuracy and audio quality, weak rhythmic dependence in continuation, and evaluation without listening tests or performer studies.The authors identify improved causal audio representations, rhythmic modeling, and human evaluation as future directions.Source: §5 · p. 4
  • The conclusion presents zero-shot instrument cloning and text-to-instrument synthesis as capabilities of one streaming synthesizer.Source: §5 · p. 4
Loading 2610.13054v1…