Source-linked AI summary
MuseTimbre: Zero-Shot Timbre Transfer by Controlling a Frozen Music Generator
Yuan-Chiao Cheng, Zhiyao Duan
TL;DR
Instrument-name prompts miss nuances in target timbre, while audio-reference approaches use dedicated generators with narrow or fixed instrument sets. MuseTimbre adds source-pitch and reference-timbre controls to a pretrained music generator; across four held-out datasets it matches reference timbre more closely than baselines while remaining competitive on pitch preservation.
Problem
Text prompts naming an instrument miss finer timbral nuances, while audio-reference systems use dedicated generators for narrow or fixed instrument sets [1, 2].
Source: §1 · p. 1Method
MuseTimbre conditions a pretrained music generator on source pitch estimated as a piano roll and reference timbre encoded by a fine-tuned CLAP encoder.
Source: §1 · p. 1Results
Takeaways & Limitations
Fine-tuned CLAP produces a pitch-robust timbre embedding that can serve as a timbre-similarity metric for future evaluation.
Source: §1 · p. 1
Abstract
from arXiv · showhide
Instrument timbre transfer re-voices a performance using the timbre of another instrument. Extracting the target timbre from an audio reference capture more nuances than inferring it from a text prompt. Systems that read timbre from such a clip train a dedicated model for the task, which captures the timbre cleanly but stays a narrow, single-purpose system. More versatile approaches add control to a pretrained music generator, yet a reference clip entangles timbre with genre and melody, so these systems fall back on text to name the timbre. We present MuseTimbre, the first system, to our best knowledge, that transfers timbre from an audio reference to a polyphonic source through conditioning a pretrained music generator. This system employs a multi-pitch estimator to extract pitch information from the source and finetune a CLAP encoder to extract timbre information from the reference audio. Experiments show that across four datasets of real polyphonic recordings, MuseTimbre achieves pitch alignment on par with the baselines while matching the reference timbre far more closely. Results also show that the finetuned CLAP-based timbre extractor is robust to pitch variations, making it useful in timbre similarity measures.
1. INTRODUCTION
MuseTimbre addresses the challenge of transferring nuanced audio-reference timbre to polyphonic sources while preserving their pitch content. It adds separate pitch and timbre controls to a pretrained generator and matches reference timbre more closely than baselines across four held-out datasets while remaining competitive on pitch preservation.
1. INTRODUCTION
- Audio references capture finer timbral nuances than instrument-name prompts, while prior audio-reference systems use dedicated generators for narrow or fixed instrument sets [1, 2].The task requires separating source pitch from reference timbre and fusing them.Source: §1 · p. 1
- MuseTimbre uses Basic Pitch to encode source pitch as a piano roll and a fine-tuned LAION-CLAP encoder to extract reference timbre for separate conditioning of a pretrained generator.Pitch enters through cross-attention, while timbre enters through adaptive layer normalization; the generator remains pretrained.Source: §1 · p. 1, Figure 1 · p. 2
- Across four unseen datasets, MuseTimbre matches reference timbre far more closely than every baseline while staying competitive on pitch preservation.Its multi-pitch estimator supports polyphonic sources, and its piano-roll input also accepts MIDI with the generator’s original text control applied on top.Source: §1 · p. 1
- Fine-tuning CLAP on the transfer task yields a pitch-robust timbre encoder, verified with retrieval and proposed as a timbre-similarity metric for future evaluation.MuseTimbre is presented as the first system to transfer audio-reference timbre to a polyphonic source by conditioning a pretrained music generator.Source: §1 · p. 1
2. METHOD
MuseTimbre adds separate pitch and timbre controls to a frozen pretrained music generator. It represents polyphonic pitch as a piano roll and audio-reference timbre as a global embedding, then trains and runs the conditioned system for transfer.
2.1. Scope and definitions
- MuseTimbre defines polyphonic music content through temporal pitch organization and uses a pitch estimator and CLAP encoder to extract pitch and reference timbre separately.The paper notes that pitch and timbre are naturally related, making disentangled representation challenging.Source: §2.1 · p. 2
2.2. System overview
- Two conditioning pathways control the frozen Stable Audio 3 Medium generator: piano-roll pitch enters through cross-attention, while reference timbre modulates features through adaptive layer normalization.The generator’s audio autoencoder, DiT backbone, and original text-conditioning pathway remain frozen.Source: §2.2 · p. 2
2.3. Pitch conditioning
- The pitch pathway converts Basic Pitch transcription into a binary 88-note, 88-onset piano roll, then encodes it with a 1-D CNN for cross-attention at every DiT layer.Binarization removes harmonic structure correlated with source timbre; the roll is resampled to about 10.8 frames per second.Source: §2.3 · p. 2
2.4. Timbre conditioning
- A music-trained LAION-CLAP audio encoder maps the reference waveform to one 512-dimensional embedding, which AdaLN projects into per-block scale and shift controls.AdaLN is applied after pitch cross-attention in each layer, behind a zero-initialized gate.Source: §2.4 · p. 2
2.5. Training objective and disentanglement
- The system is trained with a rectified-flow objective on a 50/50 mix of synthesized and real audio, using ground-truth or estimated pitch and a different same-clip timbre segment.The synthesized data uses Slakh MIDI stems rendered with seven soundfonts; real clips come from MoisesDB and URMP.Source: §2.5 · p. 3
2.6. Inference
- Inference starts from noise and uses 25 Euler steps; on the controlled retrieval grid, finetuned CLAP tops frozen CLAP and other baselines at every pool size.The figure compares timbre-identification retrieval against chance, which is 1/N.Source: Figure 3 · p. 3, §2.6 · p. 3
3. EXPERIMENTS
MuseTimbre matches reference timbre more closely than the baselines while preserving pitch, and its fine-tuned encoder retrieves timbre across pitch changes. Ablations and listening tests further examine the separation of its controls and its perceptual performance.
3.1. Setup
- The zero-shot evaluation covers 304 cross-instrument pairs from four held-out datasets and compares MuseTimbre with four text-conditioned and three audio-reference baselines.The datasets include thirteen instruments; each source and reference comes from a different dataset, with no shared instrument identity or recording condition.Source: §3.1 · p. 3
3.5. Pitch-timbre separation
- Removing the timbre module reduces timbre match to 16.1%, while removing the pitch module reduces frame-F1 to 0.041 and leaves timbre match high.The timbre ablation is near the 11.8% random-guess rate; the authors suggest pitch information is not captured by CLAP or not used in decoding.Source: Table 1 · p. 4, §3.5 · p. 4
3.2. Main results
- 57.6% timbre match at frame-F1 0.316 outperforms the baselines on timbre, while the baselines generally trade timbre match against pitch preservation.CTD and SS-VQ-VAE reach 48.7% and 45.4% timbre match; TokenSynth-A and TokenSynth-T reach higher frame-F1 but 34.6% and 30.4% timbre match, respectively.Source: §3.2 · p. 4
3.3. A pitch-robust timbre encoder
- 0.88 top-1 accuracy at the largest retrieval pool, up from 0.53 for frozen CLAP, shows the fine-tuned encoder can identify timbre across different note content.The controlled retrieval includes distractors playing the query’s exact notes in another timbre; evaluation uses synthetic renders because real recordings rarely provide identical content in different timbres.Source: §3.3 · p. 4
3.4. Subjective evaluation
- MuseTimbre scores highest on timbre in the listening test at 2.6, while TokenSynth-A leads on pitch at 3.4.MuseTimbre’s fidelity score is 2.5, within 0.1 of TokenSynth-A’s 2.6.Source: Figure 4 · p. 4, §3.4 · p. 4
4. CONCLUSION
MuseTimbre conditions a pretrained music generator on source pitch and reference timbre. Across four held-out datasets it achieves substantially higher timbre match than baselines while maintaining competitive pitch preservation, and fine-tuned CLAP identifies timbre across pitch changes.
4. CONCLUSION
- MuseTimbre conditions a pretrained music generator on source pitch and audio-reference timbre, enabling zero-shot instrument timbre transfer without training a dedicated transfer generator.Pitch is represented as a piano roll and reference timbre with a fine-tuned audio encoder, while the pretrained generator remains frozen.Source: §4 · p. 4
- Across four held-out datasets, MuseTimbre achieves substantially higher timbre match than baselines while maintaining competitive pitch preservation.Source: §4 · p. 4
- Fine-tuning CLAP on the transfer task produces an embedding that identifies instrument timbre across changes in pitch content, suggesting potential for timbre-similarity representation.Source: §4 · p. 4