DuoTok: Source-Aware Dual-Track Music Tokenization for Vocal-Accompaniment Generation

• The paper frames tokenization as a multi-track generation interface rather than merely a compression module, supporting synchronized streams for conditional generation, track editing, and finer control.Source: §1 · pp. 1–2
• Existing codecs preserve acoustic detail but are difficult to model, while semantic tokenizers may sacrifice fidelity, and cross-track correspondence is rarely a token-space constraint.Source: §1 · p. 1
• DuoTok first shapes source-aware structure in continuous space, then discretizes it into dual-track streams, using four training stages and diffusion-based rendering.The stages include semantic pretraining, source-aware and lyric-alignment supervision, frozen-encoder hard-routed codebooks, and diffusion reconstruction.Source: §1 · pp. 1–2
• DuoTok improves the predictability–fidelity trade-off and both unconditional and vocal-conditioned dual-track modeling, with diagnostics supporting stronger use of cross-track and temporal structure.Controlled reconstruction probes and tagging results also show that it remains reconstructable and preserves control-relevant musical attributes at low bitrate.Source: §1 · p. 2

2.1 Audio Tokenizers for Generation

• Reconstruction-oriented codecs retain fine-grained acoustic variability that makes long-context modeling harder, whereas semantic and hybrid tokens can improve modelability but risk losing music detail.Source: §2.1 · p. 2
• LM-aware and text-aligned tokenizers address intelligibility or modelability, but do not primarily target a synchronized dual-track vocal–accompaniment interface.Source: §2.1 · p. 2

2.2 Multi-Track Music Generation

• Dual-track generation systems separate vocals and accompaniment for controllability, editability, and conditional generation, but existing tokenizers rarely jointly encode source separability and cross-track correspondence.Source: §2.2 · p. 2

3 DuoTok

• DuoTok is designed as a source-aware dual-track generation interface balancing predictable tokens, track-aware semantic structure, and reconstruction fidelity.Source: §3 · p. 2

3.1 Overview

• The four-stage pipeline moves from self-supervised semantic pretraining to music-domain structure shaping, frozen-encoder dual-codebook discretization, and diffusion decoding.This ordering lets tokens focus on structured information for sequence modeling rather than carrying the full burden of waveform rendering.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 Continuous-space structure formation

• Stage 1 establishes a generic semantic foundation; Stage 2 adapts it to music and shapes source-aware, lyric-aligned features using replacement noise and auxiliary supervision.The auxiliary objectives include spectral reconstruction, ASR-based lyric alignment, and source-aware distillation from a frozen music source-separation teacher.Source: §3.2 · p. 3

3.3 Stable dual-track discretization and rendering decoupling

• Stage 3 freezes the encoder learned from Stage 1–Stage 2 and routes vocal or accompaniment features to track-specific quantizers; Stage 4 freezes the tokenizer and trains track-wise latent diffusion decoders to reconstruct the final waveform.This decouples high-fidelity rendering from token learning, and the learned dual-track token interface is not conceptually tied to this specific decoder choice.Source: §3.3 · p. 3

4.1 Data, pseudo-stems, and supervision

• Public corpora span speech, general audio, and music for Stage 1, while later stages use music and vocal-centric data, Demucs pseudo-stems, lyric-aligned clips, separation regularization, and ASR supervision.BS-RoFormer supplies a weak continuous-space structural prior, whereas Demucs stems provide separated inputs for discretization.Source: §4.1 · p. 4

4.2 Baselines

• Baselines cover reconstruction-oriented codecs, semantic-oriented tokenizers, and music-oriented systems, using official checkpoints and standard configurations when available.Examples include DAC, EnCodec, WavTokenizer, SemantiCodec, X-Codec, and MuCodec.Source: §4.2 · p. 4

4.3 Held-constant evaluation protocols

• All tokenizers use the same Qwen2-1B backbone, optimizer, schedule, training steps, and 30 s context for unconditional and track-conditioned dual-track modeling.The evaluation also probes post-quantization control attributes and reports native and matched-decoder reconstruction, including a blinded MOS study.Source: §4.3 · pp. 4–5

4.4 Metrics

• Lower codebook-normalized BPT (cnBPT) is the primary predictability metric, while entropy-normalized BPT (enBPT) is complementary when empirical code entropy is consistently estimable.cnBPT applies a first-order codebook-size correction; enBPT uses empirical unigram entropy.Source: §4.4 · p. 5
• Codebook-normalized bits per second (cnBPS) complements per-token cnBPT by accounting for total token emission rate when comparing modeling burden across tokenizers.The metrics section also reports AP and AUC for semantic accessibility and Mel distance and, in matched-decoder evaluation, MOS for reconstruction.Source: §4.4 · p. 5

5.1 Overall Predictability–Fidelity Trade-off

• DuoTok improves over closely matched MuCodec on cnBPS, 24.0 vs. 25.0, while maintaining a clear reconstruction advantage.The per-second comparison supports the same conclusion as the per-token view.Source: §5.1 · p. 5
• At 0.75 kbps, DuoTok achieves the lowest cnBPT and improves over matched-bitrate MuCodec from 0.50 to 0.48 cnBPT and from 1.37 to 0.95 Mel distance.It also achieves the strongest tagging AP at this bitrate.Source: §5.1 · p. 5, Table 1 · p. 6

5.2 Dual-Track Sequence-Modeling Ability

• DuoTok achieves the lowest average cnBPT (0.483), improving over X-Codec in YuE (0.631) and MuCodec in LeVo (0.501) in unconditional dual-track modeling.It maintains near-full utilization on both codebooks (0.9996/0.9995) and high normalized entropy (0.9188/0.9494).Source: §5.2 · pp. 5–6, Table 2 · p. 6
• Unconditional and conditioned results support a more modelable, structurally useful dual-track token space, with gains not confined to one track or evaluation direction.Controlled diagnostics show larger predictability costs under cross-track corruption and larger gains from longer context.Source: §5.2 · p. 6
• In vocal-conditioned accompaniment prediction, DuoTok reaches 0.464 cnBPT and 0.489 enBPT, the lowest predictability costs under both normalization schemes.This task probes whether vocal tokens expose useful structure for predicting accompaniment.Source: Table 3 · p. 6, §5.2 · p. 6

5.3 Reconstruction and Semantic Accessibility

• On MTT tagging, a probe on frozen post-quantization embeddings gives DuoTok AP 0.35 and AUC 0.87, with the strongest AP among compared methods at an ultra-low bitrate.The result supports semantic accessibility for downstream control-related attribute prediction, not broad music understanding.Source: §5.3 · p. 6
• Under matched-decoder evaluation, DuoTok improves vocal reconstruction across PESQ, STOI, and Mel distance, lowers accompaniment Mel distance, and receives a clear blinded-MOS preference on both tracks.The comparison is intended to reduce decoder confounds and probe token-space reconstructability under fixed decoder capacity, though it does not fully eliminate decoder–token compatibility effects.Source: §5.3 · p. 6, Table 4 · p. 7

6.1 Structural Diagnostics and Robustness

• All three conditioning-track perturbations raise predictability cost, with uncorrelated mismatch most damaging and DuoTok more sensitive to correspondence-breaking corruption than weaker baselines.Temporal shift and masking also produce clear monotonic penalties.Source: §6.1 · p. 7, Figure 3 · p. 7
• DuoTok shows a steeper cnBPT reduction with longer context than MuCodec (LeVo), suggesting its gains extend beyond easier short-range prediction.Both tokenizers benefit from longer context.Source: §6.1 · p. 7, Figure 4 · p. 7
• Separator swap causes only a small cnBPT increase while preserving the context-length trend; controlled leakage raises cnBPT for both tokenizers, with DuoTok less sensitive at low contamination.Table 5 consolidates the separator-swap and contamination checks.Source: §6.1 · p. 7, Table 5 · p. 8
• DuoTok’s gains are best interpreted as structural rather than simple token-level simplification, and robustness checks make a single separator or trivial leakage less likely to explain them.The checks do not remove pseudo-stem concerns.Source: §6.1 · pp. 7–8

6.2 Design Ablations

• Removing feature replacement noise raises Avg. cnBPT from 0.4832 to 0.5374, while Mel changes little, supporting its role as a structural regularizer rather than a reconstruction booster.Both vocal and accompaniment cnBPT worsen when the noise is removed.Source: §6.2 · p. 8, Table 6 · p. 8
• Removing encoder freezing yields Avg. cnBPT 0.6004, the largest degradation, indicating that a fixed encoder is critical for stable, LM-friendly discretization after hard routing.Mel behavior is mixed: vocal Mel worsens to 0.250, while accompaniment Mel improves to 0.210.Source: §6.2 · p. 8, Table 6 · p. 8
• Removing MSS slightly improves Avg. cnBPT from 0.4832 to 0.4800 but worsens Mel on both tracks, especially vocals (0.240 to 0.249), illustrating a fidelity–predictability trade-off.Source: §6.2 · p. 8

7 Conclusion

• DuoTok treats music tokenization as a multi-track generation interface and combines semantic pretraining, source-aware structure shaping, hard dual-codebook routing, and diffusion decoding.The design jointly targets reconstruction, control-relevant attributes, predictable tokens, and cross-track correspondence.Source: §7 · p. 8
• Across benchmarks, DuoTok achieves the lowest cnBPT and strongest tagging AP among compared methods while maintaining competitive reconstruction at ultra-low bitrate.Source: §7 · p. 8
• Held-constant dual-track LM evaluation and structural diagnostics are consistent with stronger use of cross-track and temporal structure rather than only easier local prediction.The conclusion positions multi-track tokenizer design as an interface problem, not only a compression problem.Source: §7 · p. 8