PaperlayerView exact revision on arXiv

Source-linked AI summary

TokenMapper: A Step Toward Interoperable Speech Token Translation

Tal Kozakov, Tal Rosenwein, Eliya Nachmani

arXiv:2609.12563v1cs.LGcs.AIcs.SDeess.AS

TL;DR

Speech token spaces differ across models, so transferring information typically requires waveform decoding and re-encoding. TokenMapper translates directly between heterogeneous tokenizers in the discrete domain; cross-model WER remains close to native reconstruction on LibriSpeech and VCTK, with reduced transfer latency.

  • Problem

    Different vocabularies and codebook structures make speech-token spaces incompatible, so cross-model transfer typically requires waveform decoding and re-encoding.

    Source: §1 · p. 1

  • Method

    TokenMapper uses direction-aware mappings to translate directly between heterogeneous discrete token spaces, including single- and multi-codebook tokenizers.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    Cross-model WER increases by 2.56-5.96% absolute on LibriSpeech and 2.96-6.83% on VCTK versus native reconstruction.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Direct token-domain interoperability is feasible in the studied same-rate setting and is a step toward modular speech systems communicating through discrete tokens.

    Source: §6 · p. 8

    The formulation assumes paired same-utterance token sequences at a shared effective rate and does not directly handle different frame rates.

    Source: §7 · p. 8

Abstract

from arXiv · show

Neural audio codecs discretize speech into token sequences, but the resulting token spaces differ in vocabulary and codebook structure, preventing direct communication across models. This limitation affects applications such as conversational voice agents and speech to speech translation systems where multiple speech models must interact. As a result, transferring information between speech systems typically requires decoding to waveform audio and re-encoding with a second tokenizer, increasing latency and introducing potential information loss. To address these limitations, we present TokenMapper, a direction aware framework for direct token to token translation between heterogeneous speech tokenizers in the discrete domain. TokenMapper supports structurally mismatched token spaces, including mappings between single codebook and multi codebook representations, under a shared effective token rate. Experiments on GLM-4-Voice, MiMi and DualCodec show consistent cross model performance. Specifically, translation WER approaches native reconstructions within 2.5-6.8% absolute WER, human MOS for TokenMapper outputs ranges from 2.29 to 4.39, following the same direction level trends as UTMOS and end to end latency is reduced by 4.8-94.5% relative to waveform bridging, reaching up to 972 ms per utterance. These results provide a practical step toward cross model speech token interoperability without intermediate waveform reconstruction.

1 Introduction

TokenMapper addresses incompatible speech-token spaces by translating directly between heterogeneous tokenizers in the discrete domain. Its introduction presents cross-tokenizer translation as feasible for same-rate tokenizers, including structurally mismatched codebooks.

1.1 Related work

  • Neural audio codecs use varied quantization, semantic-acoustic, and temporal-resolution strategies, contributing to divergent tokenization designs.These differences accompany the incompatibility that prevents direct interaction across models.Source: §1 · p. 1

1 Introduction

  • 4.8-94.5% end-to-end latency reduction, up to 972 ms per utterance, is reported for direct token translation.TokenMapper avoids waveform decoding and re-encoding in the transfer path.Source: §1 · p. 2
  • TokenMapper uses direction-aware mappings across incompatible token spaces, including single-codebook and multi-codebook RVQ representations with differing vocabulary sizes and depths.The work focuses on source and target sequences with a shared effective temporal rate.Source: §1 · p. 2
  • Translation WER is 5.85-9.98%, and TokenMapper remains near native reconstructions.The introduction reports cross-tokenizer transfer across three same-rate, structurally different tokenizers.Source: §1 · p. 2

2 Method

TokenMapper translates between heterogeneous speech-token sequences directly in the discrete domain, using a shared effective token rate and direction-aware routing. Its routing adapts to the source and target codebook structures, and training predicts target tokens across codebooks and timesteps.

2.1 Problem Formulation

  • TokenMapper maps source token sequences to corresponding target-token sequences under a shared effective token rate, using a learned source-to-target direction embedding.Each source timestep is supervised against the corresponding target timestep; the formulation supports different codebook counts, while different or variable rates require an additional temporal-alignment module.Source: §2.1 · p. 3

2.2.1 Input Representation

  • Each input token embedding is summed with positional, codebook, and direction embeddings before entering the encoder.The direction embedding represents the source-to-target tokenizer pair.Source: §2.2.1 · pp. 3–4

2.2.2 Encoder Backbone

  • The encoder processes each codebook channel independently, restricting self-attention to the temporal axis.The resulting encoded streams are routed to the direction-specific output heads.Source: §2.2.2 · p. 4

2.2.3 Direction Specific Output Heads / 2.2.4 Single to Single Output Head

  • A direction-specific routing function transforms encoder outputs into streams for a one-layer Transformer head and a linear projection to the target vocabulary.When both tokenizers use one codebook, routing is the identity.Source: §2.2.3 · p. 4, §2.2.4 · p. 4

2.2.5 Multi to Single Output Head

  • For multi-codebook sources and a single-codebook target, routing uses learned attention pooling across source codebooks.This combines the multiple encoded streams for the target head.Source: §2.2.5 · p. 4

2.2.6 Multi to Multi Output Head

  • For multi-codebook sources and targets, each target stream concatenates a shared base stream—the first source codebook—with its corresponding encoded channel.The design reflects the first codebook’s typical association with higher-level semantic structure.Source: §2.2.6 · p. 4

2.2.7 Single to Multi Output Head

  • For a single-codebook source and multi-codebook target, routing duplicates the source stream across target codebooks.Figure 3 depicts this as broadcasting one encoder stream to all target codebooks.Source: §2.2.7 · pp. 4–5, Figure 3 · p. 4

2.3 Training Objective

  • Training minimizes token-level cross-entropy over all target codebooks and timesteps, comparing predicted logits with ground-truth target tokens.All directions share a conditioned encoder and use direction-specific output heads.Source: §2.3 · p. 5

3 Experiments

Experiments compare ordered translations among GLM-4-Voice, Moshi (MiMi), and DualCodec, spanning single- and multi-codebook tokenizers at a shared effective token rate. Evaluation uses LibriSpeech and VCTK and combines intelligibility and perceptual-quality measures.

3.1 Experimental Setup

  • All ordered source-to-target pairs among GLM-4-Voice, Moshi (MiMi), and DualCodec are evaluated, covering single- and eight-codebook tokenizers at a shared effective token rate.The shared rate permits one-to-one temporal supervision despite codebook and vocabulary mismatch.Source: §3.1 · p. 5

3.2.1 LibriSpeech

  • LibriSpeech training uses the train subset, with evaluation on test-clean, providing high-quality, consistently recorded audiobook speech.The corpus is described as a standard benchmark for speech representation learning and reconstruction quality.Source: §3.2.1 · p. 5

3.2.2 VCTK

  • VCTK adds 110 speakers with diverse accents, complementing LibriSpeech for evaluating translation across varied speaking styles and speaker characteristics.Each speaker reads approximately 400 utterances under controlled recording conditions.Source: §3.2.2 · p. 5

3.3 Evaluation Metrics

  • Evaluation combines intelligibility and perceptual-quality metrics with qualitative mel-spectrogram residual analysis.The residual analysis examines acoustic reconstruction across translation directions.Source: §3.3 · p. 5

3.3.1 Word Error Rate (WER)

  • WER compares Whisper large-v3 ASR transcripts of translated audio with ground-truth transcripts as a proxy for semantic and linguistic preservation.Hypotheses and references are normalized for punctuation and whitespace to match LibriSpeech transcription format.Source: §3.3.1 · p. 5

3.3.2 Perceptual quality: MOS and UTMOS

  • Perceptual quality is assessed with human MOS ratings on a 1–5 scale and UTMOS-v2 predictions of speech quality.The listening study involved 32 participants.Source: §3.3.2 · p. 5

4 Results

Across LibriSpeech and VCTK, TokenMapper's cross-model WER stays near native reconstruction, while perceptual quality varies more by translation direction. It also reduces end-to-end latency relative to waveform bridging.

4.1 Semantic preservation (WER)

  • Cross-model WER is 5.85–9.98% on LibriSpeech, a 2.56–5.96% degradation from native, and stays within 2.96–6.83% of native on VCTK.The reported pattern holds despite VCTK's greater speaker and accent variability.Source: §4.1 · p. 6

4.2 Acoustic quality (UTMOS)

  • On VCTK, translations into GLM reach UTMOS 3.38 and 3.33 against native GLM 3.39, while DualCodec→Moshi scores 1.95.The paper reports that acoustic quality is more sensitive to tokenizer mismatch than semantic content transfer.Source: §4.2 · p. 6, Table 4 · p. 6

4.3 Human perceptual evaluation (MOS)

  • Human MOS follows the automatic quality trends: translations into GLM score 4.39 and 4.36 versus native 4.50, while DualCodec→Moshi scores 2.29.The listening study involved 32 participants.Source: §4.3 · pp. 6–7, Table 5 · p. 7

4.4 Latency reduction

  • Latency reductions range from 4.8% to 94.5%, reaching 94.5% for GLM→DualCodec when waveform bridging is dominated by neural decoding.TokenMapper bypasses source-side waveform synthesis and target-side re-tokenization; the comparison uses offline end-to-end transfer-path latency.Source: §4.4 · p. 7, Table 6 · p. 7

5 Ablations

Ablations test direction conditioning, output-head design, codebook structure, and context handling across single-to-multi, multi-to-single, and multi-to-multi translation directions.

5 Ablations

  • Removing direction conditioning raises WER from 9.31% to 57.62% for GLM→Moshi and from 7.91% to 77.21% for Moshi→DualCodec.The evaluations cover single-to-multi and multi-to-multi directions.Source: §5 · pp. 7–8
  • A single shared output head raises Moshi→GLM WER from 8.55% to 83.01%, while an MLP head raises GLM→Moshi WER from 9.31% to 74.23%.These comparisons test direction-specific projections and Transformer-based output heads, respectively.Source: §5 · pp. 7–8
  • Cross-codebook attention raises WER from 8.55% to 18.86% and from 7.91% to 22.83%, while adding significant computational overhead.The ablation enables self-attention across time and codebook dimensions.Source: §5 · pp. 7–8
  • Full-context concatenation raises multi-to-multi WER from 7.91% to 26.79%; the simple supervised token-domain baseline nearly fails without TokenMapper's routing and structured output design.The baseline removes the direction-aware shared encoder, structured routing, and Transformer output heads.Source: §5 · pp. 7–8

6 Conclusion

TokenMapper translates directly between heterogeneous speech tokenizers, preserving semantic content with modest degradation and reducing transfer latency. The results establish feasibility in the studied same-rate setting, while perceptual quality is more challenging for multi-codebook targets.

6 Conclusion

  • On LibriSpeech, cross-model WER increases by 2.56–5.96% absolute over native reconstruction; on VCTK, the gap is 2.96–6.83%.The paper reports these results as evidence that learned mappings capture transferable tokenizer-level structure beyond dataset-specific acoustics.Source: §6 · p. 8
  • Translations into GLM remain close to native GLM reconstruction, while RVQ-based targets, especially Moshi, are more challenging.The authors interpret this pattern as semantic information transferring more readily than fine acoustic detail encoded across residual codebooks.Source: §6 · p. 8
  • TokenMapper reduces offline transfer-path latency by up to 94.5%, with the largest gains where waveform bridging is dominated by neural decoding.It bypasses source-side waveform synthesis and target-side re-tokenization.Source: §6 · p. 8
  • The results show direct token-domain interoperability is feasible for the studied same-rate setting, with extensions to other rates, tokenizer families, and streaming left for future work.The paper also identifies stronger acoustic-detail recovery as future work.Source: §6 · p. 8

7 Limitations

The experiments cover same-rate speech tokenizers and paired utterances, leaving different frame rates and strong domain shifts outside the evaluation. Reproducibility details specify paired-token construction, training, checkpoint selection, and evaluation procedures.

7 Limitations

  • The study covers three same-rate tokenizers and six directions, assumes paired same-utterance sequences and access to tokenizer checkpoints, and does not evaluate strong domain shifts.Different frame rates require an additional temporal-alignment module; the study also notes greater difficulty recovering fine residual acoustic detail across heterogeneous RVQ hierarchies.Source: §7 · pp. 8–9

A Additional Reproducibility Details

  • Paired training examples come from encoding the same utterance with each tokenizer; tokenizer-specific sample-rate conversion and normalization follow each tokenizer’s inference pipeline.Token tensors are matched by utterance or file ID, with temporal padding masks applied consistently to the token-level loss.Source: A Additional Reproducibility Details · pp. 12–13
  • Training uses a shared configuration across directions, selects checkpoints by validation cross-entropy, and evaluates on LibriSpeech test-clean and VCTK.WER uses Whisper large-v3 transcripts; UTMOS uses UTMOS-v2, while MOS and latency protocols are specified separately.Source: A Additional Reproducibility Details · pp. 12–13

B Human MOS Evaluation

The human MOS study compares TokenMapper outputs with native target-tokenizer reconstructions, and its ratings follow the automatic perceptual-metric trends. The section also reports preliminary adaptation to a new target and defines latency measurements as offline transfer-path comparisons.

B Human MOS Evaluation

  • Translations into GLM rated close to native GLM, while Moshi-targeted outputs—especially DualCodec→Moshi—were most challenging among the 32 listeners.The study used 15 samples, rated on a 1–5 scale; MOS compares outputs with achievable target-tokenizer reconstruction quality, not clean speech.Source: B Human MOS Evaluation · p. 13

C Adaptation to an Additional Target Tokenizer

D Latency Protocol and Module Level Timing

  • Latency measurements compare offline utterance-level transfer paths, not streaming or real-time deployment; module timings separately exclude tokenizer encoding and decoding.The path comparison omits intermediate source-side waveform synthesis and target-side re-tokenization for TokenMapper; module timing includes token transfer, forward inference, and argmax conversion.Source: D Latency Protocol and Module Level… · p. 14

E Token domain Baseline

The supervised token-domain baseline tests whether local token correspondences suffice. Its near-failure performance supports the need for sequence-aware, direction-conditioned, and codebook-aware modeling.

E Token domain Baseline

  • The baseline uses paired source-target token data but replaces TokenMapper’s structured components with lightweight local token mapping.It tests whether direct token correspondences alone are sufficient.Source: E Token domain Baseline · p. 14
  • The local token-mapping baseline yielded WER close to 100% and severely degraded UTMOS, supporting sequence-aware, direction-conditioned, and codebook-aware modeling.The baseline removes TokenMapper’s direction-aware shared encoder, structured routing, and Transformer output heads.Source: E Token domain Baseline · p. 14

F Scope, Data Dependence, and Rate Mismatch

TokenMapper’s direction embeddings remain distinct despite occupying a compact region, while the formulation is limited to paired same-utterance sequences at a shared effective token rate. The discussion identifies domain mismatch as a further boundary on the scope of the results.

F Scope, Data Dependence, and Rate Mismatch

  • The formulation assumes paired same-utterance token sequences at a shared effective rate, so it does not address different frame rates or variable-rate streams.The authors suggest temporal alignment as a possible extension.Source: F Scope, Data Dependence, and Rate… · p. 14
  • Supervised training requires paired tokenized data, obtained by encoding the same utterances with each tokenizer and matching tensors by utterance or file identity.Tokenizer-version changes or a new target tokenizer may require adapting this pairing.Source: F Scope, Data Dependence, and Rate… · p. 14
  • VCTK results suggest robustness beyond LibriSpeech speakers and audiobook conditions, but do not establish robustness to full domain shift; music-versus-speech tokenizers may encode substantially different factors.Semi-supervised, weakly paired, or alignment-based objectives are proposed as possible ways to reduce pairing dependence.Source: F Scope, Data Dependence, and Rate… · pp. 14–15

F.1 Direction Embedding Analysis

  • Direction embeddings occupy a compact but non-collapsed region, with separation by source or target tokenizer and by single- versus multi-codebook structure.The authors describe this as diagnostic evidence of direction-dependent transformations, not proof of a universal shared token space.Source: §F.1 · p. 15

G Qualitative Spectrogram Analysis

Qualitative mel-spectrogram comparisons show that TokenMapper generally retains broad speech organization, with closer visual agreement for mappings into GLM and more visible fine-texture differences for multi-codebook targets. The comparisons isolate mapping-related acoustic differences from the target tokenizer’s own reconstruction characteristics.

G Qualitative Spectrogram Analysis

  • Each direction’s spectrogram rows compare TokenMapper, native-target, and ground-truth signals; frequency-wise differences help distinguish mapping-related changes from target reconstruction differences.The figures show three example utterances per direction, with WER and UTMOS values above the spectrograms.Source: G Qualitative Spectrogram Analysis · p. 15
  • Mappings into GLM show particularly close visual agreement with native target spectrograms, while speech-active regions, silences, and dominant harmonic patterns generally remain aligned.The comparisons cover the 0–8000 Hz range and show TokenMapper, native-target, and ground-truth reconstructions.Source: G Qualitative Spectrogram Analysis · pp. 15–16, Figure 5 · p. 17, Figure 9 · p. 19
  • Mappings into Moshi and DualCodec retain broad speech structure but show more fine-texture differences, especially at high frequencies and in harmonic sharpness and intensity.TokenMapper reconstructions usually remain structurally closer to the native target reconstruction than to an unrelated acoustic representation.Source: G Qualitative Spectrogram Analysis · p. 16, Figure 6 · p. 17, Figure 7 · p. 18
Loading 2609.12563v1…