PaperlayerView exact revision on arXiv

Source-linked AI summary

FuseCodec: Semantic-Contextual Fusion and Supervision for Neural Codecs

Md Mubtasim Ahasan, Rafat Hasan Khan, Tasnim Mohiuddin, Aman Chadha, Tariq Iqbal, M Ashraful Amin, Amin Ahsan Ali, Md Mofijul Islam, A K M Mahbubur Rahman

arXiv:2509.11425v2cs.SDcs.AIcs.CLeess.AS

TL;DR

Neural codecs capture acoustic features but leave semantic and contextual cues, and their alignment, incompletely addressed. FuseCodec fuses and supervises these signals, reporting state-of-the-art LibriSpeech results and strong Codec-SUPERB performance. The paper also reports applicability to zero-shot speech synthesis.

  • Problem

    Neural codecs capture low-level acoustic features, while existing approaches do not fully unify semantic and contextual representations or align speech and text modalities.

    Source: §1 · pp. 1–2

  • Method

    FuseCodec combines acoustic, semantic, and contextual signals through latent-space fusion, global supervision, and temporally aligned contextual supervision.

    Source: §1 · p. 2

  • Results

    FuseCodec reports state-of-the-art LibriSpeech results and strong Codec-SUPERB performance, and extends to zero-shot speech synthesis.

    Source: §1 · p. 2, §4 · p. 10

  • Takeaways & Limitations

    The findings highlight the value of semantic and contextual grounding in discrete speech modeling.

    Source: §4 · p. 10

Abstract

from arXiv · show

Speech tokenization enables discrete representation and facilitates speech language modeling. However, existing neural codecs capture low-level acoustic features, overlooking the semantic and contextual cues inherent to human speech. While recent efforts introduced semantic representations from self-supervised speech models or incorporated contextual representations from pre-trained language models, challenges remain in aligning and unifying the semantic and contextual representations. We introduce FuseCodec, which unifies acoustic, semantic, and contextual representations through strong cross-modal alignment and globally informed supervision. We propose three complementary techniques: (i) Latent Representation Fusion, integrating semantic and contextual features directly into the encoder latent space for robust and unified representation learning; (ii) Global Semantic-Contextual Supervision, supervising discrete tokens with globally pooled and broadcasted representations to enhance temporal consistency and cross-modal alignment; and (iii) Temporally Aligned Contextual Supervision, strengthening alignment by dynamically matching contextual and speech tokens within a local window for fine-grained token-level supervision. We further introduce FuseCodec-TTS, demonstrating our methodology's applicability to zero-shot speech synthesis. Empirically, FuseCodec achieves state-of-the-art performance in LibriSpeech, surpassing EnCodec, SpeechTokenizer, and DAC in transcription accuracy, perceptual quality, intelligibility, and speaker similarity. Results highlight the effectiveness of contextually and semantically guided tokenization for speech tokenization and downstream tasks. Code and pretrained models are available at https://github.com/mubtasimahasan/FuseCodec.

1 INTRODUCTION

Neural codecs provide discrete speech representations but are limited in capturing and aligning acoustic, semantic, and contextual information. FuseCodec combines these signals through latent fusion and supervision, and reports strong benchmark and downstream text-to-speech results.

1 INTRODUCTION

  • Neural codecs capture acoustic features but struggle with high-level semantics; prior approaches also leave contextual grounding and effective cross-modal alignment unresolved.The comparison identifies acoustic and partial semantic coverage without contextual grounding, direct integration, or modality alignment in most prior codecs.Source: §1 · pp. 1–2, Table 1 · p. 2
  • FuseCodec combines latent-space integration, globally pooled semantic-contextual supervision, and dynamic temporal matching to unify speech and language representations.Its three variants are FuseCodec-Fusion, FuseCodec-Distill, and FuseCodec-ContextAlign.Source: §1 · p. 2, Figure 1 · p. 3
  • FuseCodec reports state-of-the-art LibriSpeech results over EnCodec, SpeechTokenizer, and DM-Codec in intelligibility and perceptual quality, plus strong Codec-SUPERB performance at 4 kbps.It also reports zero-shot speech-synthesis applicability.Source: §1 · p. 2
  • The paper validates its components through ablations and demonstrates practical utility in downstream text-to-speech generation.The framework is also described as substantially improving speech reconstruction and representation quality.Source: §1 · p. 2

2 FUSECODEC

FuseCodec addresses the difficulty of jointly representing acoustic, semantic, and contextual speech information with three strategies for unifying those signals and aligning them with discrete tokens. The section also describes the codec pipeline, training objective, and extension to zero-shot TTS.

2.1 DISCRETE SPEECH REPRESENTATION

  • An encoder compresses speech into latent frames, and a residual vector quantizer converts them into discrete token indices across quantization layers.The quantized vectors are mapped to codebook embeddings.Source: §2.1 · p. 3

2.2 MULTIMODAL REPRESENTATION EXTRACTION

  • FuseCodec extracts contextual representations from a pre-trained language model and semantic representations from a self-supervised speech model.Contextual embeddings adapt to surrounding text; semantic embeddings capture high-level structure and meaning.Source: §2.2 · pp. 3–4

2.3.1 LATENT REPRESENTATION FUSION

  • Latent Representation Fusion pools semantic and contextual features, broadcasts them across token positions, and uses cross-attention and additive fusion before quantization.Modality dropout during training helps prevent over-reliance on fused signals and permits inference using only the encoder signal.Source: §2.3.1 · p. 4

2.3.2 GLOBAL SEMANTIC-CONTEXTUAL SUPERVISION

  • Global Semantic-Contextual Supervision broadcasts global semantic and contextual vectors to supervise RVQ outputs across time, promoting temporal consistency and cross-modal coherence.The method applies distillation along the temporal rather than feature axis.Source: §2.3.2 · pp. 4–5

2.3.3 TEMPORALLY ALIGNED CONTEXTUAL SUPERVISION

  • Temporally Aligned Contextual Supervision addresses unequal sequence lengths by matching contextual embeddings to the most similar RVQ tokens within shifting local windows.A contextual embedding can be assigned to multiple tokens when they share the maximum similarity.Source: §2.3.3 · p. 5

2.4 ARCHITECTURE AND TRAINING OBJECTIVE

  • The method retains a standard neural-codec architecture and adds semantic-contextual fusion or supervision without changing that architecture.The training objective combines reconstruction, adversarial, feature-matching, and commitment losses; the distillation loss depends on the FuseCodec variant.Source: §2.4 · pp. 5–6

2.5 DOWNSTREAM EXTENSION TO TTS MODEL

  • FuseCodec-TTS predicts quantized acoustic tokens from phonemes and a FuseCodec-extracted acoustic prompt, with first-layer tokens predicted autoregressively and higher-layer tokens nonautoregressively.The predicted tokens are decoded by FuseCodec to synthesize speech.Source: §2.5 · p. 6

3 EXPERIMENTS

Experiments evaluate FuseCodec on speech reconstruction, representation quality, zero-shot TTS, and additional tests. FuseCodec variants show strong results across these settings, with outcomes varying by variant and task.

3.1 EXPERIMENTAL SETUP

  • FuseCodec is trained on LibriSpeech train-clean-100; FuseCodec-TTS uses LibriTTS, and comparisons use official released baseline checkpoints.The evaluation covers content preservation and speech naturalness, with reference-based metrics omitted for TTS because references are unavailable.Source: §3.1 · pp. 6–7

3.2.1 SPEECH RECONSTRUCTION EVALUATION

  • FuseCodec-Fusion achieves the strongest reconstruction performance, recording the lowest WER (3.99) and WIL (6.45), and the highest STOI (0.95), ViSQOL (3.47), and PESQ (3.13).Source: §3.2.1 · p. 7
  • FuseCodec-Distill achieves top UTMOS (3.65) and Similarity (0.996), while FuseCodec-ContextAlign matches top UTMOS (3.65) and improves over the baseline across all reported metrics.ContextAlign remains slightly below Fusion and Distill overall; its reported scores include WER (6.70), WIL (4.15), STOI (0.94), ViSQOL (3.18), and PESQ (2.85).Source: §3.2.1 · p. 8

3.2.2 REPRESENTATION QUALITY EVALUATION

  • On Codec-SUPERB, FuseCodec-Fusion leads signal-level performance with Audio (0.785) and second-highest Speech (0.744), while Distill leads AEC (57.25) and Fusion leads ER (73.96).At 4 kbps, FuseCodec variants are strong across metrics, although DAC has lower ASR error (3.26) and FunCodec the lowest ASV error (1.60).Source: Table 3 · p. 8, §3.2.2 · pp. 8–9

3.2.3 DOWNSTREAM SPEECH GENERATION EVALUATION

  • In zero-shot TTS, Distill-TTS has the lowest WER (8.55 / 3.66) and WIL (12.07 / 6.02), while ContextAlign-TTS has the best UTMOS scores (3.86 / 3.96).Fusion-TTS offers a well-rounded trade-off, with joint-best similarity (0.83 / 0.79), competitive UTMOS, and second-best WER/WIL.Source: Table 4 · p. 9, §3.2.3 · p. 9

3.3 ADDITIONAL AND ABLATION STUDY RESULTS

  • On seven unseen languages, Fusion achieves the strongest content and perceptual scores, with Distill second-best; ablations report benefits from cross-before projection, dual-signal guidance, temporal alignment, 10% dropout, and first-layer supervision.The ablation findings are reported in Appendix D.Source: §3.3 · p. 9

4 CONCLUSION

FuseCodec unifies acoustic, semantic, and contextual signals through multimodal fusion and supervision, achieving state-of-the-art speech reconstruction results.

4 CONCLUSION

  • FuseCodec integrates acoustic, semantic, and contextual signals through multimodal representation fusion and supervision.The methods enable fine-grained alignment.Source: §4 · p. 10
  • FuseCodec achieves state-of-the-art speech reconstruction results, improving intelligibility, quality, and speaker similarity.The paper highlights semantic and contextual grounding in discrete speech modeling.Source: §4 · p. 10
  • The findings highlight the value of semantic and contextual grounding in discrete speech modeling.Source: §4 · p. 10

5 REPRODUCIBILITY STATEMENT

The paper describes resources for reproducing its models and results, reviews related work, and points to additional multilingual reconstruction results.

5 REPRODUCIBILITY STATEMENT

  • The reproducibility materials include a Docker environment, code, trained checkpoints, preprocessing and training scripts, and dependencies.The experimental setup is described in Section 3.1, and resources are linked in Appendix A.Source: §5 · p. 10

A RESOURCES

  • Appendix A lists the Docker environment, code and configuration, and trained model checkpoints as replication resources.Source: A RESOURCES · p. 14

B RELATED WORK

  • The related-work review covers codec and self-supervised speech representation methods alongside language-model-based speech synthesis.It discusses approaches including VQ-VAE, HuBERT, SoundStream, AudioLM, VALL-E, EnCodec, DAC, and SpeechTokenizer.Source: B RELATED WORK · p. 14

C ADDITIONAL RESULTS

C.1 EXTENSION TO UNSEEN MULTILINGUAL SPEECH RECONSTRUCTION

The appendix evaluates FuseCodec variants on multilingual speech reconstruction for unseen languages and describes the controlled setup for its ablation studies.

C.1 EXTENSION TO UNSEEN MULTILINGUAL SPEECH RECONSTRUCTION

  • The multilingual evaluation uses German, Dutch, Spanish, French, Italian, Portuguese, and Polish test data, with baselines selected for comparable codec configurations.The comparison includes SpeechTokenizer, EnCodec, DM-Codec, and FaCodec.Source: §C.1 · pp. 14–15
  • Across seven unseen languages, FuseCodec variants rank first or second for content accuracy, while Fusion leads perceptual metrics and improves PESQ by at least 0.3 over the next-best model.Fusion achieves the highest PESQ and ViSQOL scores across all languages; Distill is consistently second-best on these perceptual metrics.Source: §C.1 · p. 15, Table 5 · p. 15

D ABLATION STUDIES

  • The ablation studies vary specific design choices while keeping hyperparameters, training procedures, and configurations fixed.Source: D ABLATION STUDIES · p. 15

D.1 ABLATION: ATTENTION-PROJECTION CONFIGURATION IN REPRESENTATION FUSION

The ablation compares attention and projection configurations in latent fusion, with Cross-Before yielding the strongest combined intelligibility and naturalness results.

D.1 ABLATION: ATTENTION-PROJECTION CONFIGURATION IN REPRESENTATION FUSION

  • STOI (0.95), ViSQOL (3.47), and PESQ (3.13) are highest for Cross-Before, which also has second-best UTMOS (3.63).Source: Table 6 · p. 15, §D.1 · p. 16
  • The ablation compares cross- and self-attention before or after projection, plus direct projection without attention.Cross variants exchange semantic and contextual information; Self variants apply self-attention.Source: Table 6 · p. 15, §D.1 · p. 16
  • Self-Before achieves the best WER (3.92) and WIL (6.36), but lacks explicit cross-modal exchange and trails on naturalness metrics including UTMOS and PESQ.Source: §D.1 · p. 16
  • Cross-Before applies cross-modal attention in the lower-dimensional space before projection, capturing richer semantic-contextual interactions for intelligibility and perceptual quality.The results identify attention before projection, especially cross-modal attention, as important for extracting the benefit of semantic-contextual signals during fusion.Source: §D.1 · p. 16
  • Cross-After performs poorly, while removing attention gives the weakest intelligibility and perceptual scores despite the highest UTMOS.Source: §D.1 · p. 16

D.2 ABLATION: ATTENTION-GUIDANCE CONFIGURATION IN SEMANTIC-CONTEXTUAL GUIDANCE

This ablation tests attention and guidance choices for semantic-contextual distillation; direct supervision with both global signals performs best, while adding cross-attention or omitting semantic guidance degrades performance.

D.2 ABLATION: ATTENTION-GUIDANCE CONFIGURATION IN SEMANTIC-CONTEXTUAL GUIDANCE

  • None + Semantic-Contextual directly compares projected first-layer RVQ tokens with both semantic and contextual guidance vectors, without attention.The ablation also tests variants that omit semantic guidance, add cross-attention to contextual guidance, or combine cross-attention with both signals.Source: §D.2 · p. 17
  • WER 4.09 and WIL 6.60 were lowest, while STOI 0.940, ViSQOL 3.43, PESQ 3.06, UTMOS 3.65, and Similarity 0.996 were highest for None + Semantic-Contextual.The comparison covers four attention and guidance configurations.Source: §D.2 · p. 17, Table 7 · p. 17
  • Excluding semantic guidance or using attention degrades performance across all reported metrics, despite Cross + Contextual being the second-best configuration.The results identify direct supervision with both guidance signals as the strongest configuration.Source: §D.2 · p. 17
  • Contextual-only variants perform worse, indicating that semantic signals help guide representations toward higher-level content fidelity and intelligibility.The authors conclude that the original global guidance signals should be applied directly, without attention, for stable timestep-aligned distillation.Source: §D.2 · p. 17

D.3 ABLATION: FIXED VS. DYNAMIC WINDOW CONFIGURATION IN TEMPORAL ALIGNMENT

This ablation compares fixed and dynamic windows for contextual token alignment, finding that dynamic alignment performs best on content-preservation measures while fixed windows constrain fine-grained temporal matching.

D.3 ABLATION: FIXED VS. DYNAMIC WINDOW CONFIGURATION IN TEMPORAL ALIGNMENT

  • Dynamic-window Contextual adaptively matches each contextual embedding to a local RVQ-token region using token similarity and the previous match to guide the next search range.The resulting aligned sequence is used for timestep-level distillation.Source: §D.3 · p. 18
  • The fixed-window variants align each contextual embedding to its most similar RVQ token within a predefined window; the Semantic-Contextual variant adds frame-aligned semantic supervision.The fixed window size is w = ⌊T′/n⌋, where T′ is RVQ sequence length and n is the number of contextual embeddings.Source: §D.3 · p. 18
  • WER 4.15 and WIL 6.70 were lowest, while STOI 0.93 was highest for Dynamic-window Contextual; it also had the best PESQ (2.85), high ViSQOL (3.18), and top Similarity (0.995).Dynamic-window Semantic-Contextual achieved the best UTMOS (3.75), second-best WER (4.21) and WIL (6.78), and matched the top Similarity.Source: §D.3 · p. 18
  • Both fixed-window configurations score lower across most metrics, particularly Fixed-window Semantic-Contextual, with the lowest ViSQOL (3.10) and PESQ (2.62) despite UTMOS 3.74.Fixed windows impose rigid alignment and do not account for local content variation; adding semantic supervision does not compensate for that constraint.Source: §D.3 · pp. 18–19
  • Both Semantic-Contextual variants improve UTMOS, but semantic-only supervision without dynamically aligned contextual guidance does not improve content accuracy.The authors conclude that dynamic alignment is important for contextual guidance, while semantic supervision can enhance fluency and naturalness without compromising content preservation only when paired with flexible alignment.Source: §D.3 · p. 19

D.4 ABLATION: DROPOUT MASK CONFIGURATION IN REPRESENTATION FUSION

This ablation varies stochastic modality dropout during latent fusion; 10% dropout performs best overall, while greater masking weakens content preservation and perceptual quality.

D.4 ABLATION: DROPOUT MASK CONFIGURATION IN REPRESENTATION FUSION

  • During training, independent element-wise masks are applied to projected semantic and contextual vectors before they are added to the encoder latent representation.The selected 10% setting gives each element in each mask a 10% chance of being set to zero.Source: §D.4 · p. 19
  • WER 3.99 and WIL 6.45 were lowest, while STOI 0.95, ViSQOL 3.47, and PESQ 3.13 were highest with 10% dropout.At 50% dropout, UTMOS (3.66) and Similarity (0.996) had minor gains, while increasing dropout to 30–90% worsened most content-preservation and naturalness metrics.Source: §D.4 · p. 19, Table 9 · p. 19
  • Increasing dropout removes more auxiliary modality information, reducing latent-token alignment with multimodal supervision and harming intelligibility and perceptual quality.The authors attribute the best trade-off to light dropout, which retains enough auxiliary representations for multimodal grounding.Source: §D.4 · pp. 19–20
  • UTMOS and Similarity remain relatively stable or improve at moderate dropout, suggesting speaker and prosodic features persist in the base latents even as content preservation worsens.This distinction indicates that stability in those measures does not imply preservation of semantic-contextual information.Source: §D.4 · p. 20

D.5 ABLATION: QUNTIZER LAYER CONFIGURATION IN SEMANTIC-CONTEXTUAL GUIDANCE

The ablation finds first-layer RVQ supervision stronger than averaging all layers for semantic-contextual distillation; the accompanying tokenizer details describe its codec backbone and components.

D.5 ABLATION: QUNTIZER LAYER CONFIGURATION IN SEMANTIC-CONTEXTUAL GUIDANCE

  • First-layer supervision yields the strongest reported performance for both global Distill and temporally aligned ContextAlign; all-layer supervision degrades most metrics in both settings.For Distill, first-layer supervision gives WER 4.09, WIL 6.60, STOI 0.94, ViSQOL 3.43, PESQ 3.06, UTMOS 3.65, and Similarity 0.996; ContextAlign improves WER, WIL, ViSQOL, PESQ, and Similarity over all-layer supervision.Source: §D.5 · p. 20, Table 10 · p. 20
  • First-layer RVQ tokens provide more abstract representations aligned with semantic intent and global context, whereas deeper layers encode acoustic and residual details that mismatch global guidance when averaged.UTMOS and Similarity remain relatively stable with all-layer supervision, but are insufficient to guide semantic alignment.Source: §D.5 · p. 21

E.1 MODEL DETAILS

  • The tokenizer uses a standard neural-codec architecture, with a convolutional and BiLSTM encoder, a mirrored decoder, and an eight-codebook residual vector quantizer.Each codebook has 1024 entries; successive codebooks quantize residual errors, and the decoder reconstructs speech from summed codebook embeddings.Source: §E.1 · p. 21
  • Three discriminators guide speech reconstruction: MS-STFT operates on multiple-resolution complex STFTs, MSD processes waveform scales, and MPD models periodic waveform structure.The MSD and MPD channel counts are standardized to match the MS-STFT discriminator.Source: §E.1 · pp. 21–22

E.2 TRAINING OBJECTIVE

FuseCodec’s training objective draws on established techniques and includes reconstruction, adversarial, feature-matching, and commitment losses. The qualitative comparison presents spectrograms for FuseCodec and baseline models across distinct speech samples.

E.2 TRAINING OBJECTIVE

  • Reconstruction losses compare original and reconstructed speech in the time domain and across multiple Mel-spectrogram resolutions.The Mel-spectrograms use 64 bins and STFT scales indexed from 5 to 11.Source: §E.2 · p. 22
  • A GAN-based adversarial objective uses multi-period, multi-scale, and multi-scale STFT discriminators.The passage states that this objective is adopted to reduce the discriminability of reconstructed speech.Source: §E.2 · p. 22
  • Feature matching compares intermediate discriminator features to stabilize training and align those features.Source: §E.2 · p. 22
  • A commitment penalty encourages encoder outputs to remain close to their quantized representations during residual vector quantization.Source: §E.2 · p. 22

F QUALITATIVE COMPARISON

  • Figure 2 arranges spectrograms by model and speech sample for qualitative reconstruction comparison, with images linking to the corresponding audio.The compared models include FuseCodec, SpeechTokenizer, DM-Codec, and EnCodec.Source: F QUALITATIVE COMPARISON · p. 23, Figure 2 · p. 23
Loading 2509.11425v2…