PaperlayerView exact revision on arXiv

Source-linked AI summary

LACE: Layer-Wise Compression for Dynamic Frame Rate Codecs

Thanapat Trachu, Samuele Cornell, William Chen, Shinji Watanabe

arXiv:2609.17509v1cs.SDcs.AIcs.CL

TL;DR

High frame rates make neural-audio-codec sequences costly, while prior multi-layer dynamic codecs force all quantization layers to share segmentation boundaries. LACE compresses each layer independently and adds union alignment plus boundary anchor for consistent TTS durations; on LibriTTS, it improves the rate-quality tradeoff and TTS inference efficiency while maintaining competitive synthesis quality.

  • Problem

    High frame rates increase computational cost, and shared segmentation boundaries may be suboptimal because residual embeddings change at different rates across quantization layers.

  • Method

    LACE independently compresses residual embeddings at each quantization layer, using union alignment and boundary anchor to make durations consistent for downstream TTS.

  • Results

    On LibriTTS, LACE offers a better rate-quality tradeoff than prior dynamic frame-rate methods and improves TTS inference efficiency while maintaining competitive synthesis quality.

  • Takeaways & Limitations

    Layer-specific compression preserves the dynamic frame-rate approach while supporting downstream TTS with consistent durations.

Abstract

from arXiv · show

Neural audio codecs are a key component in speech language modeling. However, their high frame rates lead to long sequence lengths, increasing computational costs. Dynamic frame rate codecs mitigate this by reducing the effective frame rate using a compression step to merge multiple frames together. However, most prior methods either operate on single-codebook codecs or apply a single compression step before multi-layer quantization. This forces all quantization layers to share the same segmentation boundaries, despite the residual embeddings at different quantization layers exhibiting different rates of change over time. We propose LACE (Layer-Adaptive Codec Encoding), a dynamic frame rate codec that applies an independent compression step at each quantization layer, enabling layer-specific segmentation boundaries. To use LACE tokens in downstream text-to-speech (TTS), we further introduce union alignment and boundary anchor mechanisms to make durations consistent across layers while preserving compression benefits. Experiments on LibriTTS show that LACE offers a better rate-quality tradeoff than prior dynamic frame rate methods on the reconstruction task and improves TTS inference efficiency while maintaining competitive synthesis quality. Our code is released as part of the ESPnet3 codec recipe.

I. INTRODUCTION

High frame rates make neural audio codec sequences computationally costly and harder to align with speech-language-modeling needs. LACE addresses shared-boundary limitations by compressing residual embeddings independently at each quantization layer and adding alignment mechanisms for downstream TTS.

  • I. INTRODUCTION: Long discrete-code sequences increase transformer computation and can hinder speech-language modeling when speech sequences substantially exceed text-token lengths.
  • I. INTRODUCTION: Dynamic frame-rate codecs reduce code counts by grouping frames into variable-length segments represented by one code and a duration.Effective frame rate counts segments per second; assigning longer durations to lower-information regions reduces the number of speech codes.
  • I. INTRODUCTION: LACE independently compresses residual embeddings at each quantization layer, allowing layer-specific segmentation boundaries.Earlier RVQ layers capture main signal structure while deeper layers encode finer details, so their representations can change at different rates over time.
  • I. INTRODUCTION: Union alignment and boundary anchor make layer-specific LACE durations consistent for downstream TTS while controlling effective-frame-rate growth.Union alignment shares boundaries across layers, while boundary anchor limits which deeper layers may introduce new boundaries.
  • I. INTRODUCTION: LACE has a lower upper bound on expected quantization error than single compression and improves the rate-quality tradeoff and TTS inference efficiency on LibriTTS.The reported experiments maintain competitive synthesis quality while improving TTS inference efficiency.

B. Compression in Dynamic Frame-Rate Codecs

Dynamic frame-rate compression converts fixed-rate frame embeddings into variable-length segments with codes and durations. The paper describes several boundary-generation methods and uses layer-wise compression to let LACE operate with layer-specific segmentations.

  • B. Compression in Dynamic Frame-Rate Codecs: Compression groups T frame embeddings into M variable-length segments, averages embeddings within each segment, and pairs each segment with a duration.The duration of segment m is d_m = b_m − b_{m−1}, where B contains sorted segmentation boundaries.
  • B. Compression in Dynamic Frame-Rate Codecs: Dynamic programming chooses boundaries by minimizing reconstruction loss subject to segment constraints and recovers them by backtracking optimal segment lengths.The recurrence records the minimizing segment length for each state and reconstructs boundaries from the final frame and segment count.
  • B. Compression in Dynamic Frame-Rate Codecs: Cosine-similarity compression places boundaries where adjacent-frame similarity falls below a threshold, so the segment count is controlled implicitly.
  • B. Compression in Dynamic Frame-Rate Codecs: Density-peak compression identifies segment centers from local density and peak-distance scores, then expands centers to adjacent frames meeting similarity and size criteria.The process repeats until all frames are assigned, after which segment endpoints form the boundary set.
  • B. Compression in Dynamic Frame-Rate Codecs: LACE applies the compression step independently to each layer’s residual embeddings, producing layer-specific boundaries, durations, segment embeddings, and codes.The approach is agnostic to the compression model used for each layer.
  • B. Compression in Dynamic Frame-Rate Codecs: The LACE overview depicts independently compressed residual embeddings at each quantization layer, with same-colored frames belonging to one segment.

III. METHOD

LACE compresses residual embeddings independently at each quantization layer, allowing layer-specific boundaries, then aligns those boundaries for consistent downstream durations.

  • A. Layer-Wise Compression: Layer-wise compression reflects differing temporal structure across residual layers, unlike prior methods that force shared segmentation boundaries.The shared-boundary constraint can be suboptimal for multi-codebook residual quantization.
  • A. Layer-Wise Compression: LACE applies compression independently to each layer’s residual embeddings, producing layer-specific segment embeddings, durations, boundaries, and codes.Each layer’s compressed residuals are passed to that layer’s quantizer.
  • B. Union Alignment: Union alignment takes the union of all layer boundaries and re-segments every layer so all codes share boundaries and durations.A union boundary inside an existing segment splits that segment while preserving its code and assigning new durations.
  • B. Union Alignment: Union alignment resolves duration inconsistency for downstream TTS but can increase the effective frame rate by adding boundaries.The resulting aligned representation preserves layer-wise compression while making durations consistent.

C. Boundary Anchor

Boundary anchor limits the frame-rate growth caused by union alignment by restricting deeper layers to reuse boundaries from earlier layers, with a tunable quality-rate tradeoff.

  • C. Boundary Anchor: Union alignment can approach the uncompressed frame rate when layers introduce distinct boundaries, potentially negating compression benefits.Its boundary count is at least as large as every individual layer’s boundary count.
  • C. Boundary Anchor: Boundary anchor permits new boundaries only through an anchor layer l∗ and restricts deeper layers to positions already present in the anchor boundary set.The restriction limits growth of the union boundary set and is model-agnostic beyond the DP implementation.
  • C. Boundary Anchor: The anchor layer l∗ is a tunable hyperparameter that trades off reconstruction quality against effective frame rate.Higher anchor depth allows more layers to introduce boundaries, whereas stricter reuse limits boundary growth.
  • D. Theoretical Analysis: Theoretical analysis assumes bounded layerwise quantization error and a positive lower bound on compression-captured residual energy.Under these assumptions, the expected residual error follows a multiplicative hierarchy governed by λ_l < 1.
  • D. Theoretical Analysis: Full layer-wise compression drives the theoretical error bound to zero, unlike single compression, which retains an error floor from shared boundaries.The bound concerns codec quantization error, not downstream TTS effects from union alignment.

E. TTS Training with LACE Tokens

LACE TTS predicts durations only for the first quantization layer after union alignment, reusing them across layers, and is trained and evaluated on LibriTTS.

  • E. TTS Training with LACE Tokens: After union alignment, all quantization layers share boundaries and durations, so TTS predicts duration only for the first layer and reuses it for subsequent layers.This avoids separate duration prediction for every quantization layer.
  • E. TTS Training with LACE Tokens: The autoregressive decoder-only Transformer predicts segment-level codes in delay-pattern format and jointly models code and duration outputs.Duration prediction uses a dedicated head and learned duration embeddings.
  • E. TTS Training with LACE Tokens: Training uses cross-entropy for code prediction and focal loss for duration prediction, with loss weights of 1 and 3, respectively.Focal loss addresses duration-class imbalance.
  • E. TTS Training with LACE Tokens: Experiments use LibriTTS at 24 kHz, with test-clean evaluation and same-speaker reference utterances for TTS speaker conditioning.The same dataset supports both reconstruction and TTS experiments.
  • E. TTS Training with LACE Tokens: LACE codecs are fine-tuned end-to-end from pretrained SoundStream, EnCodec, and DAC weights using the original codec objective.Fine-tuning runs for 80k iterations with Adam and a learning rate of 10^-4.

2) TTS Model:

The TTS evaluation uses a Transformer-based model and compares LACE with uncompressed and single-compression DAC baselines on LibriTTS. LACE improves quality over single compression and reduces inference time relative to no compression, while union alignment increases effective frame rate and introduces quality-efficiency tradeoffs.

  • B. Text-to-Speech Task: LACE outperforms single compression on all TTS quality metrics at the same target compression rate γ, but union alignment increases RTF.The TTS experiments use DAC with DP compression, and rate is measured after union alignment.
  • B. Text-to-Speech Task: LACE achieves lower RTF than no compression by reducing the effective frame rate, with a corresponding quality cost partly attributed to lower codec reconstruction quality.The target compression rate γ controls the quality–efficiency tradeoff: increasing γ improves synthesis quality while increasing RTF.
  • B. Text-to-Speech Task: Increasing the anchor layer l* beyond 2 does not improve quality and degrades WER, which the authors associate with repeated code IDs introduced by union alignment.The proposed explanation is that shorter aligned segments bias generation toward repeatedly predicting the same code IDs.

C. Analysis

The analysis examines layer-wise residual changes and validates LACE’s quantization-error advantage over single compression. Earlier layers change more slowly and are more affected by segmentation decisions, supporting the boundary-anchor design.

  • C. Analysis: Consecutive residual embeddings have cosine similarity concentrated near 1 in earlier layers and shifting toward 0 in deeper layers, indicating faster changes at depth.This layer-dependent rate of change motivates independently compressing each quantization layer.
  • C. Analysis: Earlier layers’ greater sensitivity to segmentation decisions motivates choosing their boundaries before deeper layers through boundary anchoring.The analysis uses pretrained DAC with DP compression and examines error across compression rates and layer counts.
  • C. Analysis: LACE consistently yields lower quantization error than single compression, with the gap widening as more quantization layers are used.Single compression approaches a constant error floor, whereas LACE continues reducing error.
  • C. Analysis: At lower compression rates, LACE’s quantization-error advantage becomes more pronounced, while the gap narrows as γ approaches 1.When γ approaches 1, few frames are merged and the single-compression error floor vanishes.
  • C. Analysis: The paper reports that its research ideas, experimental design, implementation, analysis, and results were produced by the authors, with generative AI used only for language editing and code writing.

I. SUPPLEMENTARY MATERIAL

References to sections, equations, and figures in the supplementary material refer to the main paper unless otherwise stated.

  • I. SUPPLEMENTARY MATERIAL: Supplementary references to sections refer to the corresponding sections in the main paper unless stated otherwise.
  • I. SUPPLEMENTARY MATERIAL: Supplementary references to equations refer to the main paper unless stated otherwise.
  • I. SUPPLEMENTARY MATERIAL: Supplementary references to figures refer to the main paper unless stated otherwise.

A. Upper Bound of the Expected Quantization Error

The analysis bounds expected residual error by separating compression and quantization effects across layers. It shows that recomputing segmentation boundaries removes the shared-boundary error floor, while a boundary anchor interpolates between the two regimes.

  • Assumptions: The bound assumes strict compression, centroid-trained quantizers with bounded distortion, and uniformly positive energy captured by compression.These assumptions ensure positive compression error while keeping per-layer contraction factors below one.
  • Per-layer contraction: Each layer’s residual update decomposes into compression error and quantization error, whose squared norms add because the terms are orthogonal.The decomposition combines the compression projection with the quantization output and relies on the orthogonality lemma.
  • Shared segmentation boundary: Shared segmentation boundaries leave a strictly positive error floor equal to the first layer’s compression error.When l∗ = 1, all layers reuse the first layer’s boundaries and the additional contraction term vanishes.
  • Boundary anchor: A boundary anchor between the first and final layers reduces the error floor through contraction in the layer-wise phase.Layers up to l∗ compute new boundaries, while later layers reuse the anchor boundary or compression matrix.
  • Layer-wise compression without anchor (LACE): Recomputing boundaries at every layer makes the error bound converge to zero as the number of quantization layers grows.The LACE setting applies the per-layer contraction at every layer, with λmax < 1.
Loading 2609.17509v1…