L3AC: Towards a Lightweight and Lossless Audio Codec

1 Introduction

• Neural codecs also produce discrete codes for sound language modeling, connecting audio compression with generative modeling.The paper frames this token functionality as useful for modern audio processing tasks.Source: §1 · p. 1
• Existing multi-quantizer systems can improve reconstruction fidelity, but hierarchical tokens complicate downstream use and more quantizers can make token prediction inconsistent.Single-quantizer alternatives have shown only modest reconstruction gains over multi-quantizer codecs and often use resource-intensive architectures.Source: §1 · p. 1
• L3AC uses one quantizer to avoid hierarchical tokens, alongside lightweight convolutional networks, local Transformers, and TConv.TConv is designed to capture acoustic variations across short and long timescales while maintaining low computational overhead.Source: §1 · pp. 1–2
• Across diverse datasets, L3AC outperforms state-of-the-art methods in audio quality and token-generation quality.The paper also describes its evaluations as validating robustness, adaptability, and efficiency.Source: §1 · p. 2
• An order-of-magnitude reduction in parameter count and computational cost is reported while maintaining high performance.The design also introduces TConv to capture short- and long-term acoustic variations for higher-fidelity reconstruction.Source: §1 · p. 2

2.1 Neural Audio Codecs

• Neural codecs commonly use an encoder-decoder pipeline that quantizes compact latent features into discrete representations for reconstruction.SoundStream introduced a fully convolutional encoder-decoder with residual vector quantization and demonstrated performance across speech and music.Source: §2.1 · p. 2

2.2 High-Fidelity Multi-Quantizer Audio Codecs

• Multi-quantizer codecs such as EnCodec and DAC improve reconstruction, but their computational demands and hierarchical tokens complicate downstream integration.DAC combines Snake activation, improved residual vector quantization, a larger network, and adversarial and reconstruction losses.Source: §2.2 · p. 2

2.3 Single-Quantizer Audio Codecs

• Single-quantizer codecs simplify token streams, but speech-focused models may not generalize out of domain, while general-purpose models can sacrifice fidelity and remain computationally intensive.The passage cites PESQ and STOI as objective metrics where general-purpose approaches may fall short.Source: §2.3 · p. 2
• L3AC targets a single-quantizer codec that combines objective fidelity, cross-domain generalizability, and computational efficiency.It is presented as a lightweight, streamable architecture for high-quality reconstruction across varied audio types.Source: §2.3 · p. 2

3.1 Efficient Receptive Field Expansion

• L3AC combines convolutional layers with local Transformers to achieve comparable receptive fields with fewer layers, lower latency, and reduced complexity while supporting streaming.The design addresses the overhead of deep convolutional stacks and the limited parallelization or computational demands of RNN- and Transformer-based alternatives.Source: §3.1 · pp. 2–3
• TPooling applies max pooling followed by average pooling to absolute signal values, smoothing short-term fluctuations while preserving longer-term patterns.It is proposed to capture global amplitude variations and long-term signal dynamics.Source: §3.1 · p. 3

Encoder.

• The encoder’s TConv Unit pools at varying kernel sizes, convolves at multiple resolutions, and combines the resulting features before subsequent processing.Conv Units adapt ConvNeXt for time-domain audio with 1D convolutions and Snake activation; strided-convolution Down Layers reduce resolution while preserving critical information.Source: Encoder · p. 3

Quantizer.

• A single quantizer discretizes encoder features; L3AC adopts FSQ to support large codebooks without collapse issues and provide faster, more stable quantization.The quantizer uses straightforward scalar quantization in a bounded low-dimensional space; hybrid quantization perturbs encoder features with uniform noise instead of quantizing them with 50% probability.Source: Quantizer · pp. 3–4

Decoder.

• The decoder restores context with a Local Transformer and uses temporal attention in TEConv Units to improve temporal accuracy and perceptual quality.Linear upsampling avoids transposed convolutions that are prone to artifacts, while the deeper Last Unit recovers fine-grained details.Source: Decoder · p. 4

Discriminator.

• A multi-scale discriminator evaluates time- and frequency-domain signals and provides stronger gradient feedback to the generator.It is updated less frequently, such as every 15–25 steps, to prevent it from converging faster than the generator.Source: Discriminator · p. 4

3.3 Training Strategy

• Training combines element-wise, spectrogram, perception, and adversarial losses to target signal reconstruction, multi-scale frequency fidelity, perceptual quality, and realistic generation.Loss-weight clamping limits perception and adversarial losses early, then reverses their relative influence later; One Cycle Learning Rate dynamically adjusts the learning rate.Source: §3.3 · pp. 4–5

Training Datasets.

• L3AC training used speech, music, and general-audio datasets, with samples alternated randomly to give each dataset equal weight and balance learning across domains.All audio was resampled to 16,000 Hz; higher sampling rates increased computational load without empirical performance gains.Source: Training Datasets · p. 5

Training Details.

• Training used AdamW with a one-cycle learning-rate schedule, gradient clipping, and a single NVIDIA RTX 4090 GPU.The learning rate warmed from 5 × 10−5 to 5 × 10−4 before decaying to 5 × 10−6.Source: Training Details · p. 5

Evaluation Details.

• Evaluation covered signal-level metrics across 11 speech, music, and general-audio datasets, application-level tasks, and generated-token ASR and TTS tasks.The audio reconstruction evaluation used Codec-SUPERB; generated-token ASR used LibriSpeech and TTS used LJSpeech.Source: Evaluation Details · pp. 5–6

Reconstructed Audio Quality

• At 0.75 kbps, L3AC achieves ASR WER of 5.01% and AEC accuracy of 65.0%, outperforming competing models; its STOI is 0.82, PESQ is 1.68, and Audio SDR is -2.16.Speech SDR trails TAAE slightly, and Speech/Audio MEL is marginally higher than WavTokenizer.Source: Table 2 · p. 6, Reconstructed Audio Quality · p. 6
• At 3.0 kbps, L3AC’s PESQ of 2.31 exceeds Encodec’s 1.91 and DAC’s 1.92, while application-level results are better across settings except AEC accuracy (80.9% vs. Encodec’s 81.0%).Across bitrates, L3AC consistently matches or exceeds state-of-the-art codecs with lower computational complexity.Source: Reconstructed Audio Quality · pp. 6–7, Table 2 · p. 6

Generated Tokens Quality

• At 1.0 kbps, L3AC’s TTS WER of 27.18% narrowly trails WavTokenizer (26.20%) and beats UniCodec (35.69%); excluding TAAE, it leads in ASR and remains competitive in TTS.TAAE’s TTS WER exceeds 97% on out-of-domain LJSpeech.Source: Generated Tokens Quality · p. 7, Table 3 · p. 7
• Around 1 kbps with a 60 fps frame rate offers an effective compromise between manageable token-sequence length and information density.Lower frame rates shorten sequences but, with fixed codebook size, also lower bitrate and information density.Source: Generated Tokens Quality · p. 7

Ablation study

• Shrinking the Local Transformer window or replacing TConv with standard convolution causes severe performance drops or convergence failures; omitting perceptual loss or loss-weight clamping degrades metrics.The ablations identify receptive field and the loss design as important to performance.Source: Ablation study · p. 7

• L3AC achieves audio quality on par with, and often exceeding, more complex state-of-the-art systems while using a fraction of their computational cost.Source: §5 · p. 7
• Its computational efficiency and simplified training pipeline reduce training burden and facilitate rapid adaptation for specialized audio tasks.Source: §5 · p. 7
• The authors present L3AC as an efficient, high-quality, adaptable neural audio compression option for researchers and developers.Source: §5 · p. 7

A. Loss Functions

• The element-wise loss measures L1 distance between ground-truth and reconstructed audio, penalizing sample-level waveform deviations.Minimizing it encourages the output to remain numerically close to the original.Source: A A. Loss Functions · p. 10
• The spectrogram loss averages magnitude and log-magnitude L1 differences across multiple STFT scales.Smaller windows capture fine temporal dynamics, while larger windows capture frequency information.Source: A A. Loss Functions · p. 10
• The perception loss is the L2 distance between pre-trained ASR features of original and reconstructed audio, emphasizing speech intelligibility.The feature distance guides preservation of characteristics such as phonetic content.Source: A A. Loss Functions · p. 10
• The adversarial loss uses a multi-scale discriminator to encourage naturalness at multiple levels of abstraction.The discriminator setup is described as consistent with DAC.Source: A A. Loss Functions · p. 10

B. Training Details

• At 0.75 kbps, downsampling by 360 yields 44.44 frames/s, and 16.84 bits per frame gives approximately 748 bps.The example uses a 16 kHz input and an FSQ codebook with six levels, each of size seven.Source: B B. Training Details · pp. 10–11
• L3AC achieves SOTA performance despite being trained on fewer datasets than the leading models, with TAAE the stated exception.The relative-STOI comparison uses each method’s best score against the same-bitrate L3AC variant.Source: B B. Training Details · p. 11, Table 5 · p. 11
• Table 6 evaluates ablations on LibriSpeech and Common Voice using perception loss and STOI, where lower perception loss and higher STOI indicate better results.The table caption defines NC as failure to converge.Source: Table 6 · p. 11

C. Ablation Study

• The ablation experiments assess the validity of L3AC’s key design choices.Results are presented in Table 6.Source: C C. Ablation Study · p. 11
• All ablation models are trained on the Speech Domain datasets LibriSpeech and Common Voice.Source: C C. Ablation Study · p. 11
• The ablation results are summarized in Table 6.Source: C C. Ablation Study · p. 11

C.1 Receptive Field Size

• A sufficiently large receptive field is described as important for audio modeling at both raw-signal and higher-context levels.Source: §C.1 · p. 11
• Reducing the local-attention window progressively degrades performance, particularly perceptual metrics, and a window of 75 prevents convergence.The window size determines the acoustic-level receptive field.Source: §C.1 · pp. 11–12
• Replacing TConv with standard convolutions reduces reconstruction quality by restricting the signal-level receptive field.TConv expands that receptive field to capture long-term audio variations.Source: §C.1 · p. 12
• Without TConv, training fails to converge at a local-attention window of 150, which is stable in the full model.The ablation links TConv to model robustness under this setting.Source: §C.1 · p. 12

C.2 Loss Function Design

• Removing the perceptual loss significantly degrades reconstruction audio quality.L3AC combines perceptual loss with element-wise, spectrogram, and adversarial losses to prioritize perceptual quality over minimizing mathematical error.Source: §C.2 · p. 12
• Disabling loss clamping destabilizes training, substantially reduces performance, and can prevent effective data fitting.Clamping loss-component weights prevents any single term from dominating gradient updates.Source: §C.2 · p. 12
• Ablations conclude that L3AC’s architectural components and training procedures are essential for its SOTA lossless audio reconstruction.The conclusion summarizes the reported ablation studies.Source: §C.2 · p. 12