Source-linked AI summary
Frame-Wise Streaming for Efficient Learned Speech Compression
Yuhan Liu, Jun Xu, Fengxi Zhang, Zhengxue Cheng, Li Song
TL;DR
Streaming neural codecs face a trade-off between causal operation and efficient entropy-based coding, which often relies on full-sequence context. StreamLSC combines a causal long-context backbone with a frame-level streamable entropy model. It reports state-of-the-art rate-distortion performance, subjective gains, and objective improvements over a non-streamable comparator.
Problem
Streaming neural codecs often use vector quantization with limited RD performance, while entropy-based models offer coding efficiency but rely on future context.
Source: §II · p. 2Method
StreamLSC combines a frame-level causal backbone, including Boundary-Cached Causal Convolution, Intra-Frame LayerNorm, and RNN-mode RWKV, with a streamable entropy model.
Source: §I · pp. 1–2, §III · p. 2Results
−51.37% (ViSQOL) and −53.99% (PESQ) BD-RATE reductions versus FunCodec accompany objective gains of 0.2154 BD-ViSQOL and 0.5156 BD-PESQ.
Source: §IV · p. 4Takeaways & Limitations
StreamLSC achieves frame-level streamability with RD performance comparable to or better than offline models across the reported operating points.
Source: §IV · p. 5
Abstract
from arXiv · showhide
Speech compression is increasingly important in scenarios such as real-time and satellite communication. Neural codecs have shown superior Rate-Distortion (RD) performance over traditional signal-processing codecs. Models with long-range dependency encoders/decoders and entropy models outperform convolution-based structures, but they still lack streaming support since they rely on sequence-level correlations. In this paper, we propose StreamLSC, a frame-level streaming learned speech compression model built on three fundamental components of streaming networks, i.e. Boundary-Cached Causal Convolution, Intra-Frame Layer Normalization, and RNN-mode RWKV. Our model incorporates a streamable entropy quantizer module, ensuring efficient context model while preserving causal constraints. Experiments show that the streaming model achieves performance comparable to its offline counterpart, attaining state-of-the-art rate-distortion results, significant gains in subjective evaluations, -51.37% (ViSQOL) and -53.99% (PESQ) BD-RATE savings on average, and improvements of 0.2154 and 0.5156 in BD-ViSQOL and BD-PESQ.
I. INTRODUCTION
Traditional codecs can lose perceptual quality at ultra-low bitrates, while streaming neural codecs face a trade-off between strict frame-level causality and efficient rate-distortion performance. StreamLSC combines a causal backbone with a frame-level streamable entropy model to address this gap.
I. INTRODUCTION
- Traditional codecs such as Opus and EVS offer high efficiency at medium bitrates, but handcrafted features limit perceptual quality at ultra-low bitrates.Source: §I · p. 1
- StreamLSC’s MUSHRA evaluation reports superior perceptual quality at ultra-low bitrates.The introduction identifies the simultaneous pursuit of efficient rate-distortion performance and strict frame-level causality as a challenge for existing designs.Source: Figure 1 · p. 1, §I · p. 1
- Streaming neural codecs face a trade-off: vector quantization supports causality but often has limited RD performance, while entropy-based models rely on full-sequence context.Source: §I · p. 1
- StreamLSC combines a causal long-context backbone with a frame-level streamable entropy model for speech compression under strict streaming constraints.Source: §I · p. 1
- The proposed backbone reformulates convolution, normalization, and sequence modeling for strict causality and streaming/offline inference equivalence; its entropy model supports causal bit allocation.Source: §I · pp. 1–2
- Experiments report state-of-the-art RD performance and perceptual gains for StreamLSC at ultra-low bitrates.Source: §I · p. 2
II. RELATED WORK
Prior neural speech codecs explore convolutional, spectral, attention-based, and hierarchical designs, while streaming approaches must balance causality with temporal context. Entropy-based coding offers efficiency but commonly depends on full-sequence context, motivating StreamLSC’s frame-level causal design.
II. RELATED WORK
- Prior neural speech codecs span convolutional, spectral, Transformer-based, and hierarchical temporal designs to improve latency, long-term consistency, or perceptual naturalness.Source: §II · p. 2
- Causal convolutions and sliding-window attention support real-time use but can have limited temporal context or offline-training/streaming-inference mismatch.Source: §II · p. 2
- Streaming neural codecs commonly use causality-preserving vector quantization, while entropy-based models offer higher coding efficiency but rely on future context.Source: §II · p. 2
- StreamLSC addresses the gap by integrating a causal long-context backbone with a frame-level streamable entropy model.Source: §II · p. 2
III. METHOD
StreamLSC processes audio frame by frame, carrying state across frames to preserve streaming operation. Its backbone combines causal convolution, frame-local normalization, and RWKV state propagation with a streamable entropy model.
III. METHOD
- The streamable workflow divides audio into non-overlapping frames, encodes each into a partial bitstream, and propagates state for frame-to-frame consistency.This permits real-time operation without waiting for the full sequence.Source: §III · p. 2, Figure 2 · p. 3
- The FS-CRM backbone reformulates normalization, sequence modeling, and convolution for state-based streaming, while the entropy model predicts Gaussian parameters from auxiliary features and codes y autoregressively by channel group.The design supports offline training and streaming deployment without parameter changes.Source: §III · p. 2
- Intra-Frame LayerNorm normalizes independently within each frame across feature and temporal dimensions, avoiding sequence-wide statistics that introduce non-causal dependencies.It requires no state from previous frames and avoids inter-frame caching latency or complexity.Source: §III · pp. 2–3
- RWKV updates a state from the current input and prior state, using time-mix and channel-mix factorization to reduce complexity from quadratic to linear while maintaining long dependencies.GPT-mode supports parallel training, while numerically equivalent RNN-mode enables recursive causal inference.Source: §III · p. 3
- Boundary-Cached Causal Convolution carries the previous frame’s last K −1 time units into the current frame to preserve receptive field context at boundaries.Reflect padding initializes the first frame; later frames use cached units, maintaining strict causality and numerical equivalence with offline computation.Source: §III · p. 3
IV. EXPERIMENT
Experiments use LibriTTS and compare StreamLSC with traditional and neural codecs using objective and subjective metrics. Ablations examine frame length and GAN training, alongside runtime measurements.
IV-A–IV-B. Setup and Evaluation
- Experiments use LibriTTS official splits, compare traditional and neural codec baselines, and assess objective quality plus MUSHRA listening tests with 15 raters.The streaming input uses 80 ms initial frames and prepends the previous frame’s last 350 samples to subsequent frames for STFT consistency.Source: §IV · p. 4
IV-C. Objective Evaluation
- −51.37% (ViSQOL) and −53.99% (PESQ) BD-RATE reductions versus FunCodec accompany BD-ViSQOL gains of +0.2154 and BD-PESQ gains of +0.5156.The codec consistently dominates FunCodec across operating points, with performance comparable or superior to offline models despite frame-level streaming.Source: §IV · pp. 4–5
IV-D. Subjective Evaluation
- A 92.8 average MUSHRA score at 1.1 kbps exceeds all baselines, including FunCodec at 2 kbps and EnCodec at 6 and 12 kbps.The median is 100, with an interquartile range of [89, 100].Source: §IV · pp. 4–5
IV-E. Ablation Study
- Comparable performance across tested frame lengths includes high reconstruction quality with the shortest 80 ms frames.Table I reports the effect of frame length on ViSQOL quality.Source: §IV · p. 5, Table I · p. 5
- 3377.29 bps versus 3529 bps, comparable PESQ scores of 3.178 versus 3.180, and higher ViSQOL scores of 4.54 versus 4.45 favor two-stage GAN training.The two-stage strategy disables the discriminator initially and introduces it for the final 10 epochs.Source: §IV · pp. 4–5, Table II · p. 5
IV-F. Runtime Analysis
- 149.2M parameters and an RTF of 0.0539 on an RTX 4090 indicate real-time operation.RTF is inference time divided by audio duration.Source: §IV · p. 5, Table III · p. 5
V. CONCLUSION
The conclusion presents StreamLSC as a frame-level streaming codec combining a causal backbone and entropy model. It reports state-of-the-art rate-distortion performance and gains in objective and subjective evaluations.
V. CONCLUSION
- Boundary-Cached Causal Convolution, Intra-frame LayerNorm, and RNN-mode RWKV form the streamable backbone and entropy model for learned speech compression.The components capture local and non-local correlations while satisfying causal constraints.Source: §V · p. 5
- StreamLSC achieves state-of-the-art rate-distortion performance with significant subjective-evaluation gains.Source: §V · p. 5
- −51.37% (ViSQOL) and −53.99% (PESQ) BD-RATE savings accompany BD-ViSQOL and BD-PESQ improvements of 0.2154 and 0.5156.Source: §V · p. 5