Source-linked AI summary
HARP-Net: Hyper-Autoencoded Reconstruction Propagation for Scalable Neural Audio Coding
Darius Petermann, Seungkwon Beack, Minje Kim
TL;DR
Neural audio codecs must preserve perceptual quality despite quantization at the bottleneck, which hinders information flow between encoder and decoder. HARP-Net adds compressed skip autoencoders between corresponding mirrored layers, and experiments report better perceptual quality than basic autoencoder codecs.
Problem
Quantization at the bottleneck hinders information flow between encoder and decoder, while direct U-Net shortcuts require substantial bitrate.
Source: §1 · p. 1, §2.3 · p. 3Method
Results
HARP-Net systems outperform their baseline counterparts, with the 40kbps system performing within the same score span as the 64kbps baseline.
Source: §4.3 · p. 4Takeaways & Limitations
Additional skip autoencoders improve reconstruction while avoiding the bitrate cost of identity shortcuts, and listening tests favor HARP-Net over basic AE-based codecs.
Source: §3 · p. 3, §5 · p. 4The proposed coupling between corresponding encoder and decoder layers is a postulated assumption rather than an established relationship.
Source: §2.1 · p. 2
Abstract
from arXiv · showhide
An autoencoder-based codec employs quantization to turn its bottleneck layer activation into bitstrings, a process that hinders information flow between the encoder and decoder parts. To circumvent this issue, we employ additional skip connections between the corresponding pair of encoder-decoder layers. The assumption is that, in a mirrored autoencoder topology, a decoder layer reconstructs the intermediate feature representation of its corresponding encoder layer. Hence, any additional information directly propagated from the corresponding encoder layer helps the reconstruction. We implement this kind of skip connections in the form of additional autoencoders, each of which is a small codec that compresses the massive data transfer between the paired encoder-decoder layers. We empirically verify that the proposed hyper-autoencoded architecture improves perceptual audio quality compared to an ordinary autoencoder baseline.
1. INTRODUCTION
Neural audio coding seeks scalable low-bitrate compression while preserving perceptual quality, but general-purpose audio remains challenging because music tolerates little degradation. HARP-Net addresses this through compressed skip connections that improve reconstruction without the bitrate cost of U-Net shortcuts.
- Music coding requires higher bitrates than speech because subtle perceptual degradation can harm listening experience, making scalability important.Source: §1 · p. 1
- Neural codecs have achieved low speech bitrates, but complex generative decoders and limited general-purpose audio performance constrain their broader use.Source: §1 · p. 1
- Objective losses in autoencoder waveform codecs can create perceptual discrepancies, motivating perceptually weighted losses and architectural improvements.Source: §1 · p. 1
- HARP-Net uses compact skip autoencoders to transmit corresponding encoder feature maps in compressed form, reducing the bitrate burden of identity shortcuts.Source: §1 · pp. 1–2, Figure 1 · p. 2
2. NEURAL AUDIO CODING
Neural audio coding encodes signals through an autoencoder bottleneck that is quantized into a compressed code, but quantization introduces reconstruction error. The hypothetical U-Net improves reconstruction with direct feature shortcuts, although those shortcuts require substantial bitrate.
2.1. Basic Autoencoders
- A basic autoencoder encodes the input into a feature vector, quantizes its bottleneck into a bitstring, and decodes that code into an approximate reconstruction.The decoder reverses quantization before applying successive decoder layers.Source: §2.1 · p. 2
- The encoder and decoder use mirrored architectures, but the basic autoencoder does not assume coupling between corresponding layers; HARP-Net instead postulates that such ties help coding.Source: §2.1 · p. 2
2.2. Quantization and Bitrate Control
- Soft-to-hard quantization makes training differentiable by replacing hard bin assignment with soft probabilities, while inference selects the closest discrete bin.The hardness parameter α is annealed so the soft representation approaches hard quantization.Source: §2.2 · p. 2
- Entropy coding further compresses the discrete code, and an entropy regularizer controls the approximate bitrate during training.Source: §2.2 · pp. 2–3
2.3. The Hypothetical Codec with the U-Net Architecture
- U-Net identity shortcuts improve reconstruction by delivering encoder feature maps directly to corresponding decoder layers, but each shortcut consumes significant bitrate.The hypothetical codec concatenates corresponding encoder and decoder feature maps along the channel dimension.Source: §2.3 · p. 3
3. THE PROPOSED HARP-NET ARCHITECTURE
HARP-Net replaces costly identity shortcuts with compressed skip autoencoders that transmit layer-specific information across a mirrored autoencoder. These additional codes improve reconstruction, support scalable bitrate control, and remain adaptable to mirrored architectures.
3. THE PROPOSED HARP-NET ARCHITECTURE
- Skip autoencoders replace direct identity shortcuts, compressing encoder feature maps into layer-specific codes before decoder-side reconstruction.Each skip AE transforms x^(l), quantizes and codes it as z^(l), then reconstructs an approximation of the feature map with few bits.Source: §3 · p. 3
- The compressed skip paths compensate bottleneck quantization artifacts by supplying less processed encoder information to corresponding decoder layers.The decoder feature map approximates its paired encoder representation, so direct—but compressed—layer information assists reconstruction.Source: §3 · p. 3
- The complete HARP-Net code concatenates the ordinary bottleneck code with layer-wise skip codes from the selected encoder layers.The formulation includes z = [z^(L); z^(L−1); · · · z^(1)].Source: §3 · p. 3
- Additional skip connections improve decoder-side reconstruction quality compared with relying on a single bottleneck code.The architecture is presented as improving reconstruction performance through information paths beyond the bottleneck.Source: §3 · p. 3
- M additional skip autoencoders provide scalable coding, with the number of included layer-wise codes controlling the bitrate.The most abstract bottleneck representation can be supplemented progressively with codes from less abstract layers.Source: §3 · p. 3
- HARP-Net can be applied to any mirrored autoencoding DNN, and its skip autoencoder architecture permits multiple design choices.The method is not restricted to one specific skip-AE structure.Source: §3 · p. 3
4. EXPERIMENTS
Experiments evaluate HARP-Net on 44.1 kHz musical signals using matched-parameter baselines, entropy-controlled training, SNR screening, and MUSHRA listening tests. HARP-Net systems outperform their baseline counterparts perceptually at 40 and 64 kbps.
4.1. The Experimental Setup
- The study uses 6,000 songs across 13 genres, with 5,700 for training, 300 for validation, and 16 held out for testing at 44.1kHz.Models are evaluated at final bitrates of 40kbps and 64kbps after accounting for 16kbps LPC coefficients.Source: §4.1 · p. 3
- HARP-Net skip autoencoders use three hidden layers, 24 filters per layer, and 32 cluster means for each bottleneck quantization task.The main and layer-wise bottlenecks therefore use 5-bit quantization, with tanh limiting encoder-code dynamic range.Source: §4.1 · p. 4
4.2. Training Process
- Training combines squared reconstruction error with an entropy regularizer whose dynamically adjusted weight controls the target bitrate.The regularizer sums estimated entropies across all participating skip-AE layers and the training process adjusts λ until the target entropy is reached.Source: §4.2 · p. 4
4.3. Experimental Results and Discussion
- Average test SNR guides variant selection, with HARP-Net 3 and Baseline 3 chosen for low-bitrate subjective tests and HARP-Net 1 for high bitrate.The authors caution that SNR-based choices are not completely reliable because SNR is an objective measure; baselines also improve little as target bitrate increases.Source: §4.3 · p. 4, Figure 2 · p. 4
- HARP-Net 64kbps and 40kbps both outperform their corresponding baseline codecs in MUSHRA evaluation.The 40kbps HARP-Net falls within the same score span as the 64kbps baseline, while HARP-Net 64kbps lies close to MP3 performance.Source: §4.3 · p. 4
5. CONCLUSIONS
The conclusion presents HARP-Net as a lightweight neural audio codec that connects mirrored encoder-decoder layers through compressed skip autoencoders. Listening tests found that subjects preferred HARP-Net over basic autoencoder codecs.
5. CONCLUSIONS
- HARP-Net is a lightweight mirrored autoencoder whose encoder-decoder connections use additional skip autoencoders.These skip autoencoders act as small codecs for layer-wise information transfer.Source: §5 · p. 4
- The additional skip autoencoders create information paths around lossy bottleneck quantization and improve total autoencoder reconstruction performance.They are designed to avoid consuming too much bitrate.Source: §5 · p. 4
- Listening tests found that subjects preferred HARP-Net to basic autoencoder-based codecs.Source: §5 · p. 4
- Future work targets improving the LPC module’s bitrate and incorporating perceptually motivated losses proposed in.Source: §5 · p. 4