PaperlayerView exact revision on arXiv

Source-linked AI summary

Feature Encoding in VAE-based Audio Decoders: Effects of Input, Depth and Distribution

Louis McCallum, Mick Grierson

arXiv:2610.07966v1cs.SDcs.LG

TL;DR

Neural audio models can synthesize high-quality audio, but how their internal representations encode musical features remains poorly understood. The paper analyzes RAVE decoder activations using layer-wise and cross-layer approaches. It reports different encoding patterns for synthetic and natural stimuli, with stronger joint-encoding results for natural audio under nonlinear probing.

  • Problem

    How musical features are encoded within neural audio models, and where that encoding occurs, remains poorly understood.

    Source: §I · p. 2

  • Method

    The paper analyzes RAVE decoder activations layer by layer and across layers, using feature encoding measures and cluster analysis.

    Source: §I · p. 2, §VII · p. 12

  • Results

    Synthetic stimuli are strongly encoded, while natural audio shows stronger joint encoding with nonlinear probes and joint encoding peaks in middle layers across features.

    Source: A. Robustness to Complexity · p. 11, §VII · p. 12

  • Takeaways & Limitations

    Synthetic and natural stimuli may expose different aspects of the same representation, with natural variation entangled with other sources of variation.

    Source: A. Robustness to Complexity · p. 11

    The correlational analysis establishes associations between activations and musical features, not functional importance.

    Source: §VI · p. 12

Abstract

from arXiv · show

Neural audio synthesis models like the Realtime Audio Variational autoEncoder (RAVE) achieve impressive genera tion quality, yet how their internal representations encode musical features remains poorly understood. We present a systematic layer-wise and cross-layer cluster analysis of RAVE decoder activations across three models trained on different musical domains, tested with four stimulus types. We then evaluate architectural generalization with a general purpose EnCodec model. For RAVE, we find that synthetic stimuli are encoded well across models and audio features (pitch |\r{ho}|=0.45, 5.1x the null, BPM |\r{ho}| = 0.76, 8.6x the null). These results are reduced but still substantively apparent when using natural audio (mean across features |\r{ho}|=0.25, 2.8x the null). Natural audio sees a stronger encoding when nonlinear probes are used (mean across features R2=0.56, 18x the null, +0.152 nonlinear gain over the linear probe R2). Encoding strength varies throughout the layers of the decoder and an increased ability to joint-encode in the middle layers is seen across all audio features (\b{eta}2 all negative, p < 0.05). The general purpose EnCodec decoder also sees similar strong synthetic responses across audio features, similar nonlinear gains for natural audio joint encoding and similar depth profiles. We find the best cross-layer cluster improves the strength (r = 0.65, p = 0.006) and prevalence (r = 0.75, p = 0.001) of BPM encoding when compared against the best whole layers within the same section, with no effect for joint encoding. These findings advance the interpretability of neural audio models and inform targeted control strategies for neural synthesis.

I. INTRODUCTION

Neural audio models can generate high-quality audio, but which musical features their representations encode—and where—remains poorly understood. This paper addresses that gap with layer-wise and cross-layer analysis, drawing on approaches to audio probing and network interpretability.

II. RELATED WORK

  • Prior work examined latent-space constraints and controllability, while layer-wise feature hierarchies were studied mainly in discriminative speech models; generative audio decoders remained an open question.Studies of Wav2Vec, HuBERT, and WavLM reported depth-dependent acoustic feature encoding.Source: §I · p. 2

I. INTRODUCTION

  • Permutation tests establish empirical nulls, while encoding strength and prevalence measure how intensely and broadly neurons track features; nonlinear probes assess joint encoding.The analysis compares results with EnCodec where valid.Source: §I · p. 2
  • The study asks how linear and nonlinear probes compare across synthetic and natural audio, how encoding varies with network depth, and whether cross-layer clusters outperform individual layers.These questions frame the analyses of RAVE decoder activations.Source: §I · p. 2
  • RAVE decoders strongly encode pitch and BPM for synthetic stimuli, with natural-audio encoding also evident across features.The reported values are pitch |ρ| = 0.45, 5.1× the null; BPM |ρ| = 0.76, 8.6× the null; and natural-audio mean |ρ| = 0.25, 2.8× the null.Source: §I · p. 2

II. RELATED WORK

  • The paper combines neural audio synthesis and network interpretability to examine the internal encoding of musical structure and timbral information.It positions its analysis against related work in both areas.Source: §II · p. 3, §I · p. 3

1) Neural audio codecs:

Neural audio codecs learn compact representations for high-quality generation, but internal feature encoding in generative decoders has received limited study. The paper situates its analysis among feature-probing and interpretability approaches, including activation analysis, attribution, and mechanistic interpretability.

1) Neural audio codecs:

  • RAVE uses a variational autoencoder for real-time synthesis, while SoundStream and EnCodec use residual vector quantization; newer generative approaches include autoregressive and diffusion models.Convolutional VAE models remain prominent in real-time musical applications where latency constrains larger alternatives.Source: 1) Neural audio codecs · p. 3

2) Audio feature encoding:

  • Layer-wise feature probing has mainly studied discriminative speech encoders, and trained probes can introduce capacity confounds; this analysis adds rank-correlation and permutation-based prevalence measures.Prior speech-model studies found hierarchical acoustic feature learning across layers.Source: 2) Audio feature encoding · p. 3

B. Neural Network Interpretability and Mechanistic Understanding

  • The paper conducts passive, post-hoc global analysis of decoder neuron populations to identify the features represented across decoder hierarchies.It targets hidden semantics rather than individual predictions or architecture changes.Source: B. Neural Network Interpretability and… · p. 3

1) Activation-based interpretation:

  • Activation-based interpretation examines internal feature alignment, while the paper’s prevalence measure addresses features distributed across overlapping neuron populations.Prior work on visualization and network dissection often assumes alignment with individual neurons or simple linear combinations.Source: 1) Activation-based interpretation · p. 3

2) Attribution methods:

  • Attribution methods identify which inputs drive an output, whereas this study asks which internal representations carry information about musical features.The paper notes that attribution maps poorly to generative models whose outputs are high-dimensional waveforms.Source: 2) Attribution methods · p. 4

3) Circuits and mechanistic interpretability:

  • Mechanistic interpretability has focused mainly on language and vision models; prevalence and cross-layer clustering instead examine localized and distributed encoding in RAVE.The analysis measures where each feature falls on that spectrum.Source: 3) Circuits and mechanistic… · p. 4

4) Neuron clustering:

The paper addresses gaps in cross-layer neuron grouping and layer-wise analysis of generative audio decoders. It probes three RAVE models trained on distinct musical datasets and examines activations across their decoder layers.

4) Neuron clustering:

  • Cross-layer clustering tests whether functionally related neurons spanning layers form more informative groupings than neurons clustered within a single layer.Prior clustering approaches commonly operate within individual layers.Source: 4) Neuron clustering · p. 4

C. Summary of the Gap

  • The analysis addresses three gaps: limited layer-wise study of generative audio, insufficient distinction between encoding strength and prevalence, and little cross-layer clustering.Its framework combines three metrics with cross-layer clustering for RAVE and may be portable to other architectures.Source: C. Summary of the Gap · p. 4

III. METHOD

  • The study probes three RAVE models across four datasets—three natural and one synthetic—for 12 model-dataset combinations, then compares results with pretrained EnCodec.The comparison is intended to assess whether findings hold beyond RAVE’s variational design.Source: §III · pp. 4–5

A. Training Datasets

  • Three RAVE models were trained from scratch on strings, drums, and vocals datasets drawn from distinct categories of musical recordings.The datasets contain 740 strings samples, 759 drums samples, and 486 vocal tracks.Source: A. Training Datasets · p. 5

B. Model

  • RAVE compresses audio into a low-dimensional latent and reconstructs it through upsampling and convolutional stages; the study probes 28 units across 21 named modules.Activations are captured after weight normalization and nonlinearity, before residual addition.Source: B. Model · p. 5

C. Model Inputs

The input analysis uses synthetic and balanced natural-audio samples to examine feature encoding across RAVE models. Four acoustic features are labeled and evaluated, while natural datasets retain some variation in their feature distributions.

1) Sampling:

  • Synthetic pitch tones and tempo pulse trains provide reduced-variation baselines, while balanced subsets of training data test more complex natural audio and in- and out-of-distribution inputs.Sampling aims for balanced feature distributions, diverse source audio, and sufficient sample size; where appropriate, subsets contain 500 samples balanced for BPM or pitch.Source: 1) Sampling · p. 5

2) Features:

  • The analysis measures pitch, tempo (BPM), spectral centroid, and spectral bandwidth, using library metadata or librosa estimates for feature labels.Table I describes feature distributions after per-feature balancing; each included cell uses N=500 samples stratified across the feature range.Source: 2) Features · p. 5, Table I · p. 6
  • Natural-audio feature distributions are well matched overall but can still vary across datasets because of differences in source material.The paper treats this as a constraint accepted in order to work with complex, naturally occurring audio.Source: 2) Features · p. 5

D. Probes

The probes relate decoder activations to acoustic features at the neuron and population levels, using correlations and linear and nonlinear prediction. Evaluation uses cross-validation on balanced audio samples.

D. Probes

  • Activations are captured at each convolutional unit for 500 audio files balanced by feature after audio is encoded and decoded with RAVE.The nonlinear probe is a two-layer ReLU MLP with a linear output head; the linear baseline is ridge regression with α = 1.0.Source: D. Probes · p. 6
  • Spearman rank correlations measure per-neuron activation–feature relationships, while linear and nonlinear probes assess feature recovery from neuron vectors.The correlation approach is more constrained and interpretable; comparing the probes characterizes whether joint encoding is linear or requires nonlinear extraction.Source: D. Probes · p. 6, E. Metrics · p. 7
  • Probe performance is measured by pooled held-out R2 from five-fold cross-validation, with log-transformed targets for pitch and spectral centroid.Tempo and spectral bandwidth targets are used directly; both probe types use the same folds for direct comparison.Source: D. Probes · p. 6

E. Metrics

  • The metrics provide complementary views of feature encoding rather than interchangeable measures of one property.The section introduces per-neuron encoding strength, prevalence above a permutation-derived threshold, and joint-probe R2 as distinct measures.Source: E. Metrics · p. 7

1) Encoding:

The analyses quantify feature encoding at neuron and population levels, compare matched natural and synthetic conditions, and test how encoding changes with depth. Confirmatory and exploratory analyses are distinguished, with multiple-testing correction applied within test families.

1) Encoding:

  • Mean per-neuron |ρ| measures encoding strength, % > p95 measures the population prevalence of neurons above the permutation null, and probe R2 measures joint recovery.The 95th-percentile threshold is derived from empirical null distributions produced by permuting feature labels.Source: 1) Encoding · p. 7

2) Paired Comparisons:

  • Natural-versus-synthetic comparisons use layer- and model-matched Wilcoxon signed-rank tests across 84 pairs per feature.The analysis reports rank-biserial effect sizes and accounts for dependence among pairs through ICC and stratified sign-permutation tests.Source: 2) Paired Comparisons · p. 7

3) Depth:

  • Depth dependence is tested by fitting linear and quadratic regressions over normalized layer depth, then testing curvature coefficients across cells.Each cell contributes 28 layer observations; matched-pairs effect sizes and ICCs are also reported.Source: 3) Depth · p. 7

4) Inferential and Descriptive:

  • Confirmatory tests cover natural-versus-synthetic probes, natural-audio depth dependence, and cluster-versus-layer comparisons; synthetic depth and RAVE-versus-EnCodec analyses are descriptive.False-discovery rate is controlled within related test families, and confidence intervals are 95% bootstrap intervals.Source: 4) Inferential and Descriptive · p. 7

F. Cross-Layer Clustering

The study compares feature encoding across stimuli and decoder depth, using cross-layer clusters and a separate EnCodec evaluation. Synthetic and natural audio show above-chance encoding, with synthetic pitch encoding stronger in matched comparisons.

F. Cross-Layer Clustering

  • Cross-layer clustering groups neurons by activation patterns within early, middle, and late decoder sections, then evaluates cluster encoding against the best layer in each section.The sections comprise layers 0–7, 8–14, and 15+; k-means uses k = 6, and the selected neuron vectors are independent of the probed feature.Source: F. Cross-Layer Clustering · p. 7

G. EnCodec Comparison

  • The EnCodec evaluation uses a pretrained 32 kHz music model with residual vector quantization and measures activations across 13 decoder layers.All four datasets are evaluated; the model’s structural difference prevents per-layer matching with RAVE.Source: G. EnCodec Comparison · p. 7

A. Synthetic Stimuli

  • For synthetic stimuli, pitch and BPM had mean |ρ| values of 0.45 and 0.76, respectively, with 82% and 92% of neurons above chance.A total of 86.2% of neurons encoded both features, and fewer than 1% encoded neither.Source: A. Synthetic Stimuli · p. 8

B. Natural Audio

  • For natural audio, all features had above-chance |ρ| values of 0.20–0.31, and 73.4% of neurons correlated above chance for at least three features.Joint-probe R2 was above chance, with a larger linear-to-nonlinear gain than for synthetic inputs.Source: B. Natural Audio · p. 8
  • Synthetic stimuli were encoded more strongly than natural audio for pitch across all measures in every layer-matched pair (rank-biserial r ≥0.999; all p < 0.001).The direction was unanimous across models, although the magnitude of the synthetic advantage varied between models.Source: B. Natural Audio · pp. 8–9

C. Out-of-Distribution Audio

Feature encoding persists on unfamiliar audio; natural-audio joint encoding peaks in the middle decoder layers, while cross-layer clusters improve BPM encoding on per-neuron measures.

C. Out-of-Distribution Audio

  • Unfamiliar audio remained above chance: per-neuron correlations exceeded the null by 2–4×, with joint R2 of 0.74 for centroid and 0.36 for tempo.Per-neuron encoding was often as strong or stronger out-of-distribution than in-distribution.Source: C. Out-of-Distribution Audio · p. 9

D. Effect of Layer Depth

  • Across features, joint R2 peaked mid-network, while pitch, centroid, and bandwidth per-neuron |ρ| peaked toward network extremes; tempo showed no significant curvature.The joint-encoding direction was unanimous across models and datasets, but curvature magnitude was less certain for high-ICC features; no shared top layer survived multiple-test correction.Source: D. Effect of Layer Depth · p. 9

E. Cross-Layer Clustering

F. EnCodec Comparison

EnCodec reproduces stronger synthetic-than-natural feature encoding, while RAVE’s single-axis synthetic responses are near-linear and its natural-audio activations retain detectable feature correlations amid greater complexity.

F. EnCodec Comparison

  • BPM nonlinear gain was +0.20; EnCodec also showed negative R2 curvature across layers and little clustering improvement in joint encoding.Depth effects were descriptive because the number of cells was low.Source: F. EnCodec Comparison · p. 10

A. Robustness to Complexity

  • Single-axis synthetic changes produce linear activation variation, whereas natural audio retains detectable feature correlations amid timbral, temporal, and polyphonic variation.The paper characterizes synthetic and natural stimuli as revealing different aspects of the representation, with natural feature variation entangled with other sources of variation.Source: A. Robustness to Complexity · pp. 10–11

F. EnCodec Comparison

  • For pitch joint R2, synthetic versus natural was 0.99 vs 0.45; EnCodec showed stronger synthetic encoding across every measure and feature (N=13, p<0.001, r=+1.00).Tempo joint R2 was 0.99 vs 0.59 for synthetic versus natural audio.Source: F. EnCodec Comparison · p. 10, Table IV · p. 11

B. Feature Specialization

RAVE neurons commonly track multiple natural-audio features, with spectral centroid especially aligned; tempo differs across stimulus conditions, and layer aggregation may help recover it.

B. Feature Specialization

  • 58.7% of natural-audio neurons encode both pitch and tempo above the null threshold, while 73% encode at least three of four features.This overlap indicates distributed, entangled feature representations rather than dedicated neurons for individual properties.Source: B. Feature Specialization · p. 11
  • Spectral centroid has the highest per-neuron correlation and encoding across natural-audio cells.The authors interpret this as brightness being the most neuron-aligned acoustic property in the representation.Source: B. Feature Specialization · p. 11
  • Tempo is more strongly encoded than pitch on synthetic stimuli, but the two converge on natural audio, cautioning against using synthetic tempo strength as a proxy for realistic use.Tempo degrades more steeply than pitch between synthetic and natural conditions.Source: B. Feature Specialization · p. 11
  • Natural-audio tempo must be inferred from irregular, expressively timed onsets in polyphonic textures, a noisier estimation problem than reading fixed periodicity.The paper presents this as a possible explanation for differences between the two stimulus formats.Source: B. Feature Specialization · p. 11
  • Tempo alone benefits from cross-layer aggregation, consistent with its being temporally extended and potentially drawing on neurons operating at different timescales.Grouping neurons across layers may recover information no single layer holds in isolation.Source: B. Feature Specialization · p. 11

C. Joint-Layer and Per-Neuron Encoding

  • The authors propose that middle layers distribute feature information, while later layers consolidate it into fewer dominant neurons with stronger per-neuron correlations and reduced joint diversity.Joint recovery (R2) is strongest in the middle layers, whereas per-neuron correlations (|ρ|) are strongest closer to the output.Source: C. Joint-Layer and Per-Neuron Encoding · p. 11

D. Application to Other Architectures

  • RAVE's shallow convolutional decoder enables exhaustive layer analysis, and EnCodec aligns with these convolutional-decoder findings; localization is not assumed for attention, diffusion, or autoregressive models.The authors treat findings in those other architecture families as hypotheses to test, not conclusions to assume.Source: D. Application to Other Architectures · pp. 11–12

VI. LIMITATIONS

The analysis reports broad feature encoding and depth patterns, but its correlations do not establish functional importance, natural-audio effects are modest, and several validation and generalization questions remain.

VI. LIMITATIONS

  • Activation-feature correlations establish association, not functional importance, and a neuron correlated with pitch may instead track a correlated acoustic property.The passage gives an example in which predictions shift with input amplitude alone.Source: §VI · p. 12
  • Natural-audio mean correlations are modest: pitch |ρ|=0.20, tempo |ρ|=0.20, spectral centroid |ρ|=0.31, and spectral bandwidth |ρ|=0.27, or approximately 4–10% explained variance.The drum model is excluded from pitch analysis, although drums contain spectral content.Source: §VI · p. 12

VII. CONCLUSION

  • The study analyzes three RAVE models across four stimulus types with encoding-strength, permutation-validated prevalence, and nonlinear joint-encoding measures, plus a supplementary model for generalization.The supplementary model tests generalization beyond one architecture.Source: §VII · p. 12
  • Synthetic stimuli are strongly encoded throughout decoder layers, while natural and out-of-distribution audio also show encoding; nonlinear joint encoding peaks in middle layers and cross-layer clusters offer a small tempo advantage.Monotonic encoding is strongest in early and late layers for three of four features; the cross-layer advantage is compared with whole layers.Source: §VII · p. 12
  • Future work should use targeted ablations to test functional roles, compare more architectures, and investigate practical uses such as network bending, pruning, and interpretable design.The paper identifies these as directions for further study.Source: §VII · p. 12

A. Permutation Tests

The supplementary materials describe the encoding metrics and comparison procedures, while figure captions specify diagnostic plots and sample-size limits; they do not provide the plotted or tabulated outcomes here.

A. Permutation Tests

  • Table S1 reports per-cell encoding metrics across model, dataset, and feature combinations, distinguishing Spearman per-neuron statistics, ridge linear R2, and MLP joint statistics.Null prevalence thresholds come from 500 shuffled-label permutations; bracketed values are layer-level 95% bootstrap confidence intervals.Source: Table S1 · p. 14

B. Comparison Tests

  • Table S2 specifies layer-matched Wilcoxon signed-rank comparisons of natural versus synthetic encoding for RAVE and pretrained EnCodec, excluding spectral features.It defines medians and IQRs, Hodges–Lehmann differences with 95% confidence intervals, rank-biserial effects, and adjusted p-values.Source: Table S2 · p. 15
  • Table S3 presents descriptive EnCodec depth-curvature statistics; with only 2–3 cells per feature, the across-cell tests cannot reach significance.Its caption reports p ≥0.25 throughout and defines positive β2 as U-shaped and negative β2 as inverted-U depth profiles.Source: Table S3 · p. 15
  • Fig. S1 plots EnCodec layer-wise natural-audio encoding measures and synthetic per-layer mean |ρ|; significance tests are unavailable with N=3 natural and N=1 synthetic.The caption notes that other synthetic measures are near ceiling throughout.Source: Figure S1 · p. 16
  • Table S4 compares the best natural-audio cross-layer cluster with the best single layer within a section for RAVE and EnCodec.The caption defines positive Hodges–Lehmann differences as favoring the cluster and describes effect sizes, selected-unit counts, and dependence measures.Source: Table S4 · p. 17
  • Fig. S2 plots silhouette score, cluster imbalance, and feature correlation across k, while Fig. S3 plots mean cluster size ranked by size across k.The captions identify the diagnostics but state no winning value of k.Source: Figure S2 · p. 18, Figure S3 · p. 18
  • Fig. S4 plots mean ARI for all clusters and mean Jaccard for the best cluster across k.The caption supplies the plotted measures, not a comparative outcome.Source: Figure S4 · p. 18

Related papers

Loading 2610.07966v1…