PaperlayerView exact revision on arXiv

Source-linked AI summary

How Much Audio Is Left In An Embedding? An Inversion Audit Of Audio Encoders

Marios Glytsos, Brian McFee

arXiv:2610.12250v1cs.SDcs.AI

TL;DR

Downstream utility does not reveal how much source detail remains recoverable from pretrained audio embeddings, a question that matters when derived representations are shared instead of recordings. The paper reconstructs paired music from frozen encoder representations using a shared diffusion probe, finding differing reconstructability across encoder families and improved recovery when finer temporal or spectral structure is exposed.

  • Problem

    Downstream utility does not show how much source detail remains reconstructable from pretrained audio embeddings, despite their reuse for tasks unknown during training.

    Source: §1 · p. 1

  • Method

    The paper trains a shared-protocol conditional latent-diffusion probe to reconstruct held-out paired music from frozen audio representations.

    Source: §3.1 · p. 2

  • Results

    Reconstructability orders VGGish and CLAP below ConvNeXt and EnCodec, while finer CLAP temporal resolution and preserved ConvNeXt frequency structure improve recovery.

    Source: §4 · p. 4

  • Takeaways & Limitations

    Even compressed VGGish and CLAP embeddings support reconstructions with measurable source specificity and musical content, though exact pitch or timing may be missed.

    Source: §4 · p. 4

    Validation loss is an optimization diagnostic, not a measure of reconstruction quality, and failed reconstructions may also reflect decoder limitations.

    Source: §1 · p. 1, §3.4 · p. 3

Abstract

from arXiv · show

Pretrained audio encoders are reused for downstream tasks that are often unknown when the encoder is trained, so their usefulness depends partly on which signal properties survive the pretext objective. We study this retained information through paired source reconstruction. Using a shared Stable Audio Open latent diffusion decoder, we reconstruct five-second, 44.1-kHz stereo music from frozen representations produced by supervised classifiers (VGGish, ConvNeXt), an audio-text contrastive model (CLAP), and a waveform-reconstruction model (EnCodec). These objectives impose different pressures to preserve source detail, while their exposed interfaces vary substantially in temporal and spectral resolution. Evaluating on the Million Song Dataset (MSD), we find clear differences in reconstructability across encoder families, while within encoder comparisons show improved recovery when finer temporal or spectral structure is exposed. Even compressed task oriented embeddings support reconstructions that preserve measurable source specificity and high level musical content.

1. INTRODUCTION

The paper audits which source-specific information survives in frozen audio embeddings by reconstructing paired recordings. It compares encoder families under a shared generative probe, while recognizing that reconstruction failure may reflect decoder limits.

1. INTRODUCTION

  • Paired source reconstruction tests which source-specific information is accessible from a frozen embedding, although failures may also reflect decoder limitations.Because the encoder need not be injective, the paper uses a conditional generative decoder that can draw on its learned audio prior for ambiguous details.Source: §1 · p. 1
  • A standardized pretrained-generator probe compares representations without claiming maximal recoverability; encoder-adaptive inversion may recover more but confounds comparisons.The generator provides an expressive prior for probing what source information an embedding can support.Source: §1 · p. 1
  • The comparison spans supervised classifiers VGGish and ConvNeXt, audio-text contrastive CLAP, and reconstruction-trained EnCodec, whose objectives encourage different source detail.Classification rewards discriminative information, contrastive learning emphasizes language-aligned semantics, and reconstruction requires local signal detail.Source: §1 · p. 1
  • Music is a demanding test because reconstruction must preserve pitch, timbre, rhythm, and temporal structure, and a practical setting for representation sharing under licensing constraints.Instrumental music also lies within the pretrained decoder’s domain, reducing domain shift.Source: §1 · p. 1
  • The paper adapts a pretrained latent-diffusion generator for paired reconstruction and compares classification-, contrastive-, and reconstruction-trained encoders, motivating caution when distributing embeddings from sensitive recordings.Source: §1 · p. 1

2. RELATED WORK

Prior work uses inversion to inspect retained information, recover audio, or study representation changes. This paper distinguishes paired source reconstruction across encoder families from synthesis compatibility, privacy attacks, and controlled transformation tests.

2. RELATED WORK

  • Vision inversion has optimized inputs to match representations or learned decoders to recover image structure, while related work frames inversion as a privacy attack.Source: §2 · p. 2
  • Audio work reconstructs speech or resynthesizes content, prosody, and speaker identity, while other attacks recover audio from activations or shared gradients.Source: §2 · p. 2
  • AudioLDM, AudioLDM 2, and Diff-A-Riff use representations to guide compatible synthesis, but do not require recovery of the particular conditioned recording.Source: §2 · p. 2
  • Controlled-transformation studies examine which changes affect embedding geometry, whereas reconstruction tests what source-specific information remains jointly recoverable.The paper applies this reconstruction view across independently trained representations using a standardized generative probe.Source: §2 · p. 2

3. EMBEDDING RECONSTRUCTION AUDIT

The audit trains a separate conditional generator for each frozen encoder interface and evaluates held-out paired-source reconstructions. It standardizes the generator and training setup while varying the exposed representation and its temporal or spectral structure.

3.1. Problem formulation

  • A separate conditional generator maps each frozen encoder representation to paired audio; all conditions share initialization, architecture, target VAE, and training protocol, varying only the conditioning interface.Source: §3.1 · p. 2

3.2. Exposed interfaces

  • For five-second targets, interfaces differ in temporal and spectral resolution; Table 1 reports their temporal granularity and conditioning shape, while alternative CLAP and ConvNeXt interfaces isolate compression effects.Table 1 describes shapes as feature dimension × number of conditioning tokens.Source: §3.2 · p. 2, Table 1 · p. 2

3.3. Conditional latent diffusion probe

  • The probe replaces Stable Audio Open’s T5 conditioning with projected frozen audio tokens and uses a near-zero projection initialization for more stable optimization and convergence.For dense interfaces, temporally local cross-attention biases and masks queries toward conditioning tokens from corresponding audio regions; the qualitative figure shows standard-interface reconstructions of an MTG-Jamendo track.Source: §3.3 · pp. 2–3, Figure 1 · p. 3

3.4. Training objective

  • Training uses Stable Audio Open’s v-prediction objective, but validation loss is an optimization diagnostic rather than a measure of reconstruction quality.The setup uses zero conditioning dropout and guidance weight 1, with deterministic 150-step DDIM sampling to reduce variability from sampling noise.Source: §3.4 · p. 3

3.5. Dataset

  • The dataset is an artist-disjoint Million Song Dataset split across 15 genres; vocals are removed because the pretrained generator was trained on royalty-free instrumental audio and sound effects.The split contains 219,810 training, 26,190 validation, and 26,367 test tracks; finetuning takes approximately 36 hours per encoder interface across four GPUs.Source: §3.5 · p. 3

3.6. Evaluation

  • Evaluation compares acoustic reconstruction, paired-source retrieval, and high-level musical content; Table 2 reports these measures on five-second MSD test crops.The measures include MR-STFT distance, similarity in the fixed Stable Audio Open VAE space calibrated against random source pairs, retrieval against held-out sources, and MusicNN top-10 tag overlap.Source: §3.6 · pp. 3–4, Table 2 · p. 4

4. RESULTS AND DISCUSSION

Reconstructability varies across encoder families, with finer temporal or spectral structure improving recovery in within-encoder comparisons. The complementary evaluations show that source specificity, acoustic similarity, and musical content capture different aspects of reconstruction.

4. RESULTS AND DISCUSSION

  • Reconstructability orders from VGGish and CLAP through ConvNeXt to EnCodec; finer CLAP temporal resolution and preserved ConvNeXt frequency structure each improve reconstruction.The across-family ordering is consistent with objectives that encourage different amounts of source detail, though architectures and training data also differ.Source: §4 · p. 4
  • The common probe approaches native EnCodec in MR-STFT and paired-source retrieval, but its reconstructions remain farther and more variable in SAO-VAE space.Listening examples are recognizably close to the source; across models, MR-STFT and SAO-VAE similarity are only weakly correlated (|r| = 0.02–0.17, R2 < 0.03).Source: §4 · p. 4
  • Musical content and source specificity are not equivalent: compressed VGGish and CLAP reconstructions can retain timbral texture and instrument mixtures while missing exact pitch or timing.MusicNN tag agreement is relatively strong, while paired-source retrieval improves more sharply as finer temporal and spectral structure is exposed; Fig. 2 shows most source–reconstruction similarities above the random source–source distribution.Source: §4 · p. 4, Figure 2 · p. 4
  • CLAP 5s retains frequency-specific temporal patterns, VGGish preserves coarse rhythm and overall spectral character, and pooled ConvNeXt tracks temporal patterns and harmonic structure while losing finer spectral detail.EnCodec most closely reproduces the source’s time–frequency structure.Source: §4 · p. 4
  • Conditioning ablations show clearer validation denoising-loss gains from near-zero projection initialization and temporally local attention than from final reconstruction metrics, especially for denser interfaces.Learning-rate choice also affects reconstruction variability.Source: §4 · p. 4

5. CONCLUSION

The conclusion frames downstream utility as distinct from how much source detail remains reconstructable. It presents a controlled audit of held-out source recovery and identifies broader coverage and more direct privacy measures as future work.

5. CONCLUSION

  • Downstream utility does not reveal how much source detail remains reconstructable from an audio embedding.Source: §5 · p. 4
  • The paper frames inversion as a controlled audit comparing held-out source recovery from supervised, contrastive, and reconstruction-trained representations with a shared pretrained diffusion probe.Source: §5 · p. 4
  • Future work should extend the analysis beyond music, cover broader encoder families, and measure privacy leakage more directly.Source: §5 · p. 4
Loading 2610.12250v1…