PaperlayerView exact revision on arXiv

Source-linked AI summary

Prototype-Rectified Iterative Self-supervised Manifold Denoising under Severe Acoustic Shift

Ashish Anand Shukla, Rini Smita Thakur, Aryan Das, Vinod K. Kurmi

arXiv:2608.15037v1cs.SDcs.LG

TL;DR

Audio-Text Foundation Models need robust semantic alignment under severe, uncontrolled acoustic shift. PRISM performs training-free geometric correction in embedding space and achieves a +12.94 percentage-point gain over zero-shot on UrbanSound8K.

  • Problem

    Audio-Text Foundation Models require autonomous semantic-alignment restoration under hostile, uncontrolled acoustic environments and severe acoustic shift.

  • Method

    PRISM reverses approximately low-rank affine latent distortion through source-free, training-free geometric corrections using frozen text prototypes.

  • Results

    +12.94 percentage points over the zero-shot baseline on UrbanSound8K, where PRISM establishes a new state of the art across additive-noise benchmarks.

  • Takeaways & Limitations

    Embedding-space geometric correction provides a fast, gradient-free adaptation strategy for Audio-Text Foundation Models under severe acoustic shift.

  • Takeaways & Limitations

    PRISM’s denoising assumptions become unstable for acoustically dense polyphonic classes, where CCVD can erase discriminative semantic structure.

Abstract

from arXiv · show

Audio-Text Foundation Models (ATMs) fail catastrophically under severe acoustic noise, yet existing adaptation strategies either rely on gradient-based Test-Time Adaptation (TTA), which reinforces noise rather than signal, or on prompt tuning that requires privileged noise annotations unavailable at inference. We address these failures with PRISM (Prototype-Rectified Iterative Self-supervised Manifold Denoising), a training-free, source-free TTA framework grounded in the Affine Noise Hypothesis: severe acoustic noise induces a low-rank affine shift in the multimodal latent space, with more than 90% of distortion energy confined to the leading 60 principal components. PRISM estimates and reverses this distortion from an unlabeled target batch using frozen text prototypes as geometric anchors via three closed-form geometric corrections compiled into a single static projection matrix by Affine Bias Regression. At inference, adaptation reduces to one matrix-vector multiplication in 0.0009 ms, making it substantially faster than gradient-based TTA while requiring no additional training. On UrbanSound8K, PRISM improves over the zero-shot baseline by 12.94 percentage points and surpasses an oracle-assisted TTA baseline by 9.41 percentage points, despite never observing its privileged augmented noise prompts. We further identify the Polyphonic Trap, a principled failure mode of subspace deflation for broadband classes, and resolve it via Confidence-Aware Regression (CAR), recovering up to 8.16 percentage points for the worst-affected class.

1 Introduction

PRISM addresses severe acoustic-shift failures in Audio-Text Foundation Models by modeling latent corruption as a low-rank affine distortion and analytically reversing it without gradients, source data, labels, or augmented noise prompts. It uses frozen text prototypes and closed-form corrections for fast, source-free adaptation, while Confidence-Aware Regression mitigates the Polyphonic Trap.

  • Motivation: Severe acoustic environments expose a robustness gap as Audio-Text Foundation Models move from standardized benchmarks into field operations.The introduction cites fiber-optic acoustic monitoring, bio-acoustic animal recognition, and underwater vessel classification as affected applications.
  • PRISM framework: PRISM performs source-free, training-free TTA by estimating target distortion from an unlabeled calibration batch using frozen text prototypes as clean semantic anchors.It applies OPCA, CCVD, and PCT as three closed-form geometric corrections in a self-supervised transductive calibration loop.
  • Inference efficiency: 9 × 10−4 ms is the per-sample inference adaptation time after calibration, using one matrix-vector multiplication with zero trainable parameters.Calibration is transductive, but subsequent inference is strictly batch-independent and orders of magnitude faster than gradient-based TTA.
  • Failure mode: 8.16 pp is the recovery reported for the worst-affected class after Confidence-Aware Regression resolves the Polyphonic Trap.The trap occurs when broadband sound classes geometrically overlap with noise directions during subspace deflation.

2 Related Work

Prior work frames audio-text modeling around a modality gap and addresses severe domain shifts through transductive or test-time adaptation. Existing methods use entropy minimization, Bayesian text priors, or gradient-based prompt tuning, motivating lower-latency alternatives such as backpropagation-free adaptation.

  • Audio-Text Foundation Models and the Modality Gap: CLAP dual encoders project audio and text into a shared latent space, but contrastive training creates a systematic geometric modality gap between their manifolds.The gap arises from disparate statistical densities of the underlying modalities, while clean acoustics preserve relative class geometry.
  • Test-Time Adaptation and Transductive Learning: Transductive and test-time adaptation methods address severe domain shifts using observed unlabeled test data without source data or gradient-heavy fine-tuning.Transductive reasoning targets accurate predictions for a specific unlabeled test set, unlike inductive reasoning.
  • Test-Time Prompt Tuning: Emo-TTA uses Bayesian text priors and empirical class distributions for soft assignments, while TPT and Contrastive Domain Vector Optimization tune textual inputs through gradient descent.These methods generate augmented noisy-audio views and optimize continuous prompt tokens to mirror corrupted audio geometry via self-entropy objectives.
  • Efficient Adaptation: Inference-time backpropagation creates latency bottlenecks for real-time audio, motivating backpropagation-free prompt adaptation such as E-BATS and PRISM’s alternative to explicit logit-level manipulation.The passage contrasts PRISM with explicit logit-level manipulation and prompt adaptation, but the supplied text truncates the final comparison.

3 Theoretical Foundations

The theoretical foundation models severe acoustic corruption as a low-rank affine shift in the joint audio-text embedding space. Under this hypothesis, unsupervised affine inversion can correct the modality gap without finetuning or prompt optimization.

  • Embedding Space: Under severe noise, clean and noisy audio embeddings occupy a joint space with frozen text prototypes, motivating geometric correction of zero-shot classification scores.The clean classifier selects the class whose text embedding has the highest inner product with the audio embedding.
  • Affine Noise Hypothesis: The Affine Noise Hypothesis models noisy embeddings as an affine shift constrained to a low-rank latent subspace despite nonlinear waveform-to-embedding transformation.The hypothesis treats the corrupted embedding e′ as the result of structured latent distortion rather than arbitrary noise.
  • Affine Noise Hypothesis: The affine distortion comprises an orthogonal rotation induced by noise and a translation representing the dense acoustic background centre.The rotation is denoted R, while Δ_noise represents the background’s acoustic centre.
  • Implications: If the hypothesis holds, finetuning and prompt optimization are unnecessary and may move the model away from the optimum; an unsupervised affine inverse can instead correct the modality gap.The hypothesis is preliminarily evaluated through singular-value decay of noise-induced distortion across four acoustically diverse environments.

4 Proposed Approach

PRISM is a source-free, noise-agnostic framework that uses frozen text prototypes as geometric anchors to iteratively correct noisy audio embeddings. It compiles these corrections into a static affine map for batch-independent inference.

  • Framework: PRISM separates batch-dependent transductive calibration from strictly batch-independent inference using frozen text prototypes as a label-free cross-modal reference.The text prototype matrix encodes the class-space geometry in a noise-free modality and anchors alignment at every round.
  • Geometric Corrections: Three closed-form operations correct the noisy manifold: OPCA rotation, CCVD noise-subspace projection, and per-class residual translation.OPCA aligns audio and text manifolds, CCVD removes noise-dominant directions, and translation moves samples toward predicted prototype centroids.
  • Affine Bias Regression: Affine Bias Regression compiles accumulated corrections into a static affine map whose linear component captures rotation and scaling while its bias absorbs translational noise.This affine decomposition distinguishes PRISM from purely linear projection methods.
  • OPCA: Confidence-gated pseudo-labels restrict prototype alignment to reliable samples, supporting an orthogonal Procrustes estimate of the inverse rotation.The confident set retains the top-𝑝 fraction of samples, with default 𝑝=0.8.
  • OPCA Limitation: Additional OPCA iterations cause over-rotation because confident centroids collapse toward text prototypes, degrading accuracy by an average of −1.24 pp across environments.A single Procrustes iteration is therefore used intentionally.

5 Experiments

The experiments evaluate PRISM across benchmark datasets and diverse noise conditions, testing robustness, generalization, and effectiveness under distribution shifts. They also examine the Polyphonic Trap and introduce Confidence-Aware Regression as a targeted mitigation.

  • Experimental scope: PRISM is evaluated across multiple benchmark datasets and noise conditions to assess robustness, generalization, and effectiveness under diverse distribution shifts.The evaluation includes severe acoustic corruption and environment-specific accuracy analyses on US8K and ESC-50.
  • Failure analysis: The experiments explicitly analyze the Polyphonic Trap, identified as PRISM’s primary failure mode, and introduce Confidence-Aware Regression as a targeted mitigation.The mitigation strategy is discussed in detail in Section 6.
  • Benchmark comparisons: Table 1 compares PRISM with state-of-the-art methods across multiple benchmarks, including settings with unavailable oracle noise prompts and device-mismatch assumption violations.ContextDA uses oracle noise-type prompts unavailable for ESC-50/DCASE, while DCASE tests robustness when device mismatch violates the Affine Noise Hypothesis.

5.1 Experimental Setup

The experiments evaluate PRISM across three acoustic-shift benchmarks using standardized LAION-CLAP embeddings, prompt-ensemble text prototypes, and fold structures without ground-truth labels. Comparisons include blind adaptation baselines and the oracle-assisted ContextDA† upper bound.

  • Datasets: Three benchmarks cover synthetic and natural acoustic shifts: US8K, ESC-50, and DCASE/TAU 2019.US8K contains 8,732 clips across 10 classes and yields 87,320 noisy evaluation instances; ESC-50 contains 2,000 clips across 50 classes; DCASE/TAU 2019 contains 14,400 clips across 10 scene categories.
  • Implementation: All experiments use LAION-CLAP checkpoint 630k-audioset-fusion-best.pt on an Intel Xeon Gold 5320 CPU and one NVIDIA A100 80 GB GPU.Text prototypes average embeddings from 20 diverse prompt templates per class and are ℓ2-normalised.
  • Implementation: PRISM uses three iterative rounds of OPCA, CCVD, and ABR with K=60, p=0.8, q=0.7, and λ=0.01.These are the reported experimental settings; the supplied passage truncates the remaining implementation details.
  • Comparison protocol: Comparisons use identical ℓ2-normalized LAION-CLAP embeddings, shared prompt-ensemble text prototypes T, identical fold structures, and no ground-truth labels.Evaluated blind baselines include Zero-Shot, SubTTA, PCA++, and TDA; SubTTA, PCA++, and TDA are adapted from CLIP-based vision-language models to CLAP embeddings.
  • Comparison protocol: ContextDA† receives the background noise type as an inference-time oracle prompt and serves as an upper bound for prompt-guided adaptation.The comparison protocol includes both Fully Blind Transductive and Oracle-Assisted Prompt Guided adaptation methods.

5.2 Results and Discussion

PRISM achieves state-of-the-art performance across additive-noise benchmarks, with especially large gains under severe acoustic distortion and consistent improvements across environments. Diagnostic analyses show that PRISM’s advantage increases as noise dominates, while PCA++ can erase discriminative signal structure at moderate SNR.

  • Benchmark Results: 71.71% accuracy and 72.36% Macro-F1 on US8K yield a +12.94 pp gain over zero-shot under 10 backgrounds × 10-fold CV.PRISM also surpasses PCA++ by +3.83 pp and cache-based TDA by +8.96 pp under the same transductive protocol.
  • Environment Robustness: PRISM’s superiority is consistent across all 10 noise backgrounds on US8K and ESC-50, with ESC-50 zero-shot gains ranging from +3.20 pp to +5.90 pp.PCA++ instead loses 7–16 pp across every ESC-50 environment, averaging −11.25 pp.
  • Severe-Noise Robustness: +24.76 pp over zero-shot at −6 dB: PRISM reaches 57.45%, versus 32.69% for zero-shot CLAP and 37.55% for PCA++, averaged over 4 backgrounds and 10 folds.PRISM also exceeds PCA++ by +19.90 pp in this severe-noise setting.
  • Geometric Failure Modes: At moderate SNR (≥10 dB), PCA++ indiscriminately erases discriminative audio structure, whereas PRISM discards noise directions while preserving signal-bearing structure.At high noise levels (≤6 dB), PCA++’s complement projection beneficially removes dominant noise variance directions.

5.3 Ablation Study

The ablation study identifies OPCA as PRISM’s dominant component, with CCVD and ABR providing additional gains through complementary geometric corrections. PRISM also remains robust across a broad noise-subspace-rank range and maintains batch-independent inference behavior.

  • Component Ablation: +9.58 pp: OPCA is the dominant PRISM component, aligning audio and text prototype manifolds without class labels.It corrects systematic background-noise bias by establishing geometric correspondence between the two prototype manifolds.
  • Component Ablation: +3.09 pp: CCVD contributes an additional gain by exploiting the class-conditional structure of the affine shift.The passage identifies CCVD as a separate incremental component in the strict ablation protocol.
  • Component Ablation: +0.27 pp: ABR supplies a final closed-form ridge-regression correction for residual covariate shift.This correction absorbs residual shift not captured by the preceding subspace operations.
  • Inference Robustness: 80.0% at N=1: PRISM maintains batch-independent performance, while transductive baselines collapse below 65% for N≤64.PRISM maintains 72.2 ± 0.6% across all batch sizes because its projection matrix is calibrated offline.

6 Limitations

PRISM’s CCVD denoising is effective for sparse and impulsive sounds but can erase signal-bearing variance in dense polyphonic classes through geometric subspace overlap. Confidence-Aware Regression mitigates this collapse by preserving more original embedding geometry, recovering street_music performance while introducing regressions on some impulsive classes.

  • Polyphonic Trap: CCVD improves spectrally sparse and impulsive classes but causes a −17.20 pp collapse for street_music.Reported gains include air_conditioner: +42.71 pp, gun_shot: +31.34 pp, and children_playing: +22.30 pp.
  • Polyphonic Trap: The Polyphonic Trap occurs when broad-band harmonics create within-class variance that overlaps CCVD’s targeted low-frequency noise directions, erasing semantic signal.This is characterized as a geometric failure mode rather than an acoustic artifact.
  • Confidence-Aware Regression: CAR uses confidence-gated interpolation to preserve semantically informative variance for acoustically dense polyphonic classes.High-confidence classes receive near-full geometric correction, while low-confidence classes retain more original embedding geometry; CAR is integrated into the transductive calibration loop.
  • Confidence-Aware Regression: CAR recovers street_music from 75.33% to 83.49% while preserving PRISM’s inference-time efficiency.The protection is incorporated directly into the learned affine correction matrix W.
  • Confidence-Aware Regression: CAR trades less aggressive denoising for polyphonic protection, with air_conditioner dropping from 64.77% to 58.35%.The section describes CAR as a regularizer that can cause partial regressions on impulsive classes.

7 Conclusion

PRISM is a gradient-free, source-free, and noise-prompt-free transductive TTA framework that reverses severe acoustic shift as an approximately low-rank affine distortion using a static projection matrix. This design separates batch calibration from sample-wise inference and enables sub-millisecond adaptation without gradients.

  • Conclusion: PRISM reverses approximately low-rank affine distortion under severe acoustic shift using OPCA, CCVD, and ABR compiled into one static projection matrix.The framework is gradient-free, source-free, and noise-prompt-free.
  • Conclusion: PRISM separates batch-dependent calibration from sample-wise, batch-independent inference, enabling sub-millisecond adaptation without gradients.

GenAI Usage Disclosure

Generative AI tools were used only for grammar checking and minor phrasing improvements. The human authors produced the work’s scientific content and substantive intellectual contributions.

  • Generative AI tools assisted with grammar checking and minor phrasing improvements in the manuscript.
  • All scientific claims, experimental designs, results, analyses, figures, and theoretical derivations are entirely the work of the human authors.
  • The human authors produced the methodology and substantive intellectual contributions of this work.
Loading 2608.15037v1…