Source-linked AI summary
CLEAR: Online Speech Content Leakage Estimation through Cross-ASR Disagreement
Bhawana Chhaglani, Tanvi Kandepuneni, Jeremy Gummeson, Prashant Shenoy
TL;DR
Fixed privacy settings can leave speech-content leakage varying across utterances and speakers, while transcript-based metrics cannot estimate exposure at runtime. CLEAR estimates leakage through disagreement among heterogeneous ASRs; it tracks transcript-grounded leakage across privacy settings, supporting runtime exposure feedback and adaptive protection.
Problem
Speech-content leakage varies across utterances and speakers, but conventional transcript-based metrics require ground-truth transcripts unavailable during deployment.
Source: §1 · p. 1Method
Results
Spearman ρ = 0.80 across utterance-configuration pairs and ρ = 0.98 across operating-point averages show CLEAR tracks transcript-grounded leakage.
Source: §1 · p. 2Takeaways & Limitations
CLEAR can report potentially exposed words and provide feedback for adjusting privacy aggressiveness while retaining acoustic information for downstream sensing.
Source: §1 · p. 2The threat model considers an adversary recovering linguistic content from transformed audio with modern ASR attackers, rather than concealing speaker identity or other paralinguistic attributes.
Source: §3 · p. 2
Abstract
from arXiv · showhide
Signal-level speech privacy mechanisms suppress linguistic content while preserving acoustic information needed by downstream sensing applications. However, their privacy settings are typically evaluated/selected offline and remain fixed during deployment, even though speech-content leakage can vary substantially across utterances and speakers. Adapting privacy protection at runtime requires estimating how much speech remains recoverable, but conventional measures such as WER or PER require ground-truth transcripts and therefore cannot be computed online. We present CLEAR, a reference-free approach for estimating speech-content leakage at runtime using disagreement among heterogeneous ASR systems. Our key insight is that independently trained ASRs exhibit consistent behavior when linguistic content remains recoverable and increasingly disagree as privacy transformations obscure speech. Using configurable speech-suppression mechanism, we show that cross-ASR disagreement closely tracks transcript-grounded leakage across privacy operating points, achieving a correlation of 0.8. We further characterize the latency-accuracy trade-off of heterogeneous ASR subsets and use their hypotheses to identify potentially exposed words. These capabilities enable privacy to be treated as a runtime property rather than a fixed configuration: CLEAR can communicate residual speech exposure to users and provide feedback for dynamically adjusting privacy aggressiveness while retaining acoustic utility for downstream sensing tasks.
1. INTRODUCTION
Speech leakage varies across utterances and speakers even under a fixed privacy setting, while transcript-based metrics cannot provide a runtime estimate without the original transcript. CLEAR uses disagreement among heterogeneous ASRs to estimate residual leakage and support adaptive privacy.
1. INTRODUCTION
- Privacy mechanisms suppress speech while retaining acoustic information useful for downstream sensing, with adjustable aggressiveness controlling the privacy–utility trade-off.CLEAR can guide more aggressive protection when leakage is high and less aggressive settings when leakage is low, retaining additional acoustic information.Source: §1 · pp. 1–2
- At a fixed Kirigami threshold, adversarial PER varies from 0.45 to 1.0 across utterances, and mean PER differs between speakers, showing that a setting does not ensure uniform privacy.At τ = 0.5, the reported speaker means are 0.61±0.30 and 0.94 ± 0.17.Source: §1 · p. 1
- WER, CER, and PER require a ground-truth transcript, which is unavailable and undesirable to retain during privacy-preserving deployment.Without a reference-free runtime leakage signal, systems cannot determine current exposure, communicate it to users, or decide when stronger protection is needed.Source: §1 · p. 1
- CLEAR estimates leakage from agreement among independently trained ASRs and uses consistently recovered words to communicate what content may remain exposed.It processes privacy-transformed audio without the original audio or its transcript.Source: §1 · p. 1
- CLEAR achieves Spearman ρ = 0.80 across utterance-configuration pairs and ρ = 0.98 across operating-point averages, tracking transcript-grounded leakage across 13 Kirigami privacy settings.It also identifies potentially exposed words with 61.3% correctness and achieves an ensemble latency of RTF = 0.15.Source: §1 · p. 2
2. BACKGROUND AND MOTIVATION
The background distinguishes speech-content privacy from other privacy goals, contrasts machine recoverability with human intelligibility, and describes runtime awareness as a route to adaptive privacy control.
2.1. Speech Content Privacy
- Speech-content privacy limits recovery of what was spoken; transcript-based WER, PER, and MER quantify leakage offline but are unavailable during deployment.Signal-level mechanisms suppress speech while retaining cues useful for sensing.Source: §2.1 · p. 2
2.2. Reference-Free Speech Intelligibility Prediction
- Human-intelligibility methods estimate perceptual intelligibility, whereas CLEAR measures linguistic content recoverability by modern ASR systems as a practical machine adversary.Rather than relying on one recognizer, CLEAR measures disagreement among heterogeneous ASRs.Source: §2.2 · p. 2
2.3. Runtime Privacy Awareness: Opportunities
- Runtime leakage estimates can support user-facing exposure alerts and privacy-control feedback, adjusting aggressiveness while retaining non-speech audio for sensing applications.The passage gives emergency-sound detection and health sensing as examples.Source: §2.3 · p. 2
3. METHOD
CLEAR estimates residual linguistic-content recoverability from heterogeneous ASR disagreement on privacy-transformed audio. It also identifies potentially exposed words and can inform runtime adjustment of privacy aggressiveness.
3. METHOD
- The threat model gives an adversary the transformed audio and focuses CLEAR on residual linguistic-content recoverability, not speaker identity or other paralinguistic attributes.At deployment CLEAR receives only the transformed signal; ground-truth transcripts are available only for offline evaluation.Source: §3 · p. 2
- CLEAR processes privacy-transformed audio with heterogeneous ASRs, using greater hypothesis disagreement as a signal of lower speech-content recoverability.The hypotheses are normalized before pairwise transcript similarity is used to calculate disagreement.Source: §3 · p. 2, Figure 1 · p. 2
- When CLEAR falls below a desired privacy target, the system can increase transformation aggressiveness and recompute the estimate rather than rely on a fixed configuration.The described feedback loop continues until the desired privacy level is reached.Source: §3 · pp. 2–3
- The algorithm returns a CLEAR score, an empty-hypothesis rate, and a set of potentially leaked words from ASR outputs.Words recovered by at least two heterogeneous ASRs are treated as potentially leaked.Source: §3 · pp. 2–3
4. EVALUATION
The evaluation uses heterogeneous ASRs to estimate leakage across privacy transformations, comparing CLEAR with transcript-grounded PER from a held-out HuBERT adversary. CLEAR tracks leakage across Kirigami settings and supports calibration, word-level feedback, and lower-latency ensemble choices.
4.1. Experimental Setup
- The setup evaluates 100 Common Voice utterances across 13 Kirigami thresholds and three low-pass cutoffs, using six heterogeneous ASRs for CLEAR.HuBERT, which had the lowest mean PER (0.936) among candidate adversaries, is held out for the primary transcript-grounded measure.Source: §4.1 · p. 3, §4.2 · p. 3
4.2. Results
- Spearman correlation reaches ρ = 0.80 across 1,300 utterance–configuration pairs and ρ = 0.98 across 13 Kirigami operating-point means.The operating-point comparison averages CLEAR disagreement and held-out HuBERT PER across 100 utterances.Source: Figure 2 · p. 3, §4.2 · p. 4
- Isotonic calibration predicts adversarial PER with MAE = 0.162, versus 0.172 for linear calibration and 0.316 for a mean-PER baseline.Evaluation uses five-fold cross-validation grouped by utterance; classifying whether PER is at least 0.7 achieves 82% accuracy.Source: §4.2 · p. 4
- Wav2Vec2 confidence has only moderate association with held-out HuBERT PER (Spearman ρ = −0.626), whereas CLEAR provides a cross-ASR leakage signal.The confidence comparison is reported on the same evaluation subset.Source: §4.2 · p. 4
- A three-ASR ensemble achieves ρ = 0.807 at 0.30 s sequential latency, while adding a fourth yields ρ = 0.81 at 1.34 s.Latency is reported for an approximately two-second clip; the disagreement computation itself takes less than 0.15 ms even for six ASRs.Source: §4.2 · p. 4, Table 4 · p. 4
- Correctly identified reference words increase from 10.8% at threshold 0.3 to 23.0% at 0.5 and 43.2% at 0.7.At 0.7, CLEAR identifies 32 of 74 spoken words without access to the reference transcript.Source: §4.2 · p. 4, Table 3 · p. 4
- Under low-pass filtering, CLEAR and PER have correlation 0.832 across three cutoff-frequency means, with both means decreasing from 300 Hz to 700 Hz.Mean CLEAR decreases from 0.953 to 0.594, while mean PER decreases from 0.827 to 0.576.Source: §4.2 · p. 4
5. CONCLUSION
CLEAR estimates speech-content leakage without reference transcripts by using cross-ASR disagreement. It tracks leakage across privacy configurations and offers runtime feedback through lightweight ensembles and identification of potentially exposed words.
5. CONCLUSION
- CLEAR is a reference-free approach for estimating speech-content leakage from cross-ASR disagreement.It closely tracks leakage across privacy configurations.Source: §5 · p. 4
- Lightweight ASR ensembles and identification of potentially exposed words provide practical runtime privacy feedback.The conclusion presents these results as a step toward assessing and adapting protection during deployment.Source: §5 · p. 4
- The results point toward speech-privacy systems that assess and adapt protection during deployment.Source: §5 · p. 4