Source-linked AI summary
SPARC: Spoken Profile Authorship Recognition Corpus
Efthymios Tsaprazlis, Bowen Yi, Tiantian Feng, Junghwan Kim, David Jurgens, Shrikanth Narayanan
TL;DR
Transcripts retain linguistic information that may link anonymous text to a speaker or reveal personal attributes, a question less studied than acoustic speaker recognition. SPARC benchmarks pairwise speaker and attribute verification from podcast transcripts. Speaker-linked information persists under evaluated controls, while shorter probes offer only partial mitigation.
Problem
Transcripts remove acoustic cues but preserve linguistic traces that may link anonymous text to speakers or reveal sensitive characteristics.
Source: §1 · p. 1Method
SPARC combines two podcast corpora into a benchmark of pairwise speaker and attribute verification, evaluating attribution models and LLMs under episode- and metadata-based controls.
Source: §1 · p. 1, §3.2 · p. 3, §4.2 · p. 4Results
Takeaways & Limitations
Shorter transcripts reduce speaker-verification performance, especially for the strongest models, but do not eliminate leakage.
Source: §1 · p. 2, §6 · p. 8SPARC covers English podcast speech in relatively clean, single-speaker segments, so findings may not generalize to noisy or overlapping speech, other domains, or multilingual and code-switched settings.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Speech anonymization has focused on the acoustic signal, leaving open how much speaker information the transcribed words themselves retain. In this work, we study pairwise speaker and demographic-attribute verification from transcribed speech. We introduce SPARC (Spoken Profile Authorship Recognition Corpus), a benchmark built from two podcast corpora comprising over 12,500 speakers, 1.5M quality-filtered utterances, and speaker-level inferred or source-provided labels for sex, age, and accent, and evaluate five attribution models and three LLMs on identity- and attribute-verification trials with episode- and metadata-based topic proxies. LLM adversaries distinguish same-speaker pairs from text alone. Attribute-label agreement is harder to verify, but LLMs perform sex- and accent-label verification above chance. Finally, releasing only short probe texts (50 words) degrades the strongest adversaries the most, offering a partial reduction in observed linkage.
1 Introduction
Transcripts retain linguistic traces that may link anonymous text to speakers or reveal sensitive attributes, but the strength of these signals against modern LLMs and for demographic-label verification was unknown. SPARC evaluates these risks and finds speaker-linked information persists, while shorter probes only partly mitigate linkage.
1 Introduction
- Transcripts preserve word choice, syntax, discourse markers, and conversational habits that may support speaker linkage or sensitive-attribute inference after acoustic cues are removed.Prior work found attribution in human-transcribed conversations and under ASR errors, with performance declining under conversational topic controls.Source: §1 · p. 1
- SPARC provides pairwise speaker- and attribute-verification trials from over 12,500 speakers and 1.5M quality-filtered utterances, with episode- and metadata-based topic proxies.Speaker-level labels cover sex, age, and accent.Source: §1 · p. 1
- LLMs retain speaker-verification signal under same-episode controls (0.322–0.345 EER) and reach 0.253–0.285 EER cross-episode on both corpora.Embedding-based attackers degrade within-episode, moving from 0.350 to 0.460 EER, near chance.Source: §1 · p. 1
- LLMs detect agreement on inferred sex and accent labels more reliably than embedding models, while age-label agreement remains near chance.The work frames demographic-label verification alongside identity verification as an open question.Source: §1 · p. 1
- At a 1% false-positive rate, two LLMs with nearly identical AUC and EER differ by more than 2× in the fraction of positive trials accepted.This indicates that average-case AUC and EER alone do not capture the models most relevant to privacy.Source: §1 · p. 2
- Shortening exposed probe text degrades the strongest attackers, but only partially mitigates linkage because it attenuates rather than eliminates leakage.The passage also notes that extensive reference material does not remove the probe-text bottleneck.Source: §1 · p. 2
2 Related Work
Prior work studies text-based authorship verification, personal-attribute inference, and speech or text anonymization, but SPARC addresses the gap between these lines by evaluating speaker and attribute leakage in naturally occurring podcast transcripts.
2.1 Text-Based Authorship Attribution and Verification
- Authorship-verification research raises concern that learned representations may encode discussed topics, while benchmarks test topic shortcuts and transcript-based speaker verification under topic control.Earlier conversation studies reported reduced performance after topic control; SPARC adds podcast transcripts and a unified protocol for speaker and attribute-label agreement.Source: §2.1 · p. 2
2.2 Personal Attribute Inference from Text
- Personal-attribute text benchmarks cover social-media writing, leaving attribute leakage in transcribed speech unexplored.Existing resources offer broad scale or wide attribute inventories, but focus on authored social-media text.Source: §2.2 · p. 2
2.3 Speech and Text Anonymization
- Speech anonymization work addresses acoustic signals and text anonymization work studies authorship obfuscation, whereas SPARC evaluates existing transcripts without transforming them.The cited text-anonymization work evaluates privacy–utility tradeoffs.Source: §2.3 · p. 3
3 Dataset
SPARC combines two English podcast sources, filters speech and transcripts for quality, and assigns speaker-level demographic labels using source metadata or model predictions.
3.1 Source Corpora
- NaturalVoices contributes licensed single-speaker segments with source age and sex labels, while SPoRC broadens speaker and episode coverage using its single-speaker subset and automatically inferred demographic labels.NaturalVoices includes utterance-level speaker identity, sex, speech quality, and transcript metadata; SPoRC episodes are restricted to the middle 50% by transcript word count.Source: §3.1 · p. 3
3.2 Spoken Language Data Filtering
- Segments are retained only when SNR is at least 15 dB, music probability is at most 0.1, and two ASR hypotheses have symmetric maximum word error rate at most 0.1.Canary transcripts are used as the final SPARC transcripts.Source: §3.2 · p. 3
3.3 Speaker Attribute Inference
- NaturalVoices age and sex labels are dataset-derived via a pretrained model, SPoRC labels are inferred with Vox-Profile, and both datasets’ accent labels use Voxlect and are grouped as North American versus Other.Source: §3.3 · p. 3
- Figure 1 summarizes the pipeline from eligible single-speaker speech through quality filtering, ASR-agreement screening, and source- or model-derived attribute annotation.The figure caption specifies at least 90% agreement between two ASR models.Source: Figure 1 · p. 4
4 Evaluation
The evaluation tests pairwise identity and attribute-label verification using enrollment text and an unknown probe. Trials vary topic context and difficulty, and models are compared with metrics spanning overall separation and low-false-positive operating points.
4.1 Problem Setup
- The task is pairwise verification: determine whether enrollment and probe utterances share a speaker or an evaluated attribute label, rather than identify a speaker from a population.The adversary receives an enrollment set and a probe with unknown identity or attribute membership.Source: §4.1 · p. 4
4.2 Trial Generation
- Trials vary same- versus cross-episode enrollment and positive versus negative pair difficulty; SPoRC supports only hard positives and easy negatives because retained episodes contain one speaker.Episode membership serves as a topic proxy, while SPoRC trials also match title/description similarity and podcast-category overlap; speaker negatives are additionally matched on demographic groups.Source: §4.2 · pp. 4–5
4.3 Model Scoring
- Five representation-based baselines score probe and enrollment text by cosine similarity, while three prompted LLMs return a 0–100 identity- or attribute-sharing score.The representation baselines are four neural encoders and TF–IDF; the evaluated LLMs are GPT-5-mini, Llama-3.3-70B-Instruct, and Qwen3.5-9B.Source: §4.3 · p. 5
4.4 Evaluation Metrics
- Evaluation reports EER and AUC plus TPR at 1% and 0.1% false-positive rates to assess both overall separability and stringent operating points.EER is the error rate where false acceptance and false rejection coincide; lower is better and chance is 0.5.Source: §4.4 · p. 5
5 Results
LLMs retain identity-verification signal under episode-context controls and detect sex- and accent-label agreement, while age-label verification remains near chance. Shorter probes reduce linkage, and low-FPR results can differ from average-case rankings.
5.1 Identity Verification
- 0.322 EER for Llama-3.3-70B and 0.345 for GPT-5-mini in the same-topic condition contrast with embedding baselines approaching chance.SentenceBERT rises from 0.350 to 0.460 EER; the strongest LLMs are robust to, but not immune to, episode context.Source: §5.1 · p. 5, Figure 2 · p. 6
5.2.1 Sex-Label Verification
- 0.377 and 0.420 EER for GPT-5-mini and Llama-3.3-70B, respectively, in different-topic sex-label verification outperform near-chance embedding models.The LLMs retain an advantage in the same-topic control, though the linguistic cues remain unclear.Source: §5.2.1 · p. 6
5.2.2 Age Group
- Age-group agreement remains near chance across conditions, with no reliable evidence that the tested systems can verify it from transcripts.The best reported EERs are 0.447 in different-topic and 0.440 in same-topic conditions, both from PCM.Source: §5.2.2 · p. 6
5.2.3 Accent/Dialect
- GPT-5-mini reaches 0.268 EER (0.771 AUC) for different-topic accent-label verification, while embedding models remain near chance.The task tests only a binary North American versus non–North-American distinction, not individual dialects.Source: §5.2.3 · pp. 6–7, Table 2 · p. 7
5.3 Sensitivity to Reduced Information
- Reducing enrollment from L=20 to L=5 costs less than shortening the probe: EER increases are +0.041 for GPT-5-mini and +0.024 for SentenceBERT.Enrollment texts are averaged into one reference representation and saturate quickly, whereas the probe is the sole evidence about the unknown speaker.Source: §5.3 · p. 7
- Restricting both probe length and enrollment yields the largest degradation, but the joint effect is sub-additive and governed mainly by the probe bottleneck.At the minimal setting, LLMs retain 0.35–0.36 different-topic EER, below the 0.5 chance rate and embedding baselines.Source: §5.3 · pp. 7–8
- Shortening probes to 50 words raises different-topic EER by 0.065 for GPT-5-mini, 0.047 for Llama-3.3-70B, and 0.038 for SentenceBERT.The LLMs remain below chance and embedding baselines, so shorter probes attenuate but do not eliminate leakage.Source: §5.3 · p. 7, Table 3 · p. 8
5.4 Beyond Average-Case Metrics
- At 1% FPR, GPT-5-mini accepts 0.277 of positive trials versus 0.133 for Llama-3.3-70B despite nearly identical different-topic EER and AUC.SentenceBERT also matches Llama-3.3-70B at 1% FPR (0.143 vs 0.133) despite worse EER, so average-case rankings do not hold at low FPR.Source: Table 4 · p. 8, §5.4 · p. 8
6 Conclusion
SPARC evaluates speaker and attribute verification from transcripts and finds speaker-linked information persists under episode and metadata controls. Attribute agreement is harder to verify, and shorter probes reduce but do not eliminate speaker-verification performance.
6 Conclusion
- SPARC evaluates five attribution models and three LLMs across four verification tasks using speech transcripts.The benchmark covers pairwise speaker and attribute verification.Source: §6 · p. 8
- Speaker-linked information persists under episode and metadata controls, while attribute agreement remains difficult and age-group agreement is near chance.The strongest attribute-label signal is for biological sex and accent; shorter transcripts reduce speaker-verification performance, especially for the strongest models.Source: §6 · p. 8
- Shorter transcripts reduce speaker-verification performance, but do not eliminate it; similar EER and AUC scores can conceal large differences in low-FPR trial-level TPR.The conclusion notes that two LLMs with nearly identical average metrics can differ substantially in the fraction of positive trials accepted at a 1% false-positive rate.Source: §6 · p. 8
- Speech-privacy evaluations should consider linguistic content as well as voice, and SPARC is intended to support transcript-level privacy defenses.Removing acoustic information does not completely remove speaker information from transcripts.Source: §6 · p. 8
Limitations
The findings are bounded by SPARC’s English podcast data, inferred attribute labels, evaluated model–prompt configurations, and imperfect topic controls. The benchmark also does not identify which linguistic cues drive its verification results.
Limitations
- SPARC retains relatively clean, single-speaker English podcast segments, so findings may not generalize to noisy or overlapping speech, other domains, or multilingual and code-switched settings.The corpus uses SNR, music, and dual-ASR-agreement filters.Source: Limitations · p. 9
Attribute Labels and Accent Evaluation
- Sex, age, and accent labels are model-derived rather than independently verified, so attribute results measure agreement with proxy labels.The 50-recording audit measures perceptual consistency on a small selected sample, with weakest agreement for age; the accent task's North American/Other grouping does not support fine-grained or lower-resourced dialect conclusions.Source: Attribute Labels and Accent Evaluation · p. 9
- Results are specific to the three evaluated LLMs and a single zero-shot scoring prompt; one response per trial leaves generation variability unmeasured.Rounded, uncalibrated scores limit ROC resolution at low FPR, and calibrated or likelihood-based scoring could produce different rankings.Source: Attribute Labels and Accent Evaluation · p. 9
- The study does not attribute decisions to lexical, syntactic, discourse, stylistic, or ASR-derived cues, leaving the coarse accent-label result especially unexplained.The authors identify cue analysis as needed to separate speaker habits from residual context.Source: Attribute Labels and Accent Evaluation · p. 9
- Episode membership and metadata provide topic proxies, not semantic-topic control; residual topical overlap and correlated textual cues remain possible.In the original SPoRC speaker protocol, 81.4% of positive trials are within-series while all negatives cross series; the additional cross-series control reduces this imbalance and yields higher EERs.Source: Attribute Labels and Accent Evaluation · p. 9
Ethical Considerations
SPARC is intended for privacy research, but its evaluated capabilities are dual use. Its inferred demographic labels are proxies for trial construction and aggregate analysis, not verified identities or a basis for individual-level decisions.
Ethical Considerations
- SPARC is a dual-use resource built from publicly available podcast recordings, whose speakers may not have anticipated automated linkage or demographic profiling.NaturalVoices episodes retained for the corpus are released under CC-BY 4.0.Source: Ethical Considerations · p. 9
- The authors position SPARC for privacy research and discourage forensic, law-enforcement, employment, eligibility, or other high-stakes uses.Public availability of the recordings does not imply that speakers anticipated automated linkage or demographic profiling.Source: Ethical Considerations · p. 9
- SPoRC demographic labels are model-inferred proxies, not verified identities, and are intended only for trial construction and aggregate analysis.The authors discourage downstream profiling and individual-level decisions and require reuse to preserve inferred provenance and follow applicable licenses and privacy requirements.Source: Ethical Considerations · p. 9
A.1 Verification Trial Construction
This section adds corpus-specific details to the general verification-trial generation procedure described earlier.
A.1 Verification Trial Construction
- The section supplies corpus-specific details that supplement the general trial-generation procedure in Section 4.2.The passage does not specify those details.Source: §A.1 · p. 12
A.1.1 NaturalVoices
NaturalVoices trials cover speaker and demographic verification across four trial types, with trial labels and sampling tailored to the task.
A.1.1 NaturalVoices
- Speaker trials are labeled by whether probe and enrollment texts share a speaker; demographic trials are labeled by agreement on sex, age group, or accent.Eligible partners are randomly sampled with a fixed seed; demographic trials are retained by attribute agreement without artificial class balancing.Source: §A.1.1 · p. 12
- NaturalVoices supports easy positives, hard positives, easy negatives, and hard negatives because episodes may contain multiple speakers.For the speaker ablation, probe and speaker pairs remain fixed when enrollment size L varies at fixed probe length P; only enrollment utterances change.Source: §A.1.1 · p. 12
A.1.2 SPoRC
The appendix details SPoRC trial construction, model scoring, and complete results across input budgets and conditions. Results show that the strongest LLMs retain identity signal in controlled settings, while shorter probes generally reduce performance.
A.1.2 SPoRC
- SPoRC positives pair the same speaker across distinct episodes, while negatives pair different speakers across series and match sex, age-group, and accent labels.Metadata topic proxies filter episode pairs by title-description similarity and stratify negative sampling by TF–IDF similarity and category overlap; trial classes are balanced.Source: §A.1.2 · p. 12
A.2.1 Embedding-based Models
- Embedding baselines are pretrained without task-specific supervised training; SentenceBERT, CISR, and PCM average normalized utterance embeddings, while LUAR uses 16 utterances per side.TF–IDF uses unlabeled unigram and bigram representations; LUAR truncates or pads utterances and limits each to 32 tokens.Source: §A.2.1 · pp. 12–13
A.2.2 LLM Scoring
- All three LLMs score trials zero-shot using a shared prompt structure, with prompts adapted to identity or same-sex, age-group, and accent verification.Responses are reduced to the first numeric value and constrained to 0–100; Qwen settings were not tuned on evaluation labels.Source: §A.2.2 · p. 13
A.3 Full Results
- The full-results appendix covers eight systems, both corpora, trial conditions, and four settings combining probe lengths of 50 or 200 words with enrollment sizes of 5 or 20 utterances.The main paper reports the base configuration for a representative subset; appendix tables and figures extend results across systems and settings.Source: §A.3 · pp. 13–14, §A.3.2 · p. 14, Table 9 · p. 19
A.3.1 SPoRC
- The two strongest LLMs outperform other systems on SPoRC identity verification; their advantage persists for sex and accent agreement, while age agreement stays near chance.Across settings, performance generally degrades with shorter probes, whereas reduced enrollment has a smaller effect.Source: §A.3.1 · p. 14, Figure 3 · p. 15, Figure 4 · p. 16
A.3.2 NaturalVoices
- On NaturalVoices hard trials, identity EERs reach 0.546–0.621 for embedding baselines, 0.542 for GPT-5-mini, and 0.556 for Llama-3.3-70B.The same pattern appears for demographic agreement, where embedding-model hard-condition EERs reach 0.62–0.80.Source: §A.3.2 · p. 14
- Probe shortening generally costs more than reducing enrollment from L = 20 to L = 5 in cross-episode NaturalVoices trials; Qwen3.5-9B has 0.418 EER on cross-episode identity and chance-level attribute performance.Llama-3.3-70B age-label results at 0.546 combined EER improve under reduced information, consistent with response bias rather than a reliable age signal.Source: §A.3.2 · p. 14
A.4 Cross-Series SPoRC Speaker Verification
- 0.402 GPT-5-mini EER and 0.391 Llama-3.3-70B EER remain below chance when both positive and negative SPoRC pairs cross series.At the base configuration, EER rises from 0.253 to 0.402 for GPT-5-mini and from 0.262 to 0.391 for Llama-3.3-70B; semantic-topic and speaker-specific context differences remain possible.Source: §A.4 · p. 14, Table 7 · p. 14