PaperlayerView exact revision on arXiv

Source-linked AI summary

The Machines Are Calling: Measuring Automated and Synthetic Voices in Unwanted Inbound Calls

Xingyu Shen, Tommy Duong, Muduo Xu, Xiaodong An, Jiaqi Gan, Haoyuan Tang, Jamey Z. Liang, Siyu Zhang, Yan Zhang, Simiao Ren

arXiv:2609.11137v1cs.CRcs.CYcs.SD

TL;DR

The paper addresses the lack of peer-reviewed field measurements separating machine-placed unwanted calls from replayed and synthesized speech. It uses an interactive voice honeypot with separate caller audio, replay fingerprinting, synthetic-speech detection, and blinded validation. At least 26.9% of greeted openings are machine-voiced, while detector estimates and longitudinal patterns remain conditional on measurement and seeding choices.

  • Problem

    Peer-reviewed evidence does not quantify how much unwanted call traffic is machine-placed or distinguish synthesized speech from replayed recordings.

  • Method

    An interactive honeypot records caller audio separately and combines replay fingerprinting, commercial synthetic-speech detection, and blinded listener validation.

  • Results

    At least 26.9% of greeted openings are machine-voiced, comprising 13.8% replayed recordings and 13.1% fresh audio labeled synthetic.

  • Takeaways & Limitations

    Machine-voiced openings are concentrated in lead-generation traffic, while campaigns persist through shared assets as their originating numbers change.

  • Takeaways & Limitations

    The synthetic-speech estimate is detector-conditional, listener-validated on only the flagged side, and dependent on the honeypot’s seeding schedule.

Abstract

from arXiv · show

In February 2024 the U.S. Federal Communications Commission (FCC) placed AI-generated voices under the Telephone Consumer Protection Act (TCPA). Yet no peer-reviewed measurement says how much unwanted call traffic is placed by a machine, or how much of that machine speech is synthesized rather than played from a recording. We report both with a disclosed pipeline. An interactive voice honeypot (language-model personas on real U.S. numbers, the caller recorded on its own track) recorded 10,987 calls over 66 days; 11 days on which our stack answered silently are set aside. Three instruments read each opening: an audio fingerprint that finds the same recording played on other calls, a commercial synthetic-speech detector on the caller's first ten seconds, and blinded listeners who check what it flags. Of the 7,233 calls our persona greeted on normal days, 13.8% open with a recording we also heard on another call, and 13.1% with fresh audio the detector labels synthetic. A further 9.9% open with a caller who never spoke after our greeting, 54.2% with fresh audio the detector labels human, and 9.0% could not be scored. Machine-voiced openings are therefore at least 26.9%, a further tenth of calls are silent connections we read as machine-placed, and replays of a recording make up 45% of the detector's own rate (29.3% of 6,192 scored openings). The same waveform played on two calls lands on opposite sides of the detector's threshold 13.6% of the time, and eleven listeners confirm 54.4% of what it flags. Synthetic openings concentrate in lead-generation spam (33.8%), not fraud (21.1%); 0.44% disclose automation. Prevalence tracks how long a bait number has circulated (59% against 19% in the same weeks): seeding history, not calendar time, explains the trend. Campaigns outlast their numbers: one recorded compliance notice opens calls in six campaigns, and one synthetic voice serves nine.

1 Introduction

The paper measures two unresolved dimensions of unwanted calling: how much traffic is machine-voiced and whether automated speech is synthesized or replayed. Its interactive honeypot and disclosed instruments show substantial machine involvement, concentrated in lead-generation calls and difficult to capture with detector scores alone.

  • Measurement approach: The audio fingerprint separates replayed recordings from freshly generated speech without relying on a synthesis detector.The same waveform appearing across calls identifies replay; fresh rendering does not preserve that waveform.
  • Research questions: 26.9% of greeted calls open with machine-voiced audio: 13.8% replayed recordings and 13.1% fresh audio labeled synthetic.A further 9.9% are silent connections, while 9.0% cannot be scored.
  • Measurement approach: The detector labels 29.3% of 6,192 scored openings synthetic, but 45% of those are replays and listeners confirm 54.4% of flagged calls.The paper reports detector-labeled prevalence alongside a separate listener-confirmation estimate.
  • Traffic distribution: Synthetic-labeled calls are more common in lead-generation sales traffic than fraud, at 33.8% versus 21.1%.They are especially common when callers transfer the recipient to another person on their side.
  • Disclosure: Only 0.44% of calls with synthetic-labeled openings disclose automation, while 99.6% do not.The paper treats this as a baseline for a disclosure duty rather than a compliance rate.
  • Campaign persistence: Calling campaigns persist through shared wording and assets even as their originating numbers change, making number-based defenses track disposable identifiers.The same compliance recording appears in six campaigns and one synthetic voice serves nine.

2 Related Work

Prior robocall and scambaiting studies characterize volume, campaigns, interaction, or laboratory capability, but do not measure voice provenance in live unwanted traffic. This paper positions its contribution as a field measurement that distinguishes replayed recordings from freshly synthesized speech.

  • Telephony abuse measurement: Earlier honeypots measured volume and campaigns but generally lacked interactive caller audio suitable for voice-provenance analysis.Phoneypot answered none of its calls, while other work retained campaign audio without separating replay from fresh synthesis.
  • Telephony abuse measurement: Interactive honeypots capture more analyzable audio, but interactivity alone cannot distinguish live agents, IVR menus, and LLM-driven text-to-speech.Voice provenance is therefore orthogonal to the interactive-versus-static campaign axis.
  • Scambaiting and AI-voice capability: Scambaiting research usually asks whether human scammers detect an automated defender, the reverse direction from measuring automation in inbound calls.The cited study found recognition in about 5% of 200 transcribed calls.
  • Scambaiting and AI-voice capability: Capability and compliance studies show that LLM agents can execute scam scripts or that targets may comply, but they do not report field incidence.RoboKA synthesized an evaluation corpus because no public labeled real-world corpus was available for training.
  • Competing prevalence estimate: The competing Hiya estimate is approximately 25% AI-generated audio, but its unrefereed report discloses no threshold, score distribution, aggregation rule, or engagement model.The paper’s 29.3% detector-labeled rate is presented as comparable in range, with auditability as the contribution.

3 Data and Method

The study combines an interactive, separately recorded voice honeypot with replay fingerprinting, commercial detection, listener validation, and explicit threshold and sampling analyses. These design choices enable provenance measurement while leaving important dependence on seeding history, detector behavior, and nonrandom validation.

  • 3.1 The corpus: Interactive LLM personas answered calls on real U.S. numbers seeded into lead-generation funnels, allowing silent connections and live interactions to remain in the denominator.The persona speaks first and does not reprompt, time out, or hang up, so silence after the greeting is observable.
  • 3.1 The corpus: Staggered number seeding separates exposure age from calendar time but makes pooled prevalence conditional on this seeding schedule.The authors caution that it should not be read as a transportable constant.
  • 3.1 The corpus: The corpus contains 10,987 inbound calls over 66 days, with caller and persona audio recorded on separate channels.The analysis retains calls whether or not they become conversations; 6,192 calls have scorable caller speech for detector analyses.
  • 3.3 Detector, aggregation and threshold: The detector returns per-utterance synthetic-speech scores, which the pipeline aggregates by the maximum score and labels a call synthetic at 0.85.Maximum aggregation acts as a logical-OR rule; the paper also reports the mean as a sensitivity analysis and notes that 0.85 is a conservative operating point.
  • 3.4 Validation protocol and estimator: Listener validation is nonrandom: labeled flagged clips are longer and higher-scoring than unlabeled clips, so 54.4% confirmation slightly overstates census precision.Reweighting to the score mix yields 53.2%, and transfer to all flagged calls remains an assumption.
  • 3.4 Validation protocol and estimator: Identical audio can receive opposite detector decisions, with one repeated opening scored 0.97–0.98 on fourteen calls and 0.14 on another.This demonstrates detector instability beyond cases close to the decision threshold.
  • 3.5 Replay census: Replay detection compares aligned MFCC feature sequences and treats high similarity as evidence that the same recording was played again.The method uses a 0.70 threshold for the same recording and 0.85 for the same waveform; noise or inserted names can only lower similarity, making the replay census a floor.

4 Results

The results decompose greeted calls into silent, replayed, fresh detector-labeled, and unscored openings, while distinguishing the scored corpus from the full recorded traffic.

  • 4 Results: 7,233 greeted calls are decomposed into silent, replayed, fresh detector-labeled, and unscored openings, while 2,711 outage-day calls remain outside the decomposition.The detector scored 6,192 calls; replayed and fresh-synthetic categories are defined separately from the overall AI-labeled total.
  • 4 Results: 29.3% is the detector’s headline synthetic-label rate among 6,192 scored openings, with 1,816 calls labeled synthetic.The rate is unchanged when 318 ungreeted calls are included.
  • 4 Results: The section combines a census of all greeted calls with detector-specific analyses whose denominator is the scored openings.The pipeline reports the full decomposition first, then uses the detector where fresh audio must be classified.

4.1 How much of the traffic is a machine

Among 7,233 greeted calls, at least 26.9% opened with machine-voiced audio, while another 9.9% were silent connections and 9.0% remained unscored. Replay detection and synthetic-speech labeling capture different machine-voiced populations, and detector validation shows increasing but imperfect listener confirmation at higher scores.

  • 4.1 How much of the traffic is a machine: 26.9% of greeted calls opened with machine-voiced audio: 13.8% replayed recordings and 13.1% fresh audio labeled synthetic.A further 9.9% were silent connections, while 54.2% were labeled human and 9.0% could not be scored.
  • 4.1 How much of the traffic is a machine: 45% of detector-flagged openings were replays, although the stricter fingerprint criterion shifts the split to 9.2% playback and 16.7% synthetic.The total machine-voiced estimate changes from 26.9% to 25.9% under the stricter criterion.
  • Call shapes: 19.5% of short exchanges opened with a replayed recording versus 10.7% of one-shot calls, while scored shares differed elevenfold across call shapes.Replays and detector flags concentrate in short exchanges rather than one-shot calls.
  • Coverage caveat: The two-turn corpus filter dropped 1,845 normal-day calls, 63% of them silent and 22% one-shot, so the decomposition uses greeted calls rather than the filtered corpus.Only 2.1% of dropped calls opened with a recording heard on another call, and none carried a detector score.
  • Detector validation: Listener confirmation rose from 21.9% in the 0.5–0.6 band to 57.4% in the 0.9–1.0 band, but only 54.4% of threshold-flagged clips were confirmed.The contrast remained positive within each of eight listeners meeting the minimum per-side judgment requirement.

4.2 Prevalence and threshold stability

The detector labels 29.3% of scored openings synthetic, but replay checks and blinded listeners show that this rate is threshold-sensitive and only partly validated. Identical waveforms can cross the decision boundary, while listener confirmation is about half and listener criteria are unstable.

  • Prevalence: 29.3% of 6,192 scored calls are labeled synthetic at the operating threshold, with a cluster-bootstrap 95% CI of 27.0–31.9%.Calls cluster within originating numbers, so the reported interval accounts for dependence rather than treating calls as independent.
  • Analyst choices: 21.8%–39.6% is the resulting estimate range across the tested threshold and aggregation choices.Maximum aggregation produces 39.6% at a 0.50 cutoff, while mean aggregation produces 21.8% at 0.85.
  • Listener validation: 54.4% of detector-flagged clips are confirmed synthetic by blinded listeners, but reweighting gives 53.2% or 52.8%.The corresponding human-confirmed share is 15.9% under an unsupported transfer assumption, or 15.6% and 15.5% after reweighting.
  • Interpretation: The detector ranks clips by synthetic likelihood but does not establish synthesis as the source of its score separation.Codec, bandwidth and trunk class could each produce a confident bimodal split.
  • Listener validation: A listener’s synthetic judgments wander by 10–20 points, while same-listener first judgments show mixed directions rather than learning.Listeners also disagree weakly with one another, with mean pairwise Cohen’s κ = +0.267.
  • Threshold stability: 13.6% of 2,756 matched-waveform pairs fall on opposite sides of the detector’s 0.85 threshold.The median absolute score difference is 0.008, but the 90th percentile is 0.28, showing substantial tail instability.

4.3 Automation tracks volume, not harm

Machine-voiced calls concentrate in high-volume lead-generation spam rather than fraud, although the class comparison is unadjusted and instrument-dependent. Transfer endings are also strongly associated with detector labels, while exposure age remains a competing explanation for observed differences.

  • Call class: 33.8% of lead-generation spam calls are detector-labeled synthetic, versus 21.1% of calls labeled fraud.The ordering is opposite the expectation that fraud would be the most automated class.
  • Caveats: The class comparison is unadjusted for call length and exposure age, and prerecorded blasts could concentrate in spam independently of TTS.The detector may partly measure non-liveness, while the class labels come from the study’s own classifier.
  • Call endings: 69.9% of calls ending with a caller-side transfer are AI-labeled, versus 26.8% of other calls, with OR = 6.36.Adjusting for duration and turn count raises the odds ratio to 7.29.
  • Call endings: 78.1% of multi-call originating numbers are pure with respect to the voice label, but the ending labeler also receives the caller number.Withholding the number would test whether the ending association exploits caller identity.

4.4 Prevalence tracks a number’s exposure age, not calendar time

Synthetic prevalence rises with a bait number’s exposure age rather than calendar time. The same-period contrast between an older May-seeded number and fresh July numbers disappears when calls are matched by exposure age, though the mechanism remains inferred and the effective sample is small.

  • Exposure age: Exposure age fits prevalence better than calendar date, with AIC 6,888 versus 7,372 and age β = +0.0505 per day.The model containing both terms has AIC 6,889.7, indistinguishable from age alone.
  • Exposure age: 59.4% of calls to the May-seeded number were synthetic-voiced, versus 18.6% for ten fresh numbers over the same days.Matched on each number’s first twenty exposure days, the rates were 18.4% and 18.6%, a difference of +0.2 percentage points.
  • Calendar trend: The apparent weekly rise is largely confined to local-looking 484 caller IDs; the remaining 4,115 calls move from 16.2% to 19.7%.Seven weekly observations are too few to detect a trend of the size at issue.
  • Exposure age: The 18.6%-to-59.4% contrast corresponds to about 37 days of modeled age separation, close to the actual 34-day seeding separation.The regression slope and direct two-group contrast agree within about 8%.
  • Caveats: Lead resale is inferred from correlated spoofed caller ID and rising prevalence, not directly observed.The age curve beyond roughly twenty days rests on a single number, and the labels are detector-assigned.
  • Implication: The headline 29.3% averages over seeding history: the older number reached 59%, while July-seeded numbers were 19% in the same weeks.The study therefore treats seeding history as part of the sample rather than as calendar-time background.

4.5 Automated callers almost never announce themselves

Genuine automation disclosures are nearly absent, and the few observed admissions mostly come from scripted platform banners rather than operators. Distinguishing disclosures from autodialer-consent boilerplate is essential to this count.

  • 37 of 42 apparent disclosures were autodialer-consent templates rather than admissions about who was speaking.
  • 0.44% of 1,816 AI-labeled calls genuinely disclosed automation, while 99.6% did not.The corrected count excludes statements about third-party AI notetakers and distinguishes call-level automation claims from tooling statements.
  • Every disclosure except one was a scripted banner emitted before conversation, and the lone volunteered admission followed direct questioning.
  • Recording or monitoring disclosures were routine: 306 calls announced recording and three announced both recording and automation.
  • The study swept 189,555 transcribed caller turns and hand-adjudicated 82 candidates rather than using a human-labeled comparison group.

4.6 Behavioral signatures, and four nulls

AI-labeled calls show behavioral differences from human-labeled calls, especially more irrelevant or unanswered responses, but several expected operational distinctions do not survive scrutiny. Audio evidence, rather than conversational clustering, separates replayed from fresh machine speech.

  • 13.33% of AI-labeled openings showed a robocall or IVR pattern versus 2.93% of human-labeled openings, with an adjusted odds ratio of 4.71.
  • AI-labeled calls were shorter and had fewer caller turns, but reweighting absorbed 63% of the duration difference and localized the residual to toll-free traffic.
  • Opening lines recurred more across calls from AI-associated numbers than human-associated numbers, with median Jaccard similarity 0.735 versus 0.188.Text similarity cannot distinguish a fresh script rendering from a replayed file.
  • 32.8% of questions went unanswered in AI-labeled calls versus 22.7% in human-labeled calls, and 29.7% of AI-labeled calls ignored most questions versus 14.9%.The ordering was strongest for replayed openings, followed by fresh-synthetic and fresh-human openings.
  • Robocall or IVR openings almost never ended in transfers: 1 of 370 versus 6.27% of other scored calls.
  • Five-feature k-means did not separate prerecorded blasts from conversational agents, with silhouette 0.326 versus a 0.262 noise floor.The audio fingerprint, not conversational clustering, supplied that split.

4.7 Numbers churn everywhere; recordings and voices persist across campaigns

Calling operations preserve scripts, recordings, and voices across changing numbers, while number churn itself often reflects the corpus’s disposable number supply. Repetition is increasingly associated with AI labels, but campaigns can span both detector classes.

  • The median campaign spanned 14.3 days while its longest-lived constituent number typically spanned 0 days.Campaign span therefore outlasted individual numbers, although the sign of that comparison is structurally guaranteed.
  • A permutation null showed that number disposability largely came from the corpus supply, where 2,960 of 3,814 numbers placed only one call.
  • Among 24 large campaigns, number-use patterns were bimodal: eleven used nearly fresh numbers, five used one or two numbers for 18–47 days, and eight mixed both.The fresh-number mode was AI-heavy, but that difference was not significant at n = 24.
  • Automation did not significantly accelerate number turnover: entirely AI-labeled numbers lasted 7.0 days versus 5.3 days for entirely human-labeled numbers.
  • AI labels rose with script repetition, from 14.5% for one-off scripts to 60.0% for campaigns of at least 20 calls.The data cannot determine whether repeatable scripts are automated or automation drives repetition.
  • Audio fingerprinting found 1,177 recorded calls in 239 replay groups, including 34 groups spanning multiple campaigns.One compliance notice opened calls from eight numbers across six campaigns.
  • At least one script family crossed the detector’s synthetic/human boundary, so the paper avoids equating script families with organizations.

4.8 What the scripts do, what the channel hides, and what a single call still shows

The scripts include some humanizing cues, but synthesized callers use them less than human-labeled callers, while telephony processing removes most channel-level differences. Within-call timing and repetition still reveal playback behavior.

  • Fresh-synthetic calls used fewer fillers than fresh-human calls: 1.1 versus 1.7 fillers per hundred caller words.Two thirds of fresh-synthetic calls contained at least one filler, compared with 70% of fresh-human calls.
  • Fresh-synthetic callers began with a name check in 22.6% of calls versus 35.6% for fresh-human callers, while replayed recordings did so in 6.0%.
  • The detector tagged 89.4% of fresh-synthetic calls as American versus 70.7% of fresh-human calls, but the accent tag is a property of the detector as well as the callers.
  • Noise suppression erased most background-audio distinctions: opening noise floors were −96 to −100 dBFS and speech energy centered near 1 kHz across classes.
  • Replayed recordings occupied 43% of opening frames, compared with 31% for fresh-synthetic and 26% for fresh-human callers.Recordings do not pause, making density the surviving channel-level distinction.
  • Replayed openings answered with a median 1.3-second gap and began by talking over the persona in 40% of cases.The study also examined self-replay within calls, while cautioning that replay classification and the fingerprint measure are not independent.
  • Across measurable cues, machine callers hesitated less, checked names less, talked over the persona, repeated themselves, and answered beside the point.

5 Discussion and Conclusion

The measurements show that machine-voiced openings are common, but their policy significance extends beyond voice cloning: telemarketing dominates, numbers are disposable, and longitudinal trends depend on seeding history.

  • Synthetic voice concentrates in lead generation rather than fraud, so a rule framed around impersonation reaches telemarketing at scale.The supplied discussion characterizes the result as a volume mismatch between anti-impersonation framing and the observed traffic.
  • 0.44% disclose automation, providing a baseline for a disclosure duty that does not yet exist rather than a compliance rate.Callers disclose recording more often, making the contrast specific to synthetic-voice disclosure.
  • Campaigns outlast their phone numbers: scripts persist across fresh numbers, so number blocking removes little of most operations’ remaining life.The evidence distinguishes short-lived numbers from a minority of campaigns that retain numbers for weeks.
  • Synthetic prevalence tracks seeded-number age rather than calendar time, making honeypot history a confounder for longitudinal abuse trends.The discussion states that resampling cannot remove this history-dependent confound.
  • Detector uncertainty remains material: listeners confirm 54.4% of flagged calls, and listener criteria shift across clips.The paper also notes weak listener agreement and substantial disagreement in the validation study.
  • 26.9% of greeted calls opened with a machine voice, while another 9.9% were silent connections that may also have been machine-placed.The paper reports 13.8% replayed recordings and 13.1% fresh audio labeled synthetic; silent connections remain an inference rather than a measured machine-voice category.

A Threats to validity

Validity is constrained by partial validation, detector and replay blind spots, opening-only scoring, a narrow seeded population, and exploratory inference.

  • The deployed listening study estimates precision only because it lacks known-human negatives and audits disagreement asymmetrically.The gold set contains no known-human inbound speech, while re-labeling disagreements can bias measured precision upward.
  • 13.6% of identical-waveform pairs cross the detector threshold, showing that telephony-line effects can influence scores.All scored windows are post-processed narrowband audio, so voice and transmission-channel effects cannot be fully disentangled.
  • The labels describe only the first ten seconds of caller speech, not the entire conversation or later voice changes.A call can begin synthetically and later hand off to a live agent without that transition being detected.
  • Replay detection is a floor because recordings played only once leave no repeated waveform and enter the fresh-audio rows.The paper identifies turn-taking and within-call self-replay as promising but not yet classifiers.
  • The sample is actively seeded into a narrow lead-generation vertical, so it does not represent the calling ecosystem or campaign-originating traffic.Only 15% of classified calls meet the paper’s fraud criterion.
  • The analysis is exploratory because roughly fifty comparisons lack family-wise correction and specifications vary by result.Under a Bonferroni factor of 50, only the cited clustered robocall-opening enrichment survives marginally.

B Ethics

The study creates ethical costs by injecting fabricated records and recording non-consenting callers, while limiting exposure through deception, redaction, hashing, and controlled release.

  • Fabricated lead records impose a real cost on third parties by consuming outbound capacity, despite the authors’ judgment that the harm is small.The records are inert and low-volume, but the downstream burden is acknowledged rather than denied.
  • The honeypot deceives callers on numbers the researchers own, with fictitious personas and no reliance on callers’ intent to justify the design.The ethical rationale is adapted from scambaiting while recognizing that the corpus is mostly telemarketing.
  • Listeners hear only short openings, and automated redaction removes names, contact details, and other direct identifiers before serving clips.Round-one clips are screened before exposure to listeners.
  • Released outputs use hashed caller numbers and listening-verified screening for identifiable information, while analysis code and derived labels are committed for release.The paper does not promise a specific release form.
  • Withholding the measurement would leave an already-enforced FCC policy operating without a published baseline, according to the authors.The authors frame disclosure as reducing assumption-driven regulation rather than undoing the policy.
Loading 2609.11137v1…