PaperlayerView exact revision on arXiv

Source-linked AI summary

TeleAntiFraud 2.0: A Refreshable, Profile-Grounded, and Audio-Based Benchmark for Telecom Fraud Detection

Huiyuan Liu, Zhiming Ma, Yanxing Liu, Shun Zhang, Qifan Wang, Di Liu, Yifan Wang, Yuyang Deng, Haoyang Meng, Yijin Zhou, Yuxi Zhao, Chengxian Hu, Peidong Wang, Peng Chen

arXiv:2609.18748v2cs.SDcs.CL

TL;DR

Telecom-fraud benchmarks need to track evolving scams while distinguishing fraud from realistic lawful calls. TeleAntiFraud 2.0 builds monthly frozen audio snapshots from mixed-tree dialogue paths; evaluations show that near-domain negatives expose weaknesses missed by unrelated examples.

  • Problem

    Telecom-fraud evaluation needs to incorporate evolving scam patterns without replacing prior test sets and distinguish fraud from realistic lawful calls rather than unrelated negatives.

    Source: Introduction · p. 1, Related Work · p. 3

  • Method

    TeleAntiFraud 2.0 uses mixed-tree dialogue paths with shared contexts and monthly frozen snapshots to create and preserve audio evaluations.

    Source: Introduction · p. 2

  • Results

    Near-domain siblings reduce three classifiers' Macro-F1 to 0.65–0.68, while full-set audio and ASR+LLM evaluations reveal shortcuts, prediction collapse, and snapshot sensitivity.

    Source: Introduction · p. 2, Main Results · p. 7

  • Takeaways & Limitations

    The findings motivate near-domain construction and collapse-aware reporting, including class-balanced metrics, class-conditional recall, prediction distributions, and collapse behavior.

    Source: Introduction · p. 2, Conclusion · p. 8

    Because the benchmark uses synthetic dialogues and rendered speech, it is a controlled diagnostic instrument, not a claim of matching real telephone conversations fully.

    Source: Construction Analysis · p. 10

Abstract

from arXiv · show

Telecom fraud scripts evolve rapidly and are often designed to resemble routine service conversations, creating two key requirements for audio-based telecom-fraud evaluation. First, benchmarks must incorporate newly observed scam patterns without overwriting previously established test sets. Second, they must distinguish fraud from lawful, near-domain calls rather than relying on topic-separated negative examples. We present TeleAntiFraud 2.0, constructed with our Mixed-Tree Anti-Fraud Generation Pipeline and evaluated under a monthly frozen evaluation protocol. The pipeline transforms online fraud-case abstracts into profile-grounded scenarios, expands them through mixed-tree generation, realizes fraud and non-fraud dialogue paths under shared contexts, renders validated dialogues as role-matched speech, and freezes the resulting audio, labels, prompts, manifests, and provenance records for each monthly evaluation set. Each frozen set contains 900 Chinese calls, comprising 600 fraud and 300 near-domain non-fraud cases. Controlled text experiments show that three classifiers achieve perfect macro-averaged F1 (Macro-F1) when evaluated against unrelated or ordinary negatives, but drop to 0.65-0.68 with near-domain sibling negatives. Full-set audio and automatic-speech-recognition plus large-language-model (ASR+LLM) evaluations further reveal class-prior shortcuts, prediction collapse, and snapshot sensitivity. Together, these findings establish near-domain construction and collapse-aware reporting as core requirements for evaluating audio-based telecom-fraud models under realistic confusable conditions. The accompanying research artifact includes the construction code, evaluation scripts, manifests, and documentation. Our dataset and code are available at https://anonymous.4open.science/r/TeleAntiFraud-2_0-EEB2/.

Introduction

Telecom-fraud evaluation must keep pace with changing scams while distinguishing fraud from realistic lawful calls. TeleAntiFraud 2.0 addresses these challenges with monthly frozen snapshots and mixed-tree dialogues; evaluations expose weaknesses that ordinary negatives miss.

Introduction

  • FDB, Fraud-R1, and TeleAntiFraud-28k provide foundations for fraud, multi-round inducement, and spoken telecom-fraud evaluation, but leave refreshability and realistic lawful counterparts unresolved.TeleAntiFraud-28k is an audio-text benchmark for slow-thinking analysis of fraudulent calls.Source: Introduction · p. 1
  • Fixed test sets cannot absorb newly observed scam patterns, while unrelated negatives let models exploit lexical or source-specific shortcuts instead of distinguishing caller actions.Constantly replacing test examples also makes results difficult to reproduce and compare.Source: Introduction · pp. 1–2
  • TeleAntiFraud 2.0 freezes monthly snapshots while allowing later releases to incorporate new cases, preserving audio, labels, prompts, model responses, and provenance for reproducible audits.Each published snapshot remains immutable.Source: Introduction · p. 2
  • The Mixed-Tree Anti-Fraud Generation Pipeline converts fraud summaries into profiles, dialogue trees, collaborative role-play paths, and role-matched speech.Fraud and lawful sibling paths share context and early turns, diverging when label-bearing actions emerge; labels depend on the completed interaction rather than topic cues.Source: Introduction · p. 2
  • Near-domain sibling negatives reduce Macro-F1 to 0.65–0.68, while audio and ASR+LLM evaluations reveal false-positive bias, class-prior shortcuts, prediction collapse, and snapshot variation.The two current snapshots support snapshot-sensitivity analysis, but longer-term temporal claims require additional releases.Source: Introduction · p. 2

Related Work

Prior work provides shared fraud-benchmark interfaces, interactive inducement tests, and spoken telecom-fraud resources, while dynamic evaluation and provenance have also gained attention. The remaining gap is refreshable, auditable audio evaluation with near-domain lawful counterparts.

Related Work

  • FDB aggregates public fraud datasets, Fraud-R1 tests multi-round inducements, and TeleAntiFraud-28k evaluates spoken telecom fraud, but existing benchmarks do not directly model spoken telecom calls across these lines.FDB highlights class imbalance, heterogeneous features, temporal patterns, and adversarial behavior; Fraud-R1 includes role-play settings.Source: Related Work · p. 2
  • Dynabench uses human-and-model-in-the-loop collection, while datasheets emphasize provenance, intended use, collection choices, and distribution constraints.TeleAntiFraud 2.0 applies immutable monthly sets and a shared schema and manifest contract to a domain-specific audio setting.Source: Related Work · p. 3
  • Fixed released sets cannot absorb later scam patterns or supply enough near-domain lawful counterparts; TeleAntiFraud-28k also lacks raw transcripts and has label-correlated audio-path prefixes.These gaps motivate refreshable, auditable audio evaluation with paired fraud/non-fraud siblings and collapse-aware auditing.Source: Related Work · p. 3

Method

The method turns online fraud-case abstracts into profile-grounded dialogue paths and role-matched audio, then assembles the validated calls into immutable monthly evaluation snapshots. Shared-context branches keep fraud and lawful paths comparable while their labels depend on later interaction evidence.

Method overview

  • The four-stage pipeline profiles cases, expands mixed trees, realizes dialogues collaboratively, and renders role-matched speech with signal validation before snapshot assembly.The stages are scenario profiling, mixed-tree expansion, collaborative dialogue realization, and speech rendering; corrupted audio is filtered before assembly.Source: Mixed-Tree Anti-Fraud Generation… · p. 3

Scenario Profiling

  • Scenario profiles specify participants, claimed identity and persuasion strategy, staged objectives, risk entities, and points where calls may escalate, permit verification, or terminate.The profile keeps participants, setting, and early context shared across sibling branches while leaving actions, wording, and delivery styles to later stages.Source: Scenario Profiling · p. 3

Mixed-Tree Expansion

  • Mixed-tree expansion creates outcome-divergent paths from a shared profile and opening context, so labels depend on later fraud-relevant actions rather than superficial cues.A call family can retain ambiguous, fraud-leaning, non-fraud-leaning, and naturally terminating continuations; sibling paths preserve context while varying branch actions.Source: Mixed-Tree Expansion · pp. 3, 5
  • Each open node proposes at most b plot actions, and each selected action is recorded in the path history as a candidate child one level below its parent.The child’s state and expansion flag are updated, then structural validity and profile consistency determine whether it is retained for further expansion.Source: Mixed-Tree Expansion · pp. 3–4
  • With D = 4 and b = 3, trees permit at most 81 leaves before pruning, and only terminal paths with sufficient label-bearing evidence receive class labels.Natural endings, invalid branches, and under-specified paths reduce the realized tree; early termination is retained only when the completed trajectory supports a label.Source: Mixed-Tree Expansion · pp. 4–5

Collaborative Dialogue Realization and Speech Rendering

  • Six agents separate branch and stopping controls from role-conditioned utterance generation and delivery-state assignment, keeping plot decisions distinct from surface wording.The resulting trace links turns to branch actions and delivery states; distinct role-compatible voices render caller and receiver turns, and lightweight signal checks remove corrupted audio.Source: Collaborative Dialogue Realization · p. 5, Speech Rendering and Quality Control · p. 6

TeleAntiFraud-2.0 Benchmark

  • Monthly snapshots can incorporate newly collected case abstracts while remaining immutable after release; each is a pure test set with checks before freezing and traceable manifests.The two current snapshots, June/V1 and July/V2, each contain 900 Chinese calls: 600 fraud and 300 near-domain non-fraud cases, independently sampled under the same contract.Source: TeleAntiFraud-2.0 Benchmark · p. 6

Experiments

The experiments test whether mixed-tree siblings reduce shortcut separability and whether frozen monthly snapshots reveal boundary errors, prediction collapse, and sampling sensitivity. Results show that near-domain negatives are harder, while full-set evaluations expose collapse and class-prior effects that single-score reporting can miss.

Experimental setup and evaluation scope

  • Full-set runs evaluated direct-audio and ASR+LLM configurations on the 900-call June/V1 and July/V2 snapshots, each with 600 fraud and 300 near-domain non-fraud calls.The ASR+LLM runs reused one Whisper-medium transcript per audio file; manifests recorded configuration details and raw and parsed outputs, while reporting paired fraud F1 with class-balanced metrics and prediction distributions.Source: Experimental Setup · p. 6

Mixed-tree siblings reduce shortcut separability

  • 0.650–0.680 Macro-F1 on mixed-tree siblings contrasted with perfect separation from unrelated or ordinary negatives across three classifiers.The negative calls progressed from unrelated calls to ordinary telecom-service calls and mixed-tree siblings; lexical overlap increased as negatives became more closely matched.Source: Table 4 · p. 7, Main Results · p. 7

Near-domain calls exhibit lower linear separability

  • Under the shared TF–IDF+SVM classifier, TeleAntiFraud 2.0 showed lower linear separability than TeleAntiFraud-28k, especially in the ASR-test setting.Non-fraud recall dropped and predictions became fraud-biased; differences in sources, audio pipelines, and label definitions prevent attributing the whole gap to one construction factor.Source: Table 5 · p. 7, Main Results · p. 7

Frozen snapshots expose model collapse across families

  • 11 of 27 V1 and 15 of 27 deduplicated V2 configurations showed an all-FRAUD-like signature before prompt-language averaging.Fraud recall approached 1.0, accuracy approached the 2:1 fraud prior, and fraud F1 approached 0.80.Source: Table 6 · p. 7, Main Results · p. 7

Class-prior analysis motivates collapse-aware reporting

  • Fraud F1 for the all-FRAUD baseline rose from 0.500 to 0.800 as fraud prevalence increased, while Balanced Accuracy stayed 0.500 and non-fraud recall stayed zero.MiniMax-CN remained fraud-biased even at a 1:1 ratio, predicting FRAUD for 0.830 of calls and recalling only 0.233 of non-fraud calls.Source: Main Results · p. 7

Experimental findings

  • Across experiments, near-domain siblings weakened topic-level shortcuts, frozen snapshots exposed action-recognition failures, and class-prior resampling distinguished boundary failure from majority-class prediction.The authors recommend preserving raw predictions and interpreting fraud F1 alongside class-balanced metrics and output distributions.Source: Main Results · p. 7

Conclusion

TeleAntiFraud 2.0 combines mixed-tree generation, role-matched speech, and immutable monthly snapshots to evaluate fraud against lawful near-domain siblings. Its evaluations expose shortcut separability, prediction collapse, class-prior effects, and snapshot sensitivity, motivating balanced metrics and frozen-snapshot reporting alongside fraud F1.

Conclusion

  • TeleAntiFraud 2.0 combines mixed-tree generation, role-matched speech, and immutable monthly snapshots to compare fraud calls with lawful near-domain siblings.Later releases can incorporate new scam patterns without overwriting earlier test sets.Source: Conclusion · pp. 7–8
  • Controlled text, audio, and ASR+LLM evaluations reveal reduced shortcut separability, prediction collapse, class-prior effects, and snapshot sensitivity.The findings motivate balanced metrics, output distributions, and frozen-snapshot reporting alongside fraud F1.Source: Conclusion · p. 8
  • The benchmark’s stable contract is intended to support adding emerging scam cases while keeping prior results auditable and comparable.Source: Conclusion · p. 8

Ethical Statement

The artifacts are intended solely for defensive research on telecom-fraud detection.

Ethical Statement

  • Synthetic, non-private artifacts are restricted to defensive analysis, not deception, impersonation, or scam training.Source: Ethical Statement · p. 8

Sibling Examples and Label Rationales

The examples pair fraud and lawful calls that share scenario context and early interaction, with their paths diverging at actions that provide label evidence. A path-level rationale records the basis for the final label.

Sibling Examples and Label Rationales

  • The redacted held-out pair shares the caller role, event description, receiver profile, and early risk framing; abbreviated summaries replace verbatim operational turns.The artifact retains the corresponding dialogue text, path state, metadata, and rationale records.Source: Sibling Examples and Label Rationales · p. 10
  • Each terminal leaf’s rationale records shared context, the separating action, post-action label evidence, and the final binary label.This explains how a lawful call can contain risk vocabulary and urgency while the completed trajectory determines the fraud label.Source: Sibling Examples and Label Rationales · p. 10
  • The illustrative sibling paths share a risk-control context and early interaction, then diverge as fraud moves toward sensitive off-channel actions while lawful verification stays in official channels.Source: Figure S1 · p. 10

Construction Analysis

The construction audit examines mixed-tree structure and transfer, dialogue quality and label traceability, speech-rendering controls, and frozen-snapshot metadata. It reports tree-held-out performance, expert label judgments, a single-annotator dialogue pilot, and limits on synthetic-data scope and voice representativeness.

Synthetic-data scope

  • Synthetic dialogues and rendered speech support controlled diagnostics, but the authors do not claim they fully match real telephone conversations.The pipeline grounds scenarios in collected case abstracts, pairs sibling paths under shared context, records path-level label rationales, and applies structural, safety, and signal checks.Source: Construction Analysis · p. 10

Mixed-tree structure and transfer

  • Leave-one-tree-out F1 drops from 0.883 in-domain to 0.605 on held-out trees; within-tree and cross-tree distances are 0.0188 and 0.3351, a 17.8× ratio.The authors motivate tree-aware splits if the generator is later used for training.Source: Construction Analysis · p. 10

Profile-grounded dialogue and label traceability

  • Corrected-gold agreement averages 0.792 across 1,000 expert judgments, ranging from 0.700 to 0.880; 744 judgments are marked evidence-sufficient.Ten independent experts each reviewed a 100-item packet.Source: Construction Analysis · p. 10
  • The 80-dialogue package’s single-annotator pilot scores realism 4.60, strategy coherence 4.41, victim-reaction plausibility 4.35, and audio naturalness 4.13 on a 1–5 scale.Additional packet files derived from the first annotator are excluded from independent-rating claims.Source: Construction Analysis · p. 10

Speech-rendering controls

  • The 35-voice pool comprises 29 male and 6 female profiles; removing emotion tokens from 15 paired dialogues changes aggregate F1 from 0.920 to 0.960.The authors treat emotion as a controlled rendering variable and make no causal claim about its effect; demographic representativeness remains a limitation.Source: Construction Analysis · p. 10

Class-Prior Rationale and Reporting Protocol

The benchmark keeps a fixed 2:1 fraud/non-fraud composition across monthly snapshots to maintain comparable evaluations while retaining near-domain lawful calls. Its reporting protocol pairs class-balanced metrics and both recalls with predicted-fraud ratio to reveal class-prior shortcuts.

Class composition

  • The fixed 2:1 fraud/non-fraud composition is intended to cover diverse, evolving fraud trajectories while retaining substantial near-domain lawful calls, and it stays consistent across monthly snapshots.Consistent composition enables comparable evaluation as new fraud patterns are introduced.Source: Class-Prior Rationale and Reporting… · p. 10

Reporting protocol

  • All-FRAUD predictions raise fraud F1 from 0.500 to 0.800 while Balanced Accuracy stays 0.500 and non-fraud recall remains zero.The main paper reports Macro-F1, Balanced Accuracy, both recalls, and predicted-fraud ratio; manifests also support balanced 1:1 re-evaluation.Source: Class-Prior Rationale and Reporting… · p. 10

Raw Full-Set Evaluation Results

The appendix preserves raw full-set evaluation configurations and a redacted held-out sibling example. It distinguishes configuration-level rows from prompt-language aggregation and excludes pilot files from the raw tables.

Raw configuration tables

  • Tables S2 and S3 preserve configuration-level rows for the full 900-sample evaluations before prompt-language aggregation.Each row represents a model–prompt-language–input-mode configuration.Source: Raw Full-Set Evaluation Results · pp. 10–11

Held-out example and pilot exclusion

Raw configuration tables

  • Table S2 covers 27 June/V1 configurations over 900 samples; Table S3 covers 28 July/V2 rows, retaining a repeated Seed-2.0-lite-EN-Omni-V2 entry.These captions describe table scope, not comparative outcomes.Source: Table S2 · p. 12, Table S3 · p. 12
Loading 2609.18748v2…