PaperlayerView exact revision on arXiv

Source-linked AI summary

Vishing-Tactics-Bench: Forecasting Exploitation Trajectories in Voice Phishing Calls

Jeongmin Lee, Dongmyung Sul, Seung Yun, Jinxia Huang

arXiv:2609.07151v1cs.CL

TL;DR

Post-call fraud classification does not say which harm an ongoing vishing call is approaching or when to intervene. The paper builds a turn-level tactical benchmark and forecasts two terminal harms; its results show that tactical information can improve forecasting, with gains depending on how it is integrated.

  • Problem

    Call-level fraud classification does not identify whether an ongoing vishing call is progressing toward Information Gathering or Financial Exploitation, and post-call predictions are no longer actionable for intervention.

    Source: §1 · p. 1

  • Method

    The paper adapts a vishing tactic taxonomy and defines ETF to predict, at each turn, whether either terminal harm will occur within the next k turns.

    Source: §1 · p. 1, §5 · p. 4

  • Results

    Tactical information adds predictive signal with gains dependent on integration; adding the trajectory raises gpt-4.1-mini Financial Exploitation AP@1 fivefold, from 0.07 to 0.35.

    Source: §7 · p. 5

  • Takeaways & Limitations

    The tactical trajectory is a state representation for projecting which harm a call approaches, supporting harm-specific intervention timing and targeting.

    Source: §9 · p. 8, §10 · p. 9

    Financial Exploitation occurs in only 6.4% of calls, so its low AP cannot be fully separated into intrinsic difficulty versus sample scarcity.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Voice phishing (vishing) unfolds in real time; by the time a call has ended and post-hoc classification is possible, the harm has already been done. The more actionable question is which concrete harm (Information Gathering or Financial Exploitation) an ongoing call is tactically progressing toward. We present Vishing-Tactics-Bench, a benchmark grounded in Endsley's situation-awareness (SA) framework that recasts vishing defense from after-the-fact fraud classification to harm projection: predicting at each turn whether the call will reach either terminal harm. We adapt MITRE ATT&CK to vishing as a 6-tactic taxonomy (Vishing-Tactics) and label 35,340 scammer utterances across 5,645 synthetic Chinese calls. We define Exploitation Trajectory Forecasting, a survival-style protocol over the two terminal harms with three metrics: AP@k, C-index, and divergence error. Baselines ranging from a Markov heuristic to fine-tuned LLMs show that the tactical trajectory serves as an interpretable representation of the call's tactical state, supporting harm-specific forecasting, which can then be used for the downstream application of intervention selection; a stratified lead-time analysis at a tight false-alarm budget further identifies at what point in a call the trajectory signal yields early warning.

1 Introduction

The paper shifts vishing defense from post-call fraud classification to forecasting which concrete harm an ongoing call is approaching and when intervention may be possible. It introduces a tactical trajectory representation and Exploitation Trajectory Forecasting (ETF), building on work in scam-call NLP, cybersecurity attack stages, and situation awareness.

1 Introduction

  • Existing call-level fraud labels do not specify which harm to block, and post-call predictions arrive too late for intervention during the call.Prior in-call work describes current states rather than forecasting the terminal harm a call is approaching.Source: §1 · p. 1, §2 · p. 2
  • The benchmark adapts MITRE ATT&CK to label scammer utterances, representing each call as a sequence of tactics rather than a call-level label.It applies a six-tactic taxonomy to 5,645 Chinese calls and 35,340 scammer utterances.Source: §1 · pp. 1–2
  • ETF predicts at each turn whether an ongoing vishing call will reach Information Gathering or Financial Exploitation within the next k turns.These harms call for different interventions: Information Gathering extracts personal information, while Financial Exploitation directly demands money.Source: §1 · p. 1

2 Related Work

  • The work frames tactic labeling, trajectory comprehension, and harm projection as three levels of situation awareness, while relating its trajectory approach to cybersecurity stage models and conversational forecasting.Prior conversational forecasting described in the paper predicts single outcomes rather than a harm-specific hazard trajectory.Source: §1 · p. 1, §2 · pp. 2–3

1 Introduction

  • Baselines from a Markov heuristic to fine-tuned LLMs are used to analyze ETF and the predictive value of tactical trajectories.The paper reports that this value depends on how trajectory information is integrated into a model.Source: §1 · p. 2

3 Vishing-Tactics Taxonomy

Vishing-Tactics adapts the TTP hierarchy to describe an attacker’s objectives, concrete methods, and detailed procedures. Its six tactics cover the progression from contact and persuasion to information or financial harm, as well as evasion.

3 Vishing-Tactics Taxonomy

  • A tactic names the attacker’s objective, a technique its concrete method, and a procedure its detailed process.Together, TTP describes attacker intent and plan rather than surface means.Source: §3 · p. 3
  • The six tactics are Initial Contact, Trust Building, Psychological Pressure, Information Gathering, Financial Exploitation, and Evasion.Information Gathering and Financial Exploitation are the terminal harms ETF forecasts; the other tactics describe the process leading toward them.Source: §3 · p. 3
  • Utterances with no tactical intent are labeled NONE, and the benchmark uses tactic-level labels throughout; technique-level annotation is released for future fine-grained analysis, with full definitions and distributions in Appendix A.Source: §3 · p. 3

4 Dataset Construction

The dataset adds utterance-level tactic annotations to synthetic Chinese vishing calls and characterizes their tactic and harm distributions. The paper then defines ETF as turn-level forecasting of two terminal harms from dialogue and tactical-trajectory inputs, evaluated with three metrics.

4 Dataset Construction

  • Two LLMs labeled scammer utterances in synthetic Chinese TeleAntiFraud calls, with inter-model Cohen’s κ = 0.94; human review of a stratified test sample yielded inter-human κ = 0.69 and joint Fleiss’s κ = 0.66.The 5,645 calls were selected for available tactic and original fraud-type labels and split stratified by fraud type.Source: §4 · p. 3
  • Trust Building, Psychological Pressure, and Information Gathering make up 82.2% of scammer turns, while Financial Exploitation and Evasion each account for about 1.4%.Information Gathering accounts for 31.5% of turns versus 1.4% for Financial Exploitation, a roughly 20-fold frequency gap.Source: §4 · p. 4, Table 2 · p. 4
  • Information Gathering is reached by 85.8% of calls, compared with 6.4% for Financial Exploitation, and about 36% of calls reaching Financial Exploitation do so without first reaching Information Gathering.Kidnap fraud often follows a Financial Exploitation pathway, while phishing almost always terminates at Information Gathering.Source: §4 · p. 4

5 Task: Exploitation Trajectory Forecasting

  • At each turn, ETF estimates whether either terminal harm will occur within the next k turns, conditioning on the harm not yet having occurred.It evaluates AP@k, C-index, and divergence error separately for the two harms, using dialogue alone or dialogue plus the TTP trajectory as input.Source: §5 · p. 4
  • Divergence error measures the turn distance between the steepest rise in predicted hazard and the first appearance of the harm, averaged over calls that reach it.Calls that never reach the harm have no ground-truth branch and are excluded from the average.Source: §5 · pp. 4–5

6 Baselines

The benchmark compares statistical, discriminative, and generative baselines, with the central analysis testing whether tactical information adds signal beyond dialogue text. The models differ in how they use text and the TTP trajectory.

6 Baselines

  • The baseline suite spans statistical, discriminative, and generative modeling families.It ranges from a Markov heuristic to encoder-based models and fine-tuned LLMs.Source: §6 · p. 5
  • The Markov heuristic estimates tactic transitions from training trajectories and iterates them to compute reach probabilities without using dialogue text.This baseline measures how predictable ETF is from transition structure alone.Source: §6 · p. 5
  • Encoder models use text, TTP trajectories, or both, while LLM baselines predict reach zero-shot or after supervised fine-tuning, with and without TTP.The text+TTP encoder configuration late-fuses its encoders; the TTP-included/excluded comparison tests the contribution of tactical information.Source: §6 · p. 5

7 ETF Results

Learned baselines outperform non-learned baselines across both terminal harms, while tactical-trajectory gains vary by harm and model integration. Predicted-TTP results favor late fusion over inline fusion for Financial Exploitation forecasting.

Overall results

  • Learned baselines substantially outperform the Markov heuristic and zero-shot gpt-4.1-mini on both harms, with the largest gap on Information Gathering.Table 3 reports AP@k, C-index, and divergence error on the standard test split.Source: §7 · p. 5, Table 3 · p. 6

AP@k

  • Information Gathering AP@k is 0.87–0.97 for learned models versus 0.27–0.45 for zero-shot; Financial Exploitation AP@k is 0.51–0.67 versus 0.07–0.39.Financial Exploitation has a 0.02–0.04 window base rate, so its AP should be read against that baseline.Source: §7 · p. 5

C-index

  • Information Gathering C-index reaches 0.96–0.98 for the encoder and gpt-4.1-mini SFT, versus 0.39 for text-only zero-shot gpt-4.1-mini.Financial Exploitation C-index rises from 0.67 for Markov to 0.92 for the encoder.Source: §7 · p. 5

Divergence error

  • Financial Exploitation branch timing is located closely by several transition- or trajectory-aware models, while Information Gathering divergence error is generally 1.2–2.1 turns.Information Gathering unfolds gradually, leaving no sharp branch for any model to locate.Source: §7 · p. 5

TTP integration

  • TTP carries non-redundant signal, but gains depend on integration: encoder-only TTP reaches Information Gathering AP@3 0.71 and C-index 0.84, while adding TTP raises gpt-4.1-mini Financial Exploitation AP@1 fivefold, 0.07 to 0.35.With oracle TTP, inline-fusion LLMs gain more on Financial Exploitation AP@1 than the encoder; the encoder gains only +0.01 on Information Gathering C-index, already 0.98 text-only.Source: §7 · p. 5

Model comparison

  • With oracle TTP, zero-shot GPT-4o Financial Exploitation AP@1 rises from 0.11 to 0.26, and reaches 0.17 with predicted TTP, below the encoder’s 0.60.This additional comparison shows that trajectory gains do not erase differences between model families.Source: §7 · p. 6

Predicted-TTP robustness

  • With predicted TTP, the encoder preserves Financial Exploitation AP@1 at 0.60, while gpt-4.1-mini SFT falls from 0.67 with gold TTP to 0.30.The inline-fusion decline tracks lower T1005 classifier F1 (0.58 versus 0.90 on T1004); its Information Gathering AP@1 falls by only 0.08, from 0.93 to 0.85.Source: §7 · p. 6

Harm and fraud-type differences

  • Information Gathering is common (85.8% reach) and gradual, whereas Financial Exploitation is rare (6.4%) and abrupt; TTP gains also vary by fraud type.The largest reported gains are +0.45 Information Gathering AP@1 on kidnap and +0.31 Financial Exploitation AP@1 on investment; kidnap and lottery test sets are small and results suggestive.Source: §7 · p. 6

8 Anatomy of the Two Terminal Harms

Outcome-conditioned analyses distinguish how models rank Financial Exploitation risk and update it after an Information Gathering branch. Under a tight false-alarm budget, early-warning performance varies by trajectory and model.

Call outcome typology

  • The two harms are neither strictly sequential nor independent: about 5% of Information Gathering-reaching calls also reach Financial Exploitation, while about 40% of Financial Exploitation-reaching calls bypass Information Gathering.Neither-outcome calls are short (median 2 turns) and stall in trust building or pressure.Source: §8 · p. 6

Outcome-stratified hazard

  • The encoder’s Financial Exploitation hazard@3 cleanly separates outcomes: Both 0.28, Financial Exploitation-only 0.33, Information Gathering-only 0.01, and Neither 0.06 with gold TTP.Predicted-TTP values are reported as nearly identical; adding trajectory corrects the text-only zero-shot model’s inverted ordering.Source: §8 · p. 7

Post-branch dynamic update

  • On calls reaching both harms, gpt-4.1-mini zero-shot with trajectory has the largest oracle-TTP post-branch lift in Financial Exploitation hazard@3 (Δ=+0.231), while GPT-4o has the largest predicted-TTP lift (Δ=+0.208).The encoder remains positive in both settings (+0.116 gold, +0.089 predicted); inline-fusion SFT models are near zero.Source: §8 · p. 7

Lead-time protocol

  • The lead-time analysis sets each alarm at the first turn its predicted hazard exceeds a threshold, fixes FAR at ≤10% on non-reaching calls, and bins lead time relative to the harm turn.Because all test calls are fraud calls, results are stratified by Information Gathering-only, Financial Exploitation-only, and Information Gathering → Financial Exploitation trajectories.Source: §8 · pp. 7–8

Lead-time breakdown

  • Under predicted TTP and FAR ≤10%, adding the trajectory raises encoder early warnings on Information Gathering → Financial Exploitation calls from 70% to 75%; fine-tuned gpt-4.1-mini text warns early on 93% of Financial Exploitation-only calls.The latter estimate is high-variance (n=27); inline-fusion TTP reduces early warnings to 82% on Financial Exploitation-only calls and 10% on Information Gathering → Financial Exploitation calls.Source: §8 · p. 8, Figure 2 · p. 8
  • The Markov baseline fires at or before the harm turn on 96% of predicted-TTP Financial Exploitation-only calls, mostly on-time rather than early; only encoder variants remain actionable for Information Gathering-only calls.For the Financial Exploitation-only stratum, 33% are early and 63% on-time.Source: §8 · p. 8

9 Discussion

The TTP trajectory is an interpretable tactical state whose forecasting value depends on the harm and how the model uses it. The discussion connects this projection to targeted intervention while identifying privacy, transition-learning, and validation boundaries.

9 Discussion

  • The trajectory adds little for common, gradual harms but is a dominant signal for estimating and timing rare, abrupt harms.It is useful where text alone falls short in identifying which harm a call approaches and when.Source: §9 · p. 8
  • A TTP-only encoder could support on-device tactic classification and sharing only abstract tactic sequences, but at an accuracy cost and without empirical deployment evaluation.The comparison is with text+TTP; this is presented as an architectural option.Source: §9 · p. 8
  • LLMs given the TTP trajectory still did not match Markov on Financial Exploitation branch timing, motivating learned corpus-specific transitions.The authors argue strong ETF models should combine text semantics with learned transitions.Source: §9 · p. 8

10 Conclusion

  • The conclusion presents the benchmark as an utterance-level, situation-awareness-grounded resource and identifies learned tactical trajectories as a key predictive signal.The trajectory is described as best exploited when learned from data.Source: §10 · p. 9
  • Because intervention differs by terminal harm, targeted defense depends on harm specificity; future work includes real-time validation, victim-conditioned forecasting, broader data, and alternative hazards.Before terminal tactics, intervention remains possible; the current call-as-sequence formulation does not model victim responses.Source: §10 · p. 9

Limitations

The benchmark relies on synthetic Chinese vishing conversations, with real-recording validation constrained by legal and ethical access barriers. Its scope also leaves cross-language generalization, benign-call false alarms, and the interpretation of harm-specific difficulty unresolved.

Limitations

  • Synthetic data is a deliberate design choice because real vishing recordings face legal and ethical collection and release restrictions, but synthetic conversations may not fully reflect real calls.Validation on real recordings remains future work contingent on resolving access barriers.Source: Limitations · p. 9
  • The corpus contains Chinese vishing conversations only, so generalization to other languages and cultures is unverified.Source: Limitations · p. 9
  • Financial Exploitation has far fewer positive examples than Information Gathering, making intrinsic difficulty hard to separate from sample scarcity.The reported reach rates are 6.4% for Financial Exploitation and 85.8% for Information Gathering.Source: Limitations · p. 9
  • Because the benchmark contains only vishing calls, its false-alarm rates do not measure behavior on legitimate calls.Adding benign-call controls is left for future work.Source: Limitations · p. 9

Ethics Statement

  • The authors identify possible misuse of tactic annotations, while noting that the corpus is synthetic and the abstract tactic labels add no new attack capabilities.Source: Ethics Statement · p. 9

A Vishing-Tactics Taxonomy and Annotation

The benchmark adapts MITRE ATT&CK into a vishing tactic taxonomy and annotates scammer utterances using LLMs, with human validation supporting the labels at corpus scale. Sparse technique counts motivate tactic-level use in experiments.

A.1 Full Taxonomy: 6 Tactics, 24 Techniques

  • The taxonomy defines six tactics and 24 techniques, while mid-call recordings exclude Initial Contact and annotations cover the remaining five tactics and NONE.Six of the 21 annotated techniques have fewer than 35 instances, including four with single-digit counts.Source: §A.1 · p. 11

A.2 Annotation Procedure

  • Two LLMs independently labeled every scammer utterance using full-call context, with corpus-level inter-model agreement of Cohen’s κ = 0.94.The models assigned tactic, technique, confidence, and reasoning; turns without a clear fraud pattern could be labeled NONE.Source: §A.2 · p. 11
  • The validation sample was stratified to cover rare Financial Exploitation and Evasion classes, and annotators used full-call context for offline ground truth.The runtime classifier is instead restricted to prefix-only input for streaming compatibility.Source: §A.2 · p. 12
  • Human–human agreement was κ = 0.69, LLM–LLM agreement was κ = 0.97, and four-rater Fleiss’s κ was 0.66 on the validation subset.Residual disagreement concentrates on the Trust Building / Psychological Pressure boundary.Source: §A.2 · p. 12, Table 6 · p. 12

B Corpus Statistics

B.1 Composition and reach by fraud type

Calls show varied relationships between the two terminal harms, including frequent Information Gathering before Financial Exploitation and late arrival of the latter. TTP gains also vary by fraud type, and the appendix reports distinct uncertainty conventions across model families.

B.2 Co-occurrence and timing of the Two Terminal Harms

  • Of 365 calls reaching Financial Exploitation, 232 (64%) first pass through Information Gathering and 133 (36%) reach it directly; only about 5% of Information Gathering calls proceed to Financial Exploitation.The first Financial Exploitation utterance occurs at a mean of 4.0 scammer turns, about 65% of the way through the call.Source: §B.2 · p. 13, Table 11 · p. 13

B.3 Per-row uncertainty for Table 3

  • Encoder and Qwen2.5-7B SFT results average three seeds, while Markov, zero-shot LLMs, and gpt-4.1-mini SFT use turn-level bootstrap confidence intervals.The seed-based runs use seeds 42, 1337, and 2024; bootstrap intervals use B = 2000.Source: §B.3 · p. 13

B.4 Per-fraud-type heterogeneity

  • TTP gains are type-conditional: on kidnap, gpt-4.1-mini SFT Information Gathering AP@1 rises from 0.32 to 0.77 (+0.45), while investment Financial Exploitation AP@1 rises by +0.31.Kidnap is finance-direct, whereas investment trajectories resemble PII-only types before a late branch.Source: §B.4 · p. 13
  • On bank Financial Exploitation, text-only Encoder outranks gpt-4.1-mini SFT (text+TTP) by +0.26 AP@1 despite far fewer parameters and no TTP signal; TTP gains concentrate where trajectory shape deviates from the population average.TTP becomes redundant where harm-bearing phrasing is stereotyped.Source: §B.4 · p. 14

C Cascaded TTP Classifier

The section evaluates a prefix-only RoBERTa classifier that supplies predicted tactics to the forecasting pipeline, comparing context windows and reporting per-class performance. It also contrasts the predicted-TTP lead-time pattern with the oracle-TTP upper bound.

C Cascaded TTP Classifier

  • A 110M-parameter Chinese RoBERTa with a six-class linear head is trained on labeled turns, and inference uses only the prefix through the current scammer turn.The current-turn classifier is used as the cascaded TTP source for predicted-TTP rows.Source: C Cascaded TTP Classifier · pp. 14–15
  • Under oracle TTP, Figure 3 gives the upper bound on TTP-driven lead-time lift; predicted TTP preserves the encoder lead for Information Gathering → Financial Exploitation but degrades inline-fusion variants on Financial Exploitation only.The degradation tracks the per-class F1 gap.Source: Figure 3 · p. 15, C Cascaded TTP Classifier · p. 15
  • Current-turn input yields macro-F1 0.748, versus 0.701 with four preceding turns and 0.634 with the full prefix.The same ordering holds across all four non-NONE tactic classes.Source: C Cascaded TTP Classifier · p. 14, Table 14 · p. 15
  • The current-turn classifier achieves test accuracy 0.846 and macro-F1 0.748, with F1 of 0.903 for Information Gathering and 0.584 for Financial Exploitation.Financial Exploitation and Evasion errors are concentrated in the two rare tactics, corresponding to a sharper predicted-TTP Financial Exploitation AP@1 drop than for Information Gathering.Source: C Cascaded TTP Classifier · p. 15

D Sequence-Encoder Ablation

Across sequence encoders, late-fusion text+TTP results remain similar despite changing the trajectory encoder. Two examples show that trajectory information can warn before a harm, but can miss an abrupt financial demand.

E Qualitative Examples

  • In customer-service impersonation, text+TTP reaches hazard 0.99 two turns before harm; in kidnap fraud, it misses the abrupt demand (0.04) while text-only reacts (0.66).The kidnap example has no trust-building, pressure, or information-gathering trajectory before the demand.Source: E Qualitative Examples · pp. 15–16

D Sequence-Encoder Ablation

  • Late-fusion text+TTP stays within one seed’s variance across GRU, LSTM, and Transformer encoders, while predicted-TTP changes Financial Exploitation AP@1 by at most 0.01.In TTP-only results, the Transformer reaches Financial Exploitation AP@1 0.393 versus 0.108 for the GRU with gold TTP; its Information Gathering C-index reaches 0.94 under predicted TTP.Source: D Sequence-Encoder Ablation · p. 15, Table 16 · p. 16
Loading 2609.07151v1…