PaperlayerView exact revision on arXiv

Source-linked AI summary

PreScam: A Benchmark for Predicting Scam Progression from Early Conversations

Weixiang Sun, Shang Ma, Yiyang Li, Tianyi Ma, Zehong Wang, Colby Nelson, Xusheng Xiao, Yanfang Ye

arXiv:2605.12243v1cs.CL

TL;DR

Existing research offers limited evidence that models can understand how real-world scams progress through sequential psychological manipulation. PreScam structures real-world scam reports into a progression benchmark with staged annotations and two prediction tasks, finding that models capture some scam cues but still struggle to track escalation and subsequent actions.

  • Problem

    Existing research largely studies static or synthetic scams, leaving limited evidence about whether models understand real-world scam progression as sequential psychological manipulation.

  • Method

    PreScam structures real-world scam reports into staged, turn-level annotated conversations and evaluates real-time termination and subsequent scammer-action prediction.

  • Results

    Models capture useful scam-related cues but remain limited in modeling sequential psychological actions, with scammer-action prediction challenging even for frontier LLMs.

  • Takeaways & Limitations

    PreScam reframes conversational scam understanding as progression modeling, enabling evaluation of whether models track escalating risk and anticipate scammer behavior.

  • Takeaways & Limitations

    The benchmark uses publicly accessible scam reports from BBB Scam Tracker for academic and defensive evaluation.

Abstract

from arXiv · show

Conversational scams, such as romance and investment scams, are emerging as a major form of online fraud. Unlike one-shot scam lures such as fake lottery or unpaid toll messages, they unfold through multi-turn conversations in which scammers gradually manipulate victims using evolving psychological techniques. However, existing research mainly focuses on static scam detection or synthetic scams, leaving open whether language models can understand how real-world scams progress over time. We introduce PreScam, a benchmark for modeling scam progression from early conversations. Built from user-submitted scam reports, PreScam filters and structures 177,989 raw reports into 11,573 conversational scam instances spanning 20 scam categories. Each instance is hierarchically structured according to the scam lifecycle defined by the proposed scam kill chain, and further annotated at the turn level with scammer psychological actions and victim responses. We benchmark models on two tasks: real-time termination prediction, which estimates whether a conversation is approaching the termination stage, and scammer action prediction, which forecasts the scammer's subsequent actions. Results show a clear gap between surface-level fluency and progression modeling: supervised encoders substantially outperform zero-shot LLMs on real-time termination prediction, while next-action prediction remains only moderately successful even for strong LLMs. Taken together, these results show that current models can capture some scam-related cues, yet still struggle to track how risk escalates and how manipulation unfolds across turns.

1 Introduction

Conversational scams unfold as strategic, multi-turn processes in which scammers progressively manipulate victims before attempting to extract assets. PRESCAM addresses the gap between static or synthetic scam analysis and progression modeling using real-world conversations, structured stages, and turn-level psychological labels.

  • Motivation: Conversational scams progress from initial contact through sustained psychological manipulation toward attempts to extract money, credentials, or other assets.The progression is strategic rather than random and may involve authority, urgency, trust building, and fear induction.
  • Motivation: Existing fraud-analysis efforts mainly study static detection or synthetic simulations, which may miss the diversity and messiness of real-world scam reports.Synthetic conversations can reflect the generating model’s inductive biases rather than real-world variation.
  • Benchmark: PRESCAM transforms real-world scam reports into 11,573 structured multi-turn scam conversations and introduces the Scam Kill Chain to represent scams as staged processes.The representation is inspired by cybersecurity and cognitive science work on structured cyber attacks and social engineering.
  • Tasks: PRESCAM evaluates whether models can track evolving interaction risk and anticipate scammers’ next moves from partial context through two progression-modeling tasks.The tasks are real-time termination prediction and scammer action prediction.
  • Contributions: The benchmark combines structured stage annotations with turn-level psychological technique labels and evaluates language models and baselines on scam progression modeling.The study also quantitatively characterizes recurring stage-specific and scam-type-specific manipulation patterns.

2 Related Work

Prior online-scam research centers on automated detection and behavioral analysis, while most work targets static artifacts or post-hoc reports rather than dynamically unfolding multi-turn interactions. Recent conversational-scam datasets address this gap, but LLM-synthesized interactions may not reflect real-world complexity.

  • Research categories: Research on online scams generally falls into automated detection and behavioral analysis.These two categories organize the related work.
  • Detection: Automated detection spans phishing websites, smishing infrastructure, robocall content, and cryptocurrency-related scams across multiple modalities.Methods include CNN-based logo matching, knowledge graphs, and LLM-augmented reference-based systems.
  • Behavioral analysis: Behavioral studies examine victim susceptibility and persuasion, alongside scammer self-selection and social-engineering scripts.The literature considers both victim and scammer perspectives.
  • Limitations: These research lines predominantly target static webpages and messages or rely on post-hoc victim reports, limiting modeling of dynamic multi-turn scams.The limitation concerns scams that evolve through ongoing interactions.
  • Conversational scam datasets: Recent studies contribute scam-conversation datasets, but LLM-synthesized interactions often diverge from real-world scenarios and may miss rapid evolution and structural complexity.Synthetic data is presented as one response to the scarcity of conversational-scam research.

3 Preliminary

Prior cyber and social-engineering kill chains motivate viewing scams as evolving, staged psychological processes rather than isolated messages. The Scam Kill Chain formalizes this progression through three temporal phases and PT actions linking scammer utterances to psychological techniques.

  • Motivation: Cyber and social-engineering kill chains model attacks as sequential phases involving attacker tactics or staged psychological manipulation.Recent work operationalizes this perspective through explicit Psychological Techniques (PTs).
  • Motivation: Conversational scams progress from contact to trust-building and manipulation before attempting to extract money, sensitive information, or compliance.Different stages pursue different objectives and use distinct behavioral tactics and psychological techniques.
  • Scam Kill Chain: The Scam Kill Chain represents scam conversations as three phases: Initial Contact, Engagement, and Termination.Each phase contains one or more PT actions through which a scammer applies a specific psychological technique toward a phase-specific objective.
  • Scam Kill Chain: A PT action pairs an exploited psychological technique with the scammer utterance that operationalizes it within the multi-turn dialogue.The framework defines the technique set as P = {p1, p2,..., pK} and an action as at = (pt,ut).
  • Scam Kill Chain: The kill chain imposes a temporal partition on the conversation, concatenating its disjoint phases into an ordered sequence.It also maps each utterance to its corresponding underlying psychological technique to produce a structured action sequence.

4 PRESCAM Dataset

PRESCAM is built from real-world BBB Scam Tracker reports and filtered into 11,573 structured scam instances covering 20 categories. Each instance is organized into a three-stage scam lifecycle with multi-round actions, evidence spans, and psychological-technique annotations, followed by automated and human quality checks.

  • Dataset construction: 177,989 raw reports were filtered to 25,402 candidate multi-turn conversations, 13,007 conversations with at least two engagement rounds, and 11,573 final structured instances.The reports were collected between February 2024 and November 2025, and malformed or null entries were removed before finalization.
  • Dataset structure: Each instance uses three stages—Initial Contact, Engagement, and Termination—and decomposes Engagement into scammer and victim actions aligned with verbatim evidence spans and PT labels.Instances also include a scam category, victim narrative, binary scammed label, and scammed reason.
  • Dataset characteristics: 20 scam categories are represented in a pronounced long-tailed distribution, with structured conversations typically short and multi-round while original victim narratives are substantially longer and noisier.Additional category proportions, interaction-length statistics, and PT usage patterns are reported in Appendix B.
  • Quality control: An independent human study on n = 200 samples found mean quality scores of 7.10 before versus 7.37 after self-reflection, a +0.27 improvement out of 8.Three PhD-level reviewers assessed each sample using an 8-point rubric, while the self-reflection agent verified generated structures against a checklist.

5 Benchmark Task Formulation

PreScam formulates scam progression modeling as two complementary tasks over kill-chain-structured conversations: estimating proximity to termination and forecasting subsequent scammer actions. The tasks use continuous risk scoring for early intervention and dynamic, zero-shot action forecasting evaluated with judge-based and text-similarity metrics.

  • 5 Benchmark Task Formulation: The benchmark defines Real-time Termination Prediction and Scammer Action Prediction over kill-chain-structured conversations.These tasks respectively assess escalating-risk tracking and recovery of the future action–technique structure driving scam progression.
  • 5.1 Real-time Termination Prediction: Real-time Termination Prediction estimates whether a conversation is approaching the critical Termination phase early enough for intervention.Systems need not wait until the scammer explicitly requests payment before flagging danger.
  • 5.1 Real-time Termination Prediction: Given conversation history C≤t, the model outputs a continuous risk score ˆpt ∈[0,1], with ˆpt →1 indicating likely entry into Termination on the next turn.The task supports both Direct LLM Prompting and Supervised Sequence Classification.
  • 5.1 Real-time Termination Prediction: Termination prediction evaluates risk trajectories using AUC, AUPR, and Alert Time at a fixed α = 10% false-positive rate.AUPR addresses sparse positive turns, while AT@FPRα measures how many turns before termination the alert first occurs.
  • 5.2 Scammer Action Prediction: Scammer Action Prediction forecasts subsequent scammer actions from C≤t without imposing a globally fixed boundary turn.This dynamic formulation reflects ambiguity during Initial Contact and sometimes the first Engagement round.
  • 5.2 Scammer Action Prediction: The action task is evaluated zero-shot under Unlimited generation and Limited generation conditioned on the exact number of remaining scammer actions.The Limited setting uses nremain from the gold continuation after turn t.
  • 5.2 Scammer Action Prediction: Predicted actions are scored with Action HitRate, PT HitRate, BERTScore, ROUGE-L, and Precision after LLM-judge extraction of action and psychological-technique labels.Human review of 100 sampled cases covering 199 actions reached 92.0% action-level agreement with the judge.

6 Evaluation

The evaluation compares supervised and zero-shot models on real-time termination prediction and tests frontier LLMs on subsequent scammer-action prediction. Results show that models struggle more with temporally coherent progression and exact operational actions than with local cues or broad manipulative intent.

  • Real-time Termination Prediction: Real-time termination prediction compares classical, neural, and zero-shot LLM baselines, with imbalance mitigation applied to supervised models.The evaluated LLMs include GPT-4O-MINI, DEEPSEEK-V3, QWEN3-235B-A22, and GROK-4.1-FAST.
  • Scammer Action Prediction: Scammer action prediction asks models to generate structured subsequent actions from conversation prefixes using Action HitRate, PT HitRate, BERTScore, ROUGE-L, and Precision.The zero-shot LLMs include GPT-4O-MINI, GPT-5, CLAUDE-SONNET-4.5, DEEPSEEK-V3.2, and LLAMA-3.3-70B-INSTRUCT.
  • Scammer Action Prediction: 79.36 Action HitRate: CLAUDE-SONNET-4.5 achieves the highest overall score under the Unlimited setting.Action HitRate measures action recovery in the scammer-action prediction evaluation.
  • Scammer Action Prediction: 60.43 PT HitRate: DEEPSEEK-V3.2 achieves the best overall score under the Unlimited setting.The result indicates stronger performance for recovering the predicted technique-level progression under this metric.
  • Scammer Action Prediction: Restricting the output space further hurts action recovery, with the drop varying across scam categories.The recurring gap between Action HitRate and PT HitRate suggests that models often recover broad manipulative intent before the exact operational step.
  • Overall Findings: Across both tasks, models struggle to track risk buildup and action–technique progression, indicating a limitation in structural progression modeling rather than isolated cue recognition.The evaluation characterizes local cues and broad manipulative intent as easier to capture than temporally coherent progression.

7 Conclusion

The work reframes conversational scam understanding as progression modeling, showing that models still struggle to track escalation and recover scammers’ action structures in real conversations.

  • 7 Conclusion: PRESCAM introduces complementary tasks for predicting when conversations become dangerous and what scammers are likely to do next.This extends scam analysis beyond recognizing scam-related cues in static text.
  • 7 Conclusion: Strong language models can generate plausible continuations or capture parts of scam techniques, but progression modeling remains far from solved.Models still struggle to track escalation and recover the action structure of real scam conversations.
  • 7 Conclusion: PRESCAM aims to support progression-aware scam analysis, rigorous evaluation of interactive risk understanding, and more timely intervention in real-world settings.The benchmark is intended to help advance systems that understand how scam risk unfolds over conversations.

Ethics Statement · A Taxonomy of PTs

The paper uses publicly accessible scam reports solely for defensive academic research on scam understanding and progression modeling. Its scammer-action generations are restricted to controlled offline evaluation, while the paper also presents a taxonomy of psychological techniques (PTs).

  • Ethics Statement: Publicly accessible BBB Scam Tracker reports are used solely for academic research.The stated research focuses on scam understanding, progression modeling, and defensive evaluation.
  • Ethics Statement: The work aims to identify evolving scam risk and anticipate scammer behavior in realistic conversations.It explicitly distinguishes this goal from supporting real-world scam deployment or persuasive message generation.
  • Ethics Statement: The benchmark represents scammer actions and psychological techniques in structured form.These representations support modeling scam progression and scammer behavior.
  • Ethics Statement: Scammer-side continuations are generated only in a controlled offline research setting for evaluation.The restriction applies to the action-prediction experiments.
  • Ethics Statement: PreScam is intended to support defensive research, including risk forecasting, scam analysis, and safer detection and intervention systems.The stated purpose is defensive rather than operational scam deployment.
  • A Taxonomy of PTs: Table 3 presents a taxonomy of PTs.The supplied passage identifies the table but does not provide its taxonomy entries.

B Additional Dataset Statistics and Insights … D.3 Across-Reviewer Average

PRESCAM transforms noisy, real-world scam reports into 11,573 structured, multi-stage instances spanning 20 categories, while retaining evidential content for stage- and tactic-level analysis. The paper also documents reporting irregularities and evaluates generated kill chains through a blind, three-reviewer protocol with across-reviewer averages.

  • B.1 Construction Summary: 177,989 raw reports were filtered into 11,573 structured scam instances after identifying 25,402 candidate multi-turn conversations, removing short cases, and cleaning malformed entries.The reports were collected from BBB Scam Tracker between February 2024 and November 2025.
  • B.2 Instance Structure: Each instance contains seven components covering scam category, victim narrative, initial contact, engagement actions, termination, success status, and the reason for that status.The engagement sequence includes supporting verbatim spans and PT annotations.
  • B.2 Instance Structure: The structured design preserves evidence from original reports while supporting analysis at both scam-lifecycle stages and tactic levels.
  • B.3 Static Statistics: 20 scam categories show a pronounced long-tailed distribution, with Employment scams at 31.6% and Phishing at 12.3%.The remaining categories each account for much smaller shares.
  • B.3 Static Statistics: 3–15 engagement rounds occur per conversation, averaging 3.87, with most conversations concentrated between 3 and 5 rounds.Victim descriptions average 274.7 words, have a 208-word median, and reach outliers of 10,266 words.
  • B.3 Static Statistics: Category-level interaction length and PT usage vary substantially, highlighting heterogeneity in real-world scam execution strategies.
  • C Examples of Noisy and Unstructured User Reports: User-submitted reports commonly lack standardized structure and vary substantially in format and content, including retrospective single-round narratives, category-misaligned reports, and extremely short submissions.A single-round report may require interpreting an informal free-form paragraph, while a category-misaligned report may not describe a scam scenario at all.
  • C.3 Extremely Short and Underspecified Reports: Extremely short reports can omit conversation context, timelines, and explicit fraud mechanisms, limiting automated interpretation.The example consists of a single sentence without sufficient narrative context.

D.4 Reviewer-wise Detailed Results

Table 6 presents detailed human-evaluation results for all three reviewers, with 200 evaluations per reviewer scored on an 8-point scale.

  • Table 6 reports detailed results for all three human reviewers.
  • 200 evaluations were conducted for each reviewer.
  • Scores use an 8-point scale with a maximum of 8.

D.5 Validation of the LLM Judge … G Additional Scam Kill Chain Examples

The paper validates GPT-4O-MINI as an action-coverage judge and specifies prompting pipelines for extracting and structurally representing multi-round scam reports. These prompts distinguish interaction types, enforce kill-chain structure and psychological-technique annotations, and use reflection to correct extraction errors.

  • D.5 Validation of the LLM Judge: 92.0% action-level agreement and Cohen’s κ = 0.774 show substantial alignment between GPT-4O-MINI and human review on 100 sampled cases.The sample contains 199 total gold actions.
  • D.5 Validation of the LLM Judge: Agreement remains high across all gold-action-count buckets and is slightly stronger for cases with more gold actions.Table 8 also reports action-level confusion counts.
  • E Extraction Prompt: GPT-4O-MINI determines whether a raw BBB report contains usable multi-round interaction information.This is the Step 2 extraction decision.
  • E Extraction Prompt: The extraction prompt defines multi-round interaction as either back-and-forth chat transcripts or descriptions of multiple exchanges over time, while single-round reports lack follow-up exchanges.The model must return RESULT: [YES or NO] in an exact format.
  • F Structuralization Prompt: MINIMAX-2.5 converts each extracted report into a structured kill-chain instance and then applies reflection prompting to revise it against a checklist.The PT taxonomy and original BBB scam category are inserted through {pt_definitions} and {scam_type}.
  • F Structuralization Prompt: The structural output preserves chronological scammer and victim roles across initial_contact, engagement rounds, and outcome, with each engagement round requiring at least one psychological technique.Actions without psychological techniques are merged or moved rather than treated as separate engagement rounds.
  • F Structuralization Prompt: The reflection checklist reviews PT accuracy and completeness, round granularity, role separation, coverage, field placement, and exact verbatim extraction before returning corrected JSON.The prompt requires verbatim fields to copy exact text from the original report and permits merging, splitting, or relocating content.

H Evaluation Details · H.1 Real-time Termination Prediction Details · H.2 Scammer Action Prediction Details

The evaluation details define separate protocols for predicting termination and forecasting future scammer actions. They specify shared data construction, model settings, a dynamically selected scam-likelihood boundary, and evaluation regimes that distinguish action and psychological-technique recovery from surface similarity.

  • H.1 Real-time Termination Prediction Details: Termination prediction uses partial conversation prefixes labeled by whether the scammer’s next move enters the Termination stage, with an 80/20 scam-id-level train–test split.Inputs concatenate the Initial Contact summary with observed round-level scammer and victim actions.
  • H.1 Real-time Termination Prediction Details: The termination baselines include position-only logistic regression and TF-IDF logistic regression, with the latter using 50,000 features, unigram–bigram ranges, and C=1.0.The position-only model uses only the scalar round index and receives no hyperparameter tuning.
  • H.1 Real-time Termination Prediction Details: Neural termination models use standardized training procedures, while MLP-TF-IDF and MLP-Embed differ in whether they encode TF-IDF vectors or pooled token embeddings.MLP-TF-IDF uses 10,000 features; MLP-Embed uses a 20,000-word vocabulary, 256-token maximum length, and 128-dimensional embeddings.
  • H.1 Real-time Termination Prediction Details: BERT and RoBERTa use fine-tuned pretrained encoders, whereas zero-shot LLMs estimate termination risk directly from conversation histories with deterministic decoding and no task-specific training.The encoders use max-length 512 inputs and AdamW with lr=2e-5; zero-shot models return a JSON risk score between 0 and 1.
  • H.2 Scammer Action Prediction Details: Scammer action prediction selects a dynamic boundary because scam intent and conversation speed vary across cases, avoiding a globally fixed boundary turn.The Initial Contact stage and first Engagement round may remain ambiguous, motivating turn-specific boundary selection.
  • H.2 Scammer Action Prediction Details: 0.884 accuracy, 0.934 F1, and 0.881 ROC-AUC are achieved by the RoBERTa boundary classifier, with threshold 0.9 used conservatively in the main experiments.GPT-4O-MINI supplies turn-level scam-likelihood labels, and RoBERTa generalizes these labels to instantiate the boundary turn.
  • H.2 Scammer Action Prediction Details: Given the observed prefix, the action task forecasts the ordered future scammer-action sequence, potentially spanning multiple moves rather than only one next turn.Unlimited inference requires open-ended continuation and length planning, while Limited inference provides nremain and tests constrained structural recovery.
  • H.2 Scammer Action Prediction Details: Evaluation reports Action HitRate and PT HitRate alongside text-similarity metrics, using judge-based semantic coverage, precision, and psychological-technique extraction.The PT extraction judge selects only from the predefined technique names, including Pretext and Trust, Authority, and Urgency and Scarcity.
Loading 2605.12243v1…