AI-in-the-Loop: Privacy Preserving Real-Time Scam Detection and Conversational Scambaiting by Leveraging LLMs and Federated Learning

• Scams exploit personal information and psychological manipulation, while static-content and sender-based defenses offer little protection once adaptive dialogue begins.The introduction frames this gap as a need for proactive, context-sensitive, real-time defenses.Source: §1 · pp. 1–2
• The framework uses instruction-tuned LLMs to generate plausible victim-like replies, balancing scammer engagement against PII disclosure risk.The paper presents this as conversational scambaiting that can delay and disrupt scammer behavior and surface behavioral insights.Source: §1 · p. 2
• Federated learning supports continual improvement by training on local data and sharing updates rather than aggregating raw user data centrally.The described protocol uses encrypted weight updates and weighted averaging to compute a global model.Source: §1 · pp. 2–3
• Runtime monitoring flags interactions when a cumulative scam score exceeds a threshold, then consent enables AI intervention and utility-ranked replies.A hard safety threshold filters high-risk responses, while a secondary threshold governs continued engagement or disengagement as context evolves.Source: §1 · p. 2
• The introduction asks whether scams can be detected and prevented during live textual conversations while engaging scammers safely and preserving privacy.It also asks how scammers exploit user behavior to identify and target potential victims.Source: §1 · p. 3

• Detection methods progressed from blacklists and rule-based systems to supervised and deep-learning models, with early multimodal systems combining lexical, DNS, and visual features.The review notes extensive feature engineering and limited adaptability as challenges for those early multimodal systems.Source: §2 · p. 3
• LLM-based approaches include zero-shot scam-site classification and visual-cue detection, while live-call systems and RAG-based impersonation checks report 92% and 98% accuracy, respectively.The cited live-call system warns users mid-conversation; the RAG-based system integrates policy knowledge.Source: §2 · p. 3
• Automated conversational scam-baiters have been reported to increase scammer engagement and prolong conversations over a month-long study.The review describes scam-baiting as deliberately interacting with scammers to waste time, reveal tactics, or gather intelligence.Source: §2 · p. 4
• Federated learning aggregates knowledge from devices without sharing sensitive data with a central server, and FedAvg gives larger datasets greater influence on the global model.The review connects federated learning with adapting scam detection across regions and to changing scam strategies.Source: §2 · p. 4

3 System Design

• The architecture combines real-time scam detection, scambaiting, safety-aware response filtering, and federated learning for privacy-preserving adaptation.Together, these components are presented as a proactive defense for live messaging platforms.Source: §3 · p. 4, Federated Model Updates · p. 7

3.1 Threat Model

• Scam risk is scored over conversation messages and triggers active defense at a threshold; the system also models gradient leakage as a threat and applies differential privacy to updates.The described DLG and iDLG attacks reconstruct user data from gradients; calibrated noise is applied to updates as mitigation.Source: §3.1 · pp. 4–5

3.2 Privacy Goals

• The privacy goals combine PII filtering in generated responses, on-device learning with protected gradient updates, and behavioral constraints against toxic or scam-reinforcing replies.The stated protections include named-entity recognition and context-aware masking, noise-perturbed updates, and secure aggregation.Source: §3.2 · pp. 5–6

3.3 Problem Formulation and System Overview

• The workflow fine-tunes models for scam classification and response generation, then activates controlled response generation when a conversation is flagged.Conversation messages are scored using unweighted accumulation or an exponentially weighted moving average, with recent messages prioritized by the latter.Source: §3.3 · p. 6

Scambaiting Activation and Response Generation.

• After flagging, the system selects from top-k generated replies using a utility function that weighs scammer engagement, information risk, and harm reduction.The candidate pool is evaluated for whether a reply sustains interaction, leaks PII or escalates threat, and distracts, confuses, or stalls the scammer.Source: Scambaiting Activation and Response… · p. 7

Ongoing Monitoring and Risk Adaptation.

• During engagement, changes in scam score can prompt the user to terminate, continue, or report the interaction.The prompt is triggered if the score falls below θ2 or exceeds θ1, guarding against over-engagement while allowing strategic stalling.Source: Ongoing Monitoring and Risk Adaptation · p. 7

3.4 Response Utility Function f(gi)

• The Response Utility Function scores candidate replies to maximize engagement while minimizing harm, using a nonlinear rationale that gives diminishing returns to engagement and increasingly penalizes harm.Engagement quality measures whether the scammer continues interacting; harm score reflects risks such as sensitive-information disclosure or reinforcing the scam narrative.Source: §3.4 · p. 7

3.5 Federated Learning for Adaptive Improvement

• A harm threshold discards unsafe candidates before utility ranking, regardless of their engagement value.Federated training keeps client datasets on-device, accounts for non-IID client data, and aggregates updates; optional differential privacy can clip and noise updates.Source: §3.4 · p. 8, §3.5 · p. 8

4.1 Datasets

• The study uses labeled synthetic scam and legitimate dialogues for classification and seven primary or supplementary conversation datasets for generation, with roles normalized to scammer and baiter.Classification covers MASC, SASC, SSC, and SSD; generation evaluation uses five candidate replies per potential scammer turn and a safety threshold, on data not used for generation fine-tuning. The moderation-risk evaluation samples 1,200 conversations and records moderation categories plus scam, engagement, and PII scores.Source: §4.1.1 · p. 9, §4.1.2 · p. 9, §4.2.4 · p. 12

4.2.1 Baseline Model Performance Comparison

• BiGRU and BiLSTM achieve F1 ≥0.9889, FPR ≤0.0033, and FNR ≤0.0075 across four datasets, outperforming transformer baselines for conversation-level scam detection.RoBERTa is the strongest transformer, with F1 ≥0.9901 and AUPRC ≥0.9881; SSD is easiest to classify, while DistilBERT has the lowest transformer performance (F1 > 0.9650). The authors attribute RNN performance to modeling temporal dependencies and conversational flow.Source: §4.2.1 · p. 10, Table 1 · p. 11

4.2.2 Performance of Instruction-Tuned LLMs for Scam Detection

• MD-Judge achieves the highest F1 and AUPRC across datasets, including F1 = 0.8985 and AUPRC = 0.9320 on SSD, while LlamaGuard2 reaches perfect scores on SSC.LlamaGuard2’s SSC results are F1 = 1.0, AUPRC = 1.0, and FPR = FNR = 0.0; LlamaGuard consistently underperforms the later instruction-tuned variants.Source: §4.2.2 · pp. 10–11, Table 2 · p. 11

4.2.4 Scam-Baiting Response Generation Performance

• On generation evaluation, SSC yields perplexity 22.3, Dist-1 = 0.69, Dist-2 = 0.54, and DialogRPT 0.80, while SSD responses are more repetitive.Responses with Harm/PII risk scores above 0.4 were filtered before evaluation. Separately, PII scoring assigns 0.8–1.0 to highly sensitive types and about 0.4–0.6 to less sensitive items, but the scores were not validated with human annotation.Source: §4.2.3 · pp. 11–12, Table 3 · p. 12, §4.2.4 · p. 12

4.2.5 Human Evaluation of Scam-Baiting Quality

• In human evaluation of 100 conversations, fine-tuned MD-Judge significantly outperformed fine-tuned LlamaGuard3 across realism, engagement, safety, and effectiveness, scoring 4.31 ± 0.52 versus 3.92 ± 0.61 on realism.Engagement rose from 3.31 ± 0.65 to 4.05 ± 0.60, safety from 92.0% to 96.0%, and effectiveness from 3.43 ± 0.57 to 4.12 ± 0.55.Source: Table 4 · p. 13, §4.2.5 · p. 13
• Across 100 conversations, the AI baiter’s perplexity is typically 15–60 versus reference-baiter spikes above 175, and its DialogRPT scores are more concentrated while the reference has a higher upper tail.The reference baiter’s DialogRPT distribution is broader and shifted toward 0.5–0.8, indicating occasional higher engagement but less predictable scores.Source: Figure 5 · p. 14, §4.2.5 · p. 14, Figure 6 · p. 14

4.2.6 Response Time Consistency in Scam-Baiting

• Across 254 scam interactions, AI response times are typically 0.45–0.55 seconds, compared with reference-system variation from 0.1 to 0.9 seconds.The authors present this stability as supporting natural, real-time engagement without awkward delays or suspicious latency.Source: §4.2.6 · p. 14, Figure 7 · p. 15

4.2.7 Federated Learning Evaluation for Generation Tasks

• Under FedAvg, 0.1-DP (noise multiplier 0.1) reduces engagement by ≤0.5% and scam risk from 0.54 to 0.50; 0.8-DP modestly reduces utility, while PII risk remains consistently low across settings.The global model consistently improves or stabilizes across rounds, regardless of privacy configuration.Source: §4.2.7 · p. 15

4.2.8 Safeness and Risk Awareness Evaluation

• LlamaGuard3 combines engagement (0.97) and scam detection (0.95+) with moderate PII scores in high-risk scenarios.The evaluation reports that LlamaGuard and MD-Judge also distinguish safe from unsafe content, while LlamaGuard2 and LlamaGuard3 assign high engagement and PII risk to some unsafe cases.Source: §4.2.8 · p. 16
• The multidimensional protocol links scam, engagement, and PII-exposure scores to moderation decisions, helping identify model blind spots and over- or under-sensitivity.This diagnostic approach is intended to reveal responses to subtle manipulation tactics and inform safeguards for automated scam prevention.Source: §4.2.8 · p. 16

Discussion, Limitations, and Future Directions

• The system jointly optimizes detection, risk scoring, and response generation under a privacy-weighted utility function and strong safety constraints.The current work focuses on text-based scams; extending the architecture to voice through TTS, ASR, and speaker anonymization would introduce additional latency and detection challenges.Source: §5 · p. 16
• The benchmarks report superior detection accuracy, engagement stability, and unique multitask capability against standard classifiers, scam-baiting systems, and instruction-tuned LLMs.The passage does not provide numerical comparisons.Source: §5 · p. 17
• The federated setup uses IID and non-IID simulated clients with differential privacy, and experiments report detection improving over time with local fine-tuning and AI-driven engagement.Secure aggregation and personalization are identified as possible further techniques, not evaluated results in this passage.Source: §5 · p. 17
• Dynamic scoring of both participants activates the AI for the higher-risk side and reduces misactivation, with the false positive rate below 20% using LLM–MD-Judge.Users can disable AI interaction, and warnings, prompt filtering, and adaptive harm thresholding are also described.Source: §5 · p. 17
• Across 100 multi-turn scam conversations, most median BERTScore F1 values fall between 0.70 and 0.78, indicating semantic alignment with reference baiter responses.Some lower scores around 0.55–0.60 are described as reflecting intentional strategic divergence; complex or variable prompts also show broader variance.Source: §5 · p. 18, Figure 8 · p. 18

Conclusion

• The framework combines real-time scam detection and automated scam-baiting in a single instruction-tuned LLM, with adaptation through federated learning and differential privacy.Its utility-based selection and dynamic harm threshold are described as balancing engagement and safety.Source: §6 · p. 18
• Automatic-metric evaluations report improvements over baseline LLMs in realism, engagement, safety, and effectiveness while minimizing harm risk.The conclusion also reports federated experiments with local adaptation and secure aggregation for continuous improvement without centralizing sensitive data.Source: §6 · p. 18
• The current work focuses on text-based scams, with future work targeting voice-based detection, multimodal signals, adaptive adversary simulation, and large-scale deployment evaluations.Source: §6 · p. 18

Ethical Considerations and Data Privacy

• The study relies on anonymized public data and synthetic interactions, with usernames, locations, and other PII removed or excluded before analysis.Data are stored using unique identifiers unlinked to PII.Source: §7 · p. 18
• The authors state that their practices are consistent with NIST SP 800-122 and GDPR and are intended to mitigate re-identification risks.Source: §7 · p. 18
• The research is described as prioritizing transparency, fairness, and accountability while aiming to minimize potential harm.Source: §7 · p. 18

A Datasets Details

• SSD, SSC, SASC, and MASC provide labeled scam and non-scam conversations for classification, with synthetic dialogues generated using different systems and, for MASC, eight receiver personalities.SSD includes scam categories such as SSN, refund, technical support, and reward, alongside legitimate-call types.Source: A Datasets Details · p. 24

Generation Task

• YTSC, SBC, and ASB provide scam-baiting conversations, including transcribed YouTube scams, deployed email exchanges, and public advance-fee scam-baiting transcripts.SBC contains 254 valid conversations with at least one scammer reply; ASB contains 57 complete exchanges totaling 2,248 messages.Source: Generation Task · p. 24

Data Analysis and Preprocessing

• Tables 8 and 9 are described as reporting dataset statistics and scam-type distributions for the classification, generation, and synthesized instruction-tuning datasets.Source: Data Analysis and Preprocessing · p. 24, Table 8 · p. 25, Table 9 · p. 25

Synthetic Dataset Generation Prompt

• For multi-task instruction tuning, the prompt specifies 500 conversations for each of 11 scam types, yielding 6,000 scam-related samples, plus 1,000 general conversations yielding 3,000 additional samples.Each conversation generates PII Evaluation, Scam Baiting Response Generation, and Scam Risk Scoring samples; general conversations have no PII, zero PII risk, and scam scores close to zero.Source: Synthetic Dataset Generation Prompt · p. 25
• Table 10 is captioned as reporting average engagement, PII risk, and scam risk scores by conversation type in the synthesized dataset.Source: Table 10 · p. 25

B Proofs of Theorems

• Theorem 1 models scam-like continuation probability as inversely related to response utility, while Lemma 1 says high engagement without harm minimization leaves utility low and continuation probability high.The section’s assumption links next-message scam likelihood inversely to the current response’s utility score.Source: B Proofs of Theorems · p. 26
• High-engagement, low-harm replies can maximize utility and frustrate scammers, whereas high harm can dominate engagement and incentivize persistence or escalation.The canonical examples contrast a scammer kept explaining a fictitious prize with a target sharing sensitive details during a romance scam.Source: B Proofs of Theorems · p. 26

Justification for the Subtraction-Based Utility Function

• The subtraction-based utility scores candidate replies by rewarding engagement and penalizing harm, following a benefit-minus-risk formulation [Howard(1970), Russell and Norvig(1995)].Its logarithmic engagement term models diminishing returns, while the quadratic harm term increasingly penalizes rising risk [García and Fernández(2015), Moldovan and Abbeel(2012)].Source: Justification for the… · p. 27
• The tunable weights α and γ balance engagement and safety; the proposed zero-centered scale treats f(gi) > 0 as acceptable and f(gi) ≤ 0 as high risk or disengagement.The authors describe the formulation as interpretable, computationally efficient, and policy-flexible for real-time response scoring.Source: Justification for the… · p. 27

Illustrative Cases for the Nonlinear Utility Function

• E(gi) = 0.9 and H(gi) = 0.1 describe a high-engagement, very-low-harm response that maintains scammer interest without revealing personal information.The example is interpreted as highly effective and safe.Source: Illustrative Cases for the Nonlinear… · p. 28
• E(gi) = 0.9 and H(gi) = 0.5 illustrate that high engagement can still carry considerable privacy risk when a reply implies sensitive-data disclosure.The example’s interpretation is that privacy risk overrides high engagement.Source: Illustrative Cases for the Nonlinear… · p. 28
• E(gi) = 0.2 and H(gi) = 0.8 make the example catastrophic, and its response is deemed unacceptable under any scoring policy.By contrast, a safe response with E(gi) = 0.1 and H(gi) = 0.1 is ineffective at distracting the scammer.Source: Illustrative Cases for the Nonlinear… · p. 28
• 0.596 is the utility for α = 2.0, γ = 1.0 in safe contexts, while mean = +0.087 and median = +0.076 preserve positive utility for average behavior.The comparison reports 0.149 for lower α = 0.5 in a high-engagement, low-harm scenario, and −0.450 for α = 5.0, γ = 5.0 in median scenarios.Source: Illustrative Cases for the Nonlinear… · p. 29, Figure 9 · p. 29
• Only a limited number of the 5,000 sampled utility scores exceed the 0.44 threshold derived from harm cutoff δ = 0.4.For α = 2.0 and γ = 1.0, mean utility is approximately 0.00 and median utility is 0.05; most values cluster between −0.5 and 0.4.Source: Illustrative Cases for the Nonlinear… · p. 29

C Language Models Overview

• LlamaGuard-7b is a 7-billion-parameter LLaMA 2-based model that classifies prompts and responses as safe or unsafe and labels violations using Meta’s safety taxonomy.The passage describes it as widely used for moderation of LLM-generated content.Source: C Language Models Overview · pp. 29–30
• Table 13 presents utility scores for different α and γ values.Source: Table 13 · p. 30
• Meta-Llama-Guard-2-8B is described as an 8-billion-parameter LLaMA 3-family successor, while Llama-Guard-3-8B supports multilingual moderation across eight languages.MD-Judge-v0.1 is described as a 7B-parameter Mistral-based classifier for evaluating LLM-generated responses.Source: C Language Models Overview · pp. 30–31
• Figure 10 plots the utility-score distribution for 5,000 random engagement–harm pairs using α = 2.0 and γ = 1.0, with mean, median, and harm-cutoff threshold lines.The figure illustrates the utility function’s engagement–harm balance.Source: Figure 10 · p. 31

Experimental Setup

• Four open-source safety models receive multi-task tuning, alongside BERT, RoBERTa, DistilBERT, BiLSTM, and BiGRU baselines; the authors note they did not try GPT-3.5 because it is not open source.The tuning includes classification, engagement and PII scoring, and safe response generation, with LoRA and stated training and decoding settings.Source: Experimental Setup · pp. 31–32
• FedAvg aggregates client-side model updates without collecting raw data, and the simulated setup uses 10 clients over 30 rounds; its DP variant applies noise multiplier σ = 0.1 and tracks ε per round.The clients train LoRA-adapted models for three local epochs; the DP and standard setups use different batch sizes and quantization settings.Source: Experimental Setup · p. 32

Instructions Tuning

• The authors select α = 2.0 and γ = 1.0 using a grid search and 1,000 held-out conversations, and set θ1 = θ2 = 0.7 and δ = 0.4.Prompts separately assess agent-message safety, dialogue scam risk, engagement and PII, and safe scam-baiter response generation.Source: Instructions Tuning · p. 32

Evaluation

• Classification evaluates whole conversations and individual user messages, while generation selects the highest-utility response from five candidates; the stated metrics cover classification, safety, relevance, fluency, diversity, engagement, and federated output quality.The passage identifies F1, AUPRC, FPR, FNR, HarmBench, ROUGE-L, BERTScore, perplexity, Distinct-n, DialogRPT, and federated relevance, conciseness, and clarity measures.Source: Evaluation · p. 34

Evaluation on Engagement and PII

• The engagement-and-PII evaluator scores only the Agent’s response and reports engagement, PII risk, PII presence, and PII type; the example assigns an account-number disclosure PII Risk Score of 0.87.Source: Evaluation on Engagement and PII · pp. 34–35

Evalutaion on Scam Detection

• The scam-detection prompt returns a single 0.0–1.0 risk score, and the supplied Social Security impersonation dialogue receives a score of 0.91.Source: Evalutaion on Scam Detection · p. 35

Evaluation on Safety

• Safety guidelines prohibit sharing user data or impersonating sensitive actors, while allowing generic responses and requiring calm, non-escalatory time-wasting.The listed constraints also prohibit impersonating real individuals or authorities and requesting non-public personal data.Source: Evaluation on Safety · p. 36

LlamaGuard

• 0.0017 engagement, 0.0019 PII risk, and 0.0021 scam detection characterize safe SSC responses; unsafe_15 averages 36.40 engagement and 44.95 PII risk.The comparison summarizes moderation-label groups in Table 14.Source: Table 14 · p. 37, Results · p. 37

LlamaGuard2

• 0.18 engagement, 0.18 PII risk, and 0.03 scam detection mark safe SSD responses; unsafe categories often exceed 0.75 engagement and reach or near 1.0 scam detection.In SSC, safe outputs show a higher engagement score of 0.45, which the authors associate with dataset-specific distributions or ambiguous cases.Source: Table 15 · p. 38, Results · pp. 38–39

LlamaGuard3

• 1.0 scam detection scores occur for unsafe_14 in every case; MASC safe content scores 0.11, while unsafe_14 exceeds 0.90 on engagement and PII risk.Safe segments also have low engagement and PII-risk scores across the evaluated datasets.Source: Results · pp. 37–38, Table 16 · p. 38

Novelty

• Novelty measures lexical difference from the scammer’s preceding message using overlap fraction and Jaccard similarity; higher values indicate more new wording rather than repetition.The definition uses one minus the average of the two similarity measures; scores approach 1 for new words and 0 for mostly repeated wording.Source: E Evaluation Metrics · p. 39

Engagement

• Engagement combines lexical diversity, normalized response length, and a small bonus for questions to estimate how well a response sustains conversation.The length component penalizes responses that are too short or excessively long.Source: E Evaluation Metrics · p. 39

Relevance

• Relevance measures the semantic similarity between the scammer’s message and AI response using Sentence-BERT embeddings and cosine similarity.The score is normalized from [−1, 1] to [0, 1], with higher values indicating greater semantic relatedness.Source: E Evaluation Metrics · p. 40