Source-linked AI summary
Incremental Risk Assessment of Progressive Elder Financial Scams via Instruction-Tuned Small Language Models
Parviz Ghafariasl, Weimin Fu, Xiaolong Guo, Shing I. Chang
TL;DR
The study treats elderly fraud detection as incremental risk estimation over cumulative conversational context, addressing gradual risk emergence beyond static classification. It combines structured synthetic data with instruction-tuned small language models, finding that instruction adherence and output-format stability matter more than parameter scale alone for reliable multi-turn scoring.
Problem
Gradual risk emergence across conversational turns motivates moving beyond binary fraud classification toward finer-grained incremental risk estimation.
Source: B. Risk Class Definition and Numerical… · p. 6, §V · p. 10Method
The framework uses structured synthetic two-party dialogues, continuous 0–10 risk scoring mapped to semantic categories, and instruction-tuned small models for cumulative conversational assessment.
Source: A. Synthetic Data Collection and… · p. 4, B. Risk Class Definition and Numerical… · p. 6, §V · p. 10Results
LLaMA-3.2-1B achieves the lowest Mean Absolute Error (MAE = 1.56), while Qwen-1.5B records MAE = 5.58 and Qwen3-1.7B lacks sufficient valid predictions for quantitative analysis.
Source: §IV · p. 8Takeaways & Limitations
Reliable multi-turn risk scoring depends more on instruction adherence and output-format stability than parameter scale alone.
Source: §IV · p. 8, §V · p. 10The framework is evaluated under controlled synthetic, primarily text-based settings and assumes relatively coherent conversational progression, limiting coverage of real-world multimodal and disrupted interactions.
Source: A. Limitations and Real-World Deployment · p. 10
Abstract
from arXiv · showhide
Financial scams targeting older adults increasingly occur through text and voice channels such as email, SMS, and phone calls, unfolding over multiple conversational turns that begin with impersonation or casual contact, escalate through trust building and urgency, and culminate in requests for sensitive information or financial transfers. Because risk signals emerge incrementally across turns, effective detection requires models that continuously update risk estimates under resource-constrained deployment settings. We propose a cumulative turn-based risk assessment framework that incrementally aggregates conversational turns and re-estimates risk at each step, enabling dynamic scam monitoring across progressively evolving conversations. A multi-turn dialogue dataset is constructed to cover investment, charity, and tech support scam scenarios, with each dialogue containing two to eight turns and annotated at every cumulative stage with a qualitative risk level, a continuous risk score, an explanatory rationale, and a safety recommendation. Four small language models (Phi-4, LLaMA-3.2, DeepSeek-R1, and Qwen3) are fine-tuned and evaluated under a unified training framework. Fine-tuned small models capture fraud-related linguistic cues and cross-turn escalation patterns while maintaining compact architectures suitable for mobile and resource-constrained deployment settings. Among the evaluated models, Phi-4 and LLaMA-3.2 achieve stronger turn-aware risk estimation performance relative to their parameter scale. These results suggest that structured cumulative modeling can support incremental scam risk assessment in deployment-oriented settings while highlighting the potential of compact language models for privacy-aware and on-device fraud protection.
A. Synthetic Data Collection and Annotation Pipeline: Cumulative Turn-Based Data Preparation
The study constructs a controlled, balanced synthetic dataset for incremental elder-scam risk inference, using cumulative turn representations and turn-level supervision across varied fraud scenarios and channels.
Dataset Construction
- The dataset contains 600 training dialogues and 40 held-out test dialogues, each structured as a two-party scammer–victim interaction spanning two to eight turns.A turn is defined as one message paired with its corresponding reply.Source: A. Synthetic Data Collection and… · p. 4
- Scenarios are balanced across investment and charity fraud and tech support fraud, with approximately one hundred distinct scenarios defined within each domain.The scenarios span fine-grained fraud strategies and interaction patterns.Source: A. Synthetic Data Collection and… · p. 4
- Dialogues vary across email, SMS, and phone-call channels to represent differences in tone, urgency cues, and linguistic style.Source: A. Synthetic Data Collection and… · p. 5
- Paired legitimate and scam versions are generated under comparable structural conditions to encourage intent- and behavior-level distinctions rather than superficial lexical cues.The domains include account updates, transaction confirmations, security alerts, compromise claims, and device-protection warnings.Source: A. Synthetic Data Collection and… · p. 5
Annotation
- Each turn receives a binary scam indicator, ordinal risk category, and continuous 0–10 risk score through automated annotation with manual review of a subset.The annotation supports incremental, real-time monitoring.Source: A. Synthetic Data Collection and… · p. 5
Cumulative Turn-Based Preparation
- Training samples incrementally aggregate turns from the initial exchange through the full dialogue, converting each interaction into a temporally ordered sequence.This formulation supports analysis of early warning detection, delayed recognition, and uncertainty calibration.Source: A. Synthetic Data Collection and… · p. 5
Limitations
- Synthetic dialogues may not fully represent the linguistic diversity and behavioral complexity of real-world scam interactions, despite separate synthesis and annotation models and manual review.The stated objective is controlled exposure to representative scam patterns under privacy-preserving conditions, with future extension to real-world annotated data.Source: A. Synthetic Data Collection and… · p. 6
B. Risk Class Definition and Numerical Scoring Scheme
The framework represents scam risk continuously from benign interaction to explicit imminent intent, while also mapping scores to semantic classes for tolerance-aware evaluation and calibration analysis.
Continuous Risk Scale
- Risk is scored continuously from 0 to 10, where 0 denotes a fully benign interaction and 10 indicates explicit and imminent scam intent.Source: B. Risk Class Definition and Numerical… · p. 6
- The continuous formulation captures degrees of suspicion and escalation intensity across conversational turns beyond binary fraud classification.Source: B. Risk Class Definition and Numerical… · p. 6
Semantic Risk Classes
- Numerical scores are mapped to discrete semantic risk categories, supporting both regression-style estimation and class-based analysis.The correspondence between score ranges and semantic classes is summarized in Table I.Source: B. Risk Class Definition and Numerical… · p. 6, Table I · p. 6
Tolerance-Aware Evaluation
- Adjacent scores are treated as semantically similar for tolerance-aware evaluation, with deviations of ±1 considered near-consistent and larger deviations indicating substantive disagreement.This perspective addresses uncertainty in early-stage conversational assessment.Source: B. Risk Class Definition and Numerical… · pp. 6–7
C. Methods and Models
The study fine-tunes compact language models on cumulative conversational risk-assessment data and compares their structured scoring behavior. LLaMA-3.2-1B provides the most stable performance, while Qwen-based models show weaker robustness and Phi-4 exhibits conservative overestimation.
Evaluation Setup
- Cumulative dialogue instances are used to evaluate risk estimation as conversational context expands.The pipeline and model comparison focus on fine-tuned small language models for structured elder-fraud risk assessment.Source: C. Methods and Models · p. 7, Figure 2 · p. 7
Model Selection
- Four small language-model architectures are compared under similar 1–4B parameter constraints, including instruction-aligned, dialogue-oriented, and reasoning-distilled designs.The models are Phi-4-mini-instruct, LLaMA-3.2-1B, Qwen3-1.7B, and DeepSeek-R1-Distill-Qwen-1.5B.Source: C. Methods and Models · p. 7
Model Comparison
- LLaMA-3.2-1B achieves the lowest reported risk-score error, with MAE = 1.56, while Phi-4-mini-instruct records MAE = 2.06.LLaMA-3.2-1B also maintains valid structured outputs and ordinal risk progression across turns.Source: §IV · p. 8
- Qwen-based models show lower robustness: Qwen-1.5B reaches MAE = 5.58, while Qwen3-1.7B has insufficient valid predictions for quantitative analysis.The reported issues include inconsistent format compliance and non-parseable outputs.Source: §IV · p. 8
Error Analysis
- Phi-4 overestimates low-risk cases, whereas LLaMA-3.2-1B usually keeps errors within ±1–2 risk-score points and preserves ordinal progression.Phi-4 valid format-compliant outputs were obtained for 35 of 40 evaluation cases; five were excluded from deviation analysis.Source: §IV · pp. 8–10
V. Conclusion and Future Work
The study frames elder-fraud detection as incremental risk estimation over cumulative conversational context. Its results indicate that structured output adherence and format stability matter more than parameter scale alone for reliable multi-turn scoring.
Conclusion
- Cumulative conversational modeling enables temporally aware, machine-parseable elder-fraud risk assessment rather than static classification.The framework combines structured synthetic data with instruction-tuned small language models.Source: §V · p. 10
- Instruction adherence and output-format stability are more critical than parameter scale alone for reliable multi-turn risk scoring.Source: §V · p. 10
A. Limitations and Real-World Deployment
The framework shows promise in controlled synthetic settings but remains bounded by text-only inputs, synthetic-data realism, coherence assumptions, and deployment challenges. Future work targets real-world, multilingual, multimodal, and privacy-preserving evaluation and deployment.
Current limitations
- The study is evaluated under controlled synthetic settings, limiting evidence about generalization to real-world scam behavior.Synthetic dialogues may not capture the linguistic diversity, unpredictability, and evolving strategies of scams across communication platforms.Source: A. Limitations and Real-World Deployment · p. 10
- The framework focuses on text-based conversations and omits voice, emotional, and behavioral signals that could improve detection reliability.Source: A. Limitations and Real-World Deployment · p. 10
- The cumulative formulation assumes coherent conversational progression, unlike interactions with interruptions, topic shifts, incomplete exchanges, or adversarial manipulation.Source: A. Limitations and Real-World Deployment · p. 10
Real-world deployment
- Real-world deployment still requires privacy preservation, continual adaptation, and robustness to dynamically evolving fraud patterns.Source: A. Limitations and Real-World Deployment · p. 10
Future work
- Future work will evaluate real-world annotated data, extend multilingual and multimodal analysis, and investigate privacy-preserving deployment strategies.Proposed strategies include on-device inference and federated adaptation.Source: A. Limitations and Real-World Deployment · p. 10