PaperlayerView exact revision on arXiv

Source-linked AI summary

ORACLE: Anticipating Scams from Partial Trajectories in Streaming App Usage

Wenbo Gao, Songbai Tan, Zhongan Wang, Fei Shen, Gang Xu, Huiping Zhuang, Yunyun Yang, Ming Li, Xiaofeng Zhu

arXiv:2605.16363v1cs.LGcs.CY

TL;DR

Smartphone scams unfold across applications and time, so early intervention requires reasoning from partial trajectories before intent becomes explicit. ORACLE addresses this gap with a long-horizon benchmark, evolving context management, and on-policy self-distillation. It improves early and reliable warnings while reducing false alerts in streaming cross-app evaluation.

  • Problem

    Scams reveal intent through temporally distributed, cross-application cues, making early anticipation from partial trajectories difficult.

  • Method

    ORACLE combines a long-horizon benchmark with a self-evolving context manager and on-policy self-distillation using reflection-informed teacher supervision.

  • Results

    ORACLE improves PAR from 77.3 to 98.2, reduces EDP from 46.4 to 29.5 and FAR from 12.8 to 1.3 versus GPT-5.1 in streaming cross-app evaluation.

  • Takeaways & Limitations

    The results support cross-temporal reasoning over partial behavioral trajectories as a basis for earlier, more reliable scam warnings with fewer false alerts.

  • Takeaways & Limitations

    The curated benchmark may not fully capture the complexity of real-world scam interactions because large-scale, fully observed victim-side histories are difficult to collect.

Abstract

from arXiv · show

Smartphone scams are increasingly prevalent and typically manifest as multi-stage, cross-application processes with gradually emerging intent. Effective intervention thus requires anticipating scams before the intent becomes explicit. This is inherently challenging, as decisions must rely on partial trajectories with temporally distributed evidence. In this paper, we propose \textbf{ORACLE} Online Reasoning for Anticipating Cross-temporal Latent thrEats, the first agentic framework for early scam anticipation from \textit{streaming app-usage} trajectories. To support this setting, we curate a real-world long-horizon benchmark of streaming app-usage trajectories, covering 12 scam types, spanning extended periods (15 days on average), involving diverse applications (95 apps), and interleaving normal and scam behaviors. To address fragmented evidence, we introduce a self-evolving context manager that adaptively consolidates entity-centric interactions over time, enabling more effective reconstruction of cross-temporal evidence from partial observations. To enhance sensitivity to latent early-stage signals, we propose an on-policy self-distillation scheme in which a teacher model, conditioned on summarized anti-scam reflections and clues by skills, supervises a student model without access to such reflections. This scheme thereby distills evidence-informed knowledge and improves recognition of emerging fraud patterns from partial trajectories. Experiments show that \method{} consistently improves early scam anticipation, yielding timely warnings while reducing false alerts in realistic streaming scenarios.

1 Introduction

ORACLE frames early scam intervention as reasoning over partial, streaming app-usage trajectories whose evidence is distributed across applications and time. It combines a long-horizon benchmark with evolving historical context and self-distillation to anticipate emerging fraud earlier.

  • Smartphone scams span communication, social, and financial applications, creating risks to daily life and financial security.
  • Scams typically unfold through multi-stage cross-application processes in which weak behavioral cues gradually reveal intent.
  • Early anticipation is difficult because streaming observations contain fragmented context and latent signals embedded within apparently normal behavior.
  • The benchmark contains 57,662 short traces aggregated into 3,061 long trajectories averaging 15 days and 96 app events across 95 apps and 12 scam types.
  • ORACLE uses a self-evolving context manager to reconstruct dispersed evidence and on-policy self-distillation to transfer reflection-informed early-scam knowledge to a student model.

2 Streaming Scam Benchmark

The benchmark transforms short scam cases and normal app logs into long-horizon streaming trajectories, then evaluates detection coverage, false alerts, and early-warning timing. Its formulation treats each partial observation window as a streaming classification decision.

  • Dataset curation: The curation pipeline converts short scam cases and normal app-usage logs into long-horizon streaming trajectories.
  • Dataset curation: Short traces combine victim-reported scam descriptions from CCL2023 with fine-grained normal app-usage logs from LSApp.
  • Dataset curation: Summarized LOCOMO dialogue content augments traces with coarse screen-level information without requiring full private conversations.
  • Dataset curation: Scam segments are sparsely interleaved with normal activity, and LLM judges inspect leakage, behavioral inconsistencies, and implausible app descriptions.
  • Evaluation metrics: Hit Rate measures trajectory-level coverage, False Alert Rate measures spurious alerts, Earliest Detection Position measures first-alert timing, and Pre-alert Rate measures early-warning consistency.
  • Streaming formulation: At each time step, the system observes a recent trajectory window and predicts whether the current behavior is NORMAL or SCAM.

3 ORACLE

ORACLE performs early scam anticipation by combining streaming-window analysis with retrieved cross-temporal context and an evolving memory-and-skill system. Its on-policy self-distillation trains the assessor to recognize latent scam cues from partial augmented windows.

  • System Overview: ORACLE parses raw events, stores entity-centered history, retrieves relevant interactions, and uses an assessor to produce scam-risk judgments.The deployment pipeline comprises a screen analyzer, person-centric memory store, skill-guided context manager, and scam risk assessor.
  • Problem Formulation: Early scam anticipation is formulated as streaming classification over recent windows augmented with retrieved history, with labels NORMAL, RISKY, or SCAM.RISKY denotes a low-confidence SCAM prediction, and the system raises an alert when the scam probability exceeds a validation-calibrated threshold.
  • Self-Evolving Context Manager: The context manager retrieves historical events associated with current-window entities and prioritizes suspicious evidence through evolving skills.Skills encode heuristics such as prioritizing financial-app events involving the same entity within 48 hours.
  • Self-Evolving Context Manager: Memory updates on every event, while assessor rationales and identified cues can refine the skill library when predictions are RISKY or SCAM.These updates operate on fast and slow timescales respectively, while the base model parameters remain frozen during deployment.
  • On-Policy Self-Distillation: On-policy self-distillation gives a teacher the same augmented window plus an anti-scam reflection, then trains a student using only the augmented window.Reflections compare partial and complete scam traces under scam-type-specific skills, aligning teacher supervision with the student’s sampled observation distribution.
  • On-Policy Self-Distillation: Scam-related windows use reverse-KL distillation together with cross-entropy, while benign windows use only binary cross-entropy.The coefficient λ > 0 controls the contribution of the cross-entropy loss in the combined objective.

4 Experiments

ORACLE is evaluated as a streaming cross-app scam anticipation system using long-horizon trajectories, ablations, qualitative cases, scaling analysis, and failure analysis. Results show earlier, more accurate, and less false-alert-prone warnings, while revealing limits when evidence is generic or retrieval becomes noisy.

  • Main Results: ORACLE improves streaming cross-app anticipation over GPT-5.1, raising PAR from 77.3 to 98.2, reducing EDP from 46.4 to 29.5, and lowering FAR from 12.8 to 1.3.In single-app content detection, ORACLE achieves 99.7 accuracy, compared with 97.8 for ScamGPT-J and 98.9 for FraudR1.
  • Qualitative Analysis: The evolving memory–skill design links weak early cues with later instructions and transfers, abstracting fragmented events into a reusable scam-stage pattern.The illustrated sequence is job-group contact → third-party app onboarding → task instruction → fund transfer.
  • Ablation Study: The evolving skill achieves 98.4 HR, 29.5 EDP, 1.3 FAR, and 98.2 PAR, outperforming context variants by converting fragmented history into retrieval knowledge.Person-centric memory raises HR and PAR, while static skill improves EDP and FAR; the consolidated evolving skill performs best overall.
  • Ablation Study: OPSD outperforms SFT and GRPO across all metrics, reaching 98.4 HR, 29.5 EDP, 1.3 FAR, and 98.2 PAR.A representative false-negative case shows the teacher reflection identifying a risky transition that the student initially treats as ordinary usage.
  • Ablation Study: Skill-generated experience improves PAR, but using skill experience with KL alone increases FAR; combining KL and CE losses yields the best OPSD performance.The analysis attributes the result to combining soft teacher guidance with grounded CE supervision.
  • Analysis: Window-level accuracy improves as stored scam-related memory grows in the low-memory regime, then plateaus once sufficient relevant evidence has accumulated.The analysis uses window-level accuracy because retrieval is performed independently for each observation window.
  • Analysis: ORACLE may delay alerts for generic early actions, produce false alerts for benign scam-like patterns, or retrieve noisy history when entities recur across unrelated interactions.These three failure modes constrain early detection, false-alert control, and retrieval effectiveness.
  • Analysis: The method produces more stable predictions across adjacent online windows, indicating a more reliable early-warning process.The reported consistency analysis appears in Table 6 in Appendix F.

5 Discussion and Conclusion

The paper evaluates ORACLE for early scam anticipation using a long-horizon benchmark and reports earlier, more reliable warnings with fewer false alerts than strong LLM baselines. However, the curated benchmark may not fully capture the complexity of real-world scam interactions.

  • Discussion: The curated benchmark may not fully capture the complexity of real-world scam interactions.Privacy-preserving, real-world victim datasets are identified as necessary for more faithful evaluation and stronger validation.
  • Conclusion: The benchmark interleaves normal and scam behaviors in long-horizon trajectories and evaluates detection coverage, alert timeliness, and reliability.It was curated from real-world normal app usage data and real scam cases.
  • Conclusion: ORACLE combines a self-evolving context manager with on-policy self-distillation to anticipate scams from fragmented app-usage trajectories.The context manager retrieves historical evidence, while self-distillation transfers reflection-informed early-scam knowledge to a student model without privileged information.
  • Conclusion: ORACLE raises earlier and more reliable warnings with fewer false alerts than strong LLM baselines.The reported outcome supports moving beyond isolated content classification toward cross-temporal reasoning over partial behavioral trajectories.

A Broader Impacts

The proposed approach uses structured app-level events and short summarized content to support cross-app temporal reasoning while reducing the sensitive information required for detection. Deployment nevertheless raises privacy, reliability, and misuse concerns that require safeguards.

  • Privacy-conscious design: The framework operates on structured app-level events and short summarized content rather than full raw messages or detailed interaction logs.This design is presented as reducing the amount of sensitive information required for detection while retaining cross-app temporal reasoning.
  • Privacy-conscious design: The system can provide earlier and more consistent scam warnings with a more privacy-conscious representation.The claim is framed as a result of using the reduced-information representation.
  • Risks and safeguards: Continuous monitoring of summarized app-usage patterns may expose sensitive behavioral traits and raises privacy concerns without appropriate safeguards.The approach still relies on partial user behavior signals despite its reduced information footprint.
  • Risks and safeguards: False alerts could disrupt benign activities or reduce user trust, while delayed or missed alerts may fail to prevent scam-related harm.Memory retrieval may also introduce noisy or outdated context that affects decision reliability.
  • Risks and safeguards: Responsible deployment should incorporate privacy-preserving mechanisms, transparent design, confidence-aware alerting, and safeguards against misuse.Suggested safeguards include local processing, minimal data retention, access control, and auditing.

B.1 Scam Anticipation

Prior smartphone scam detectors mainly analyze isolated applications or interactions, whereas long-horizon cross-app scams require reasoning over gradually emerging intent. Related distillation frameworks provide contextual knowledge but are described in different, simpler environments.

  • Existing approaches: Bot-Wars and ScamGPT-J apply LLM-based scam detection to phone scams and instant-messaging interactions, respectively.Bot-Wars uses a two-layer prompt architecture and dual-stream reasoning, while ScamGPT-J simulates scammer responses in real time.
  • Existing approaches: Single-application or isolated-interaction methods are less suited to long-horizon cross-app scams whose intent emerges across multiple contacts and actions.Their reliance on detailed conversational content also raises privacy concerns for continuous smartphone monitoring.
  • Related distillation frameworks: OPCD and Self-Distilled Reasoner train student models against context-conditioned teachers using generated trajectories and reverse-KL minimization.The contextual knowledge in these frameworks mainly comes from observations and rewards in text-game environments.
  • System example: The structured person record stores an entity name, temporal occurrence order, related application history, and metadata fields.The example records first-seen and last-seen order together with application events.

D Example of Scam Pattern Reflection Skill

The reflection skill describes a fake online investment financial scam as a cross-application sequence involving social contact, messaging, investment-app activity, and banking actions. Its examples emphasize early grooming and unfamiliar-app interactions as warning signals.

  • Scam type: The illustrated scam type is fake_online_investment_financial_scam.It concerns fraudulent investment schemes promoted through social media, messaging applications, or malicious financial apps.
  • Typical app sequence: The typical app sequence is social media app → messaging app → investment app download → banking app for fund transfer.This sequence represents the scam’s progression across application categories.
  • Early warning signals: Early indicators include unfamiliar investment-app installation, contact with an unverified representative, QR-code scanning, and joining a private messaging group.These combinations are characterized as potential fraud or early-stage grooming behavior.
  • Early warning signals: The prompt enhancement directs attention to investment-related app installations, abnormal QR-code scanning, early social or messaging interactions, test transactions, and group joining.It identifies initial test transactions and group-joining activities as critical early warning signals.

E Sensitivity Analysis

The section reports a sensitivity analysis examining the CE-loss weight and window length.

  • The sensitivity analysis varies the CE-loss weight and window length.

F Prediction Consistency

The section evaluates prediction consistency under streaming evaluation, including adjacent overlapping windows and comparisons across methods.

  • Prediction consistency is evaluated across adjacent overlapping windows in streaming evaluation.
  • Table 6 compares the prediction consistency of different methods under streaming evaluation.
  • The dataset is organized using application categories.
Loading 2605.16363v1…